We review the relation between Casimir effect and geometry, emphasizing deviations from the commonly used Proximity Force Approximation (PFA). We use to this aim the scattering formalism which is nowadays the best tool available for accurate and reliable theory-experiment comparisons. We first recall the main lines of this formalism when the mirrors can be considered to obey specular reflection. We then discuss the more general case where non planar mirrors give rise to non-specular reflection with wavevectors and field polarisations mixed. The general formalism has already been fruitfully used for evaluating the effect of roughness on the Casimir force as well as the lateral Casimir force or Casimir torque appearing between corrugated surfaces. In this short review, we focus our attention on the case of the lateral force which should make possible in the future an experimental demonstration of the nontrivial (i.e. beyond PFA) interplay of geometry and Casimir effect. ; Comment: corrected typos, added references, QFEXT'07 special issue in J. Phys. A