We compute the second-order roughness correction to the Casimir energy for two parallel metallic plates. We compare the proximity force and scattering approaches, showing that the former is obtained analytically in the limit of very smooth surfaces by neglecting non-specular reflections. We calculate the discrepancy between the two approaches for some typical numerical examples.