We present a new implicit asymptotic preserving time integration scheme for charged-particle orbit computation in arbitrary electromagnetic fields. The scheme is built on the Crank-Nicolson integrator and continues to recover full-orbit motion in the small time-step limit, but also recovers all the first-order guiding center drifts as well as the correct gyroradius when stepping over the gyration time-scale. In contrast to previous efforts in this direction, the new scheme also features exact energy conservation. In the derivation of the scheme, we find that a new numerical time-scale is introduced. This scale is analyzed and the resulting restrictions on time-step are derived. Based on this analysis, we develop an adaptive time-stepping strategy the respects these constraints while stepping over the gyration scale when physically justified. It is shown through numerical tests on single-particle motion that the scheme's energy conservation property results in tremendous improvements in accuracy, and that the scheme is able to transition smoothly between magnetized and unmagnetized regimes as a result of the adaptive time-stepping.