Radial velocity measurements are presented for 85 late M- and L-type very low mass stars and brown dwarfs obtained with the Magellan Echellette (MagE) spectrograph. Targets primarily have distances within 20 pc of the Sun, with more distant sources selected for their unusual spectral energy distributions. We achieved precisions of 2--3 km/s, and combined these with astrometric and spectrophotometric data to calculate $UVW$ velocities. Most are members of the thin disk of the Galaxy, and velocity dispersions indicate a mean age of 5.2$±$0.2 Gyr for sources within 20 pc. We find significantly different kinematic ages between late-M dwarfs (4.0$±$0.2 Gyr) and L dwarfs (6.5$±$0.4 Gyr) in our sample that are contrary to predictions from prior simulations. This difference appears to be driven by a dispersed population of unusually blue L dwarfs which may be more prevalent in our local volume-limited sample than in deeper magnitude-limited surveys. The L dwarfs exhibit an asymmetric $U$ velocity distribution with a net inward flow, similar to gradients recently detected in local stellar samples. Simulations incorporating brown dwarf evolution and Galactic orbital dynamics are unable to reproduce the velocity asymmetry, suggesting non-axisymmetric perturbations or two distinct L dwarf populations. We also find the L dwarfs to have a kinematic age-activity correlation similar to more massive stars. We identify several sources with low surface gravities, and two new substellar candidate members of nearby young moving groups: the astrometric binary DENIS J08230313$-$4912012AB, a low-probability member of the $β$ Pictoris Moving Group; and 2MASS J15104786-2818174, a moderate-probability member of the 30-50 Myr Argus Association. ; Comment: 81 pages, 77 figures, accepted for publication to ApJS