Observations of ultraluminous infrared galaxies (ULIRGs) with an achieved resolution approaching the diffraction limit in the mid-infrared from 8 to 25 μm using the Keck Telescopes are reported. We find extremely compact structures, with spatial scales of less than 0."3 (diameter) in six of the seven ULIRGs observed. These compact sources emit between 30% and 100% of the mid-infrared energy from these galaxies. We have utilized the compact mid-infrared structures as a diagnostic of whether an AGN or a compact (100–300 pc) starburst is the primary power source in these ULIRGs. In Markarian 231, the upper limit on the diameter of the 12.5 μm source, 0."13, shows that the size of the infrared source must increase with increasing wavelength, consistent with AGN models. In IRAS 05189-2524 and IRAS 08572+3915 there is strong evidence that the source size increases with increasing wavelength. This suggests heating by a central source rather than an extended luminosity source, consistent with the optical classification as an AGN. The compact mid-infrared sources seen in the other galaxies cannot be used to distinguish the ultimate luminosity source. If these ULIRGs are powered by compact starbursts, the star formation rates seen in the central few hundred parsecs far exceed the global rates seen in nearby starburst galaxies, and approach the surface brightness of individual clusters in nearby starburst galaxies.