In the traditional fermentation process of strong-aroma Baijiu, a fermentation pit mud (FPM) provides many genera of microorganisms for fermentation. However, the functional microorganisms that have an important effect on the quality of Baijiu and their changes with the age of fermentation pit (FP) are poorly understood. Herein, the Roche 454 pyrosequencing technique and a phospholipid fatty-acid analysis were employed to reveal the structure and diversity of prokaryotic communities in FPM samples that have been aged for 5, 30, and 100 years. The results revealed an increase in total prokaryotic biomass with an FP age; however, Shannon’s diversity index decreased significantly (p < 0.01). These results suggested that a unique microbial community structure evolved with uninterrupted use of the FP. The number of functional microorganisms, which could produce the flavor compounds of strong-aroma Baijiu, increased with the FP age. Among them, Clostridium and Ruminococcaceae are microorganisms that directly produce caproic acid. The increase of their relative abundance in the FPM might have improved the quality of strong-aroma Baijiu. Syntrophomonas, Methanobacterium, and Methanocorpusculum might also be beneficial to caproic acid production. They are not directly involved but provide possible environmental factors for caproic acid production. Overall, our study results indicated that an uninterrupted use of the FP shapes the particular microbial community structure in the FPM. This research provides scientific support for the concept that the aged FP yields a high-quality Baijiu.