We investigate the organization of the low energy energetic particles (≤1 MeV) by solar wind structures, in particular corotating interaction regions (CIRs) and shocks driven by interplanetary coronal mass ejections, during the declining-to-minimum phase of Solar Cycle 23 from Carrington rotation 1999 to 2088 (January 2003 to October 2009). Because CIR-associated particles are very prominent during the solar minimum, the unusually long solar minimum period of this current cycle provides an opportunity to examine the overall organization of CIR energetic particles for a much longer period than during any other minimum since the dawn of the Space Age. We find that the particle enhancements associated with CIRs this minimum period recurred for many solar rotations, up to 30 at times, due to several high-speed solar wind streams that persisted. However, very few significant CIR-related energetic particle enhancements were observed towards the end of our study period, reflecting the overall weak high-speed streams that occurred at this time. We also contrast the solar minimum observations with the declining phase when a number of solar energetic particle events occurred, producing a mixed particle population. In addition, we compare the observations from this minimum period with those from the previous solar cycle. One of the main differences we find is the shorter recurrence rate of the high-speed solar wind streams (∼10 solar rotations) and the related CIR energetic particle enhancements for the Solar Cycle 22 minimum period. Overall our study provides insight into the coexistence of different populations of energetic particles, as well as an overview of the large-scale organization of the energetic particle populations approaching the beginning of Solar Cycle 24.