The absorption spectra of congenetic wurtzite (WZ) and zincblende (ZB) CdS magic-sized clusters are investigated. We demonstrate that the exciton peak positions can be tuned by up to 500 meV by varying the strong coupling between X-type ligands and the semiconductor cores, while the addition of L-type ligands primarily affects cluster midgap states. When Z-type ligands are displaced by L-type ligands, red shifts in the absorption spectra are observed, despite the fact there is a small decrease in cluster size. Density functional theory calculations are used to explain these findings and they reveal the importance of Cd and S dangling bonds on the midgap states during the Z- to L-type ligand exchange process. Overall, ZB CdS clusters show higher chemical stability than WZ clusters but their optical properties exhibit greater sensitivity to the solvent. Conversely, WZ CdS clusters are not stable in a Lewis base-rich environment, resulting in various changes in their spectra. Our findings enable researchers to select capping ligands that modulate the optical properties of semiconductor clusters while maintaining precise control over their solvent interactions.