Multi-responsive nanomaterials based on the self-limited assembly of plasmonic nanoparticles are of great interest due to their widespread employment in sensing applications. We present a thorough investigation of a hybrid nanomaterial based on the protein-mediated aggregation of gold nanoparticles at varying protein concentration, pH and temperature. By combining Small Angle X-ray Scattering with extinction spectroscopy, we are able to frame the morphological features of the formed fractal aggregates in a theoretical model based on patchy interactions. Based on this, we established the main factors that determine the assembly process and their strong correlation with the optical properties of the assemblies. Moreover, the calibration curves that we obtained for each parameter investigated based on the extinction spectra point out to the notable flexibility of this nanomaterial, enabling the selection of different working ranges with high sensitivity. Our study opens for the rational tuning of the morphology and the optical properties of plasmonic assemblies to design colorimetric sensors with improved performances.