Connecting three dimensional structure and affinity is analogous to seeking the 'Archimedean point', a vantage point from where any observer can quantitatively perceive the subject of inquiry. Here we review current knowledge and challenges that lie ahead of us in the quest for this Archimedean point. We argue that current models are limited in reproducing measured data because molecular description of binding affinity must expand beyond the interfacial contribution and also incorporate effects stemming from conformational changes/dynamics and long-range interactions. Fortunately, explicit modeling of various kinetic schemes underlying biomolecular recognition and confined systems that reflect in vivo interactions are coming within reach. This quest will hopefully lead to an accurate biophysical interpretation of binding affinity that would allow unprecedented understanding of the molecular basis of life through unraveling the why's of interaction networks.