In this study, the failure mechanism of crumb rubber–asphalt pavement was analyzed under the combined effects of low temperature, water, and traffic load. The investigation was carried out based on the mechanical and deformational properties of crumb rubber–asphalt mixture and the typical environmental and load conditions such pavement is typically exposed to. A method was proposed for objective evaluation of the interfacial adhesion between rubber crumbs and asphalt through consideration of the effects of the characteristics of the materials and the working environment. The main evaluation method used herein included the indirect tensile strength test under freeze–thaw–boiling cycle, and the Cántabro abrasion test under water-immersion was adopted as an auxiliary method. The evaluation system has the advantages of simple implementation, realistic simulation of the actual working state of the mixture, and reliable results. Moreover, it is a durability evaluation method that can be specifically applied to asphalt mixtures with some special aggregates or stone mastic asphalt (SMA) mixtures.