The discrimination of interaural phase differences (IPDs) requires accurate binaural temporal processing and has been used as a measure of sensitivity to temporal envelope and temporal fine structure (TFS). Previous studies found that TFS-IPD discrimination declined with age and with sensorineural hearing loss (SNHL), but age and SNHL have often been confounded. The aim of this study was to determine the independent contributions of age and SNHL to TFS and envelope IPD discrimination by using a sample of adults with a wide range of ages and SNHL. A two-interval, two-alternative forced-choice procedure was used to measure IPD discrimination thresholds for 20-Hz amplitude-modulated tones with carrier frequencies of 250 or 500 Hz when the IPD was in either the stimulus envelope or TFS. There were positive correlations between absolute thresholds and TFS-IPD thresholds, but not envelope-IPD thresholds, when age was accounted for. This supports the idea that SNHL affects TFS processing independently to age. Age was positively correlated with envelope-IPD thresholds at both carrier frequencies and TFS-IPD thresholds at 500 Hz, when absolute thresholds were accounted for. These results suggest that age negatively affects the binaural processing of envelope and TFS at some frequencies independently of SNHL.