In addition to the usual superconducting current, Josephson junctions (JJs) support a phase-dependent conductance related to the retardation effect of tunneling quasi-particles. This introduces a dissipative current with a memory-resistive (memristive) character that should also affect the current noise. By means of the microscopic theory of tunnel junctions we compute the complete current autocorrelation function of a Josephson tunnel junction and show that this memristive component gives rise to both a previously noted phase-dependent thermal noise, and an undescribed non-stationary, phase-dependent dynamic noise. As experiments are approaching ranges in which these effects may be observed, we examine the form and magnitude of these processes. Their phase dependence can be realized experimentally as a hysteresis effect and may be used to probe defects present in JJ based qubits and in other superconducting electronics applications.