The counter-electrode (CE) material in electrochemical metallization memory (ECM) cells plays a crucial role in the switching process by affecting the reactions at the CE/electrolyte interface. This is due to the different electrocatalytic activity of the CE material towards reduction–oxidation reactions, which determines the metal ion concentration in the electrolyte and ultimately impacts the switching kinetics. In this study, the focus is laid on Pt, TiN, and W, which are relevant in standard chip technology. For these, the influence of CE metal on the switching kinetics of Ag/HfO2-based volatile ECM cells is investigated. Rectangular voltage pulses of different amplitudes were applied, and the SET times were analyzed from the transient curves. The results show that CE material has a significant effect on the SET kinetics, with differences being observed depending on the voltage regime. The formation of interfacial oxides at the CE/electrolyte interface, particularly for non-noble metals, is also discussed in relation to the findings. Overall, this work highlights the important role of the CE material in the switching process of Ag/HfO2-based diffusive memristors and the importance of considering interfacial oxide formation in the design of these devices.