Reversible resistive switching between high-resistance and low-resistance states in metal-oxide-metal heterostructures makes them very interesting for applications in random access memories. While recent experimental work has shown that inserting a metallic "oxygen scavenger layer" between the positive electrode and oxide improves device performance, the fundamental understanding of how the scavenger layer modifies heterostructure properties is lacking. We use density functional theory to calculate thermodynamic properties and conductance of TiN/HfO2/TiN heterostructures with and without Ta scavenger layer. First, we show that Ta insertion lowers the formation energy of low-resistance states. Second, while the Ta scavenger layer reduces the Schottky barrier height in the high-resistance state by modifying the interface charge at the oxide-electrode interface, the heterostructure maintains a high resistance ratio between high- and low-resistance states. Finally, we show that the low-bias conductance of device on-states becomes much less sensitive to the spatial distribution of oxygen removed from the HfO2 in the presence of the Ta layer. By providing fundamental understanding of the observed improvements with scavenger layers, we open a path to engineer interfaces with oxygen scavenger layers to control and enhance device performance. In turn, this may enable the realization of a non-volatile low-power memory technology with concomitant reduction in energy consumption by consumer electronics and significant benefits to society.