We investigate the Stark shift in Rydberg rubidium atoms through electromagnetically induced transparency for the measurement of direct current (dc) and 60 Hz alternating current (ac) voltages. This technique has direct application to the calibration of voltage measurement instrumentation. We present experimental results for different atomic states that allow for dc and ac voltage measurements ranging from 0 to 12 V. While the state-of-the-art method for realizing the volt, the Josephson voltage standard, is significantly more accurate, the Rydberg atom-based method presented here has the potential to be a calibration standard with more favorable size, weight, power, and cost. We discuss the steps necessary to develop the Rydberg atom-based voltage measurement as a complementary method for dissemination of the voltage scale directly to the end user and discuss sources of uncertainties for these types of experiments.