This work extends the results of a recently developed theory of a rather complete thermodynamic formalism for discrete-state, continuous-time Markov processes with and without detailed balance. We investigate whether and in what way the thermodynamic structure is invariant in a multiscale stochastic system, that is, whether the relations between thermodynamic functions of state and process variables remain unchanged when the system is viewed at different time scales and resolutions. Our results show that the dynamics on a fast time scale contribute an entropic term to the internal energy function u(S)(x) for the slow dynamics. Based on the conditional free energy u(S)(x), we can then treat the slow dynamics as if the fast dynamics is nonexistent. Furthermore, we show that the free energy, which characterizes the spontaneous organization in a system without detailed balance, is invariant with or without the fast dynamics: The fast dynamics is assumed to reach stationarity instantaneously on the slow time scale; it has no effect on the system's free energy. The same cannot be said for the entropy and the internal energy, both of which contain the same contribution from the fast dynamics. We also investigate the consequences of time-scale separation in connection to the concepts of quasi-stationarity and steady adiabaticity introduced in the phenomenological steady-state thermodynamics.