Cognitive radio networks (CRNs) rely on sensing of the licensed spectrum of a primary network to dynamically ascertain underutilized portion of the spectrum, thus affording additional communication opportunities. In a CRN, a single homogeneous spectrum access, such as interweave only deprives the secondary users (SUs) of channel access during handoff, particularly at high primary network traffic. Therefore, providing quality-of-service (QoS) to multi-class SUs with diverse delay requirements during handoff becomes a challenging task. In this paper, we have evolved a Markov-based analytical model to ascertain the gain in non-switching spectrum handoff scheme for multi-class SUs employing hybrid interweave-underlay spectrum access strategy. To satisfy the QoS requirements of the delay-sensitive traffic, we have analyzed the impact of hybrid spectrum access scheme for prioritized multi-class SUs traffic. The results show substantial improvement in spectrum utilization, average system throughput and extended data delivery time compared to conventional CRN using interweave only spectrum access. This demonstrates the suitability of the proposed scheme towards meeting QoS requirements of the delay-sensitive SU traffic while improving the overall performance for delay-tolerant SU traffic as well.