Rapid eye movement (REM) sleep is generated, in part, by activating muscarinic cholinergic receptors (mAChRs) in the medial pontine reticular formation (mPRF). Molecular cloning has identified five mAChR subtypes, and this study tested the hypothesis that the M2 subtype in the mPRF modulates the amount of REM sleep. This hypothesis cannot be tested directly, due to lack of subtype selective muscarinic agonists. However, the amount of REM sleep can be enhanced by mPRF microinjection of a muscarinic agonist, and the relative potencies of muscarinic antagonists to block the REM sleep enhancement can be determined. Two muscarinic antagonists, methoctramine and 4-DAMP, were studied. Six concentrations of each antagonist were microinjected into the mPRF of conscious cat 15 min prior to the agonist bethanechol. Nonlinear regression analysis was used to calculate the dose of antagonist that caused a 50% inhibition (ID50) of bethanechol-induced REM sleep. Bethanechol significantly increased (442%) the amount of time spent in REM sleep. Both methoctramine and 4-DAMP significantly blocked the bethanechol-induced REM sleep increase, with an ID50 of 1.8 microM and 0.6 microM, respectively. The ID50 ratio for methoctramine-to-4-DAMP (3.0) was similar to the affinity ratio of methoctramine-to-4-DAMP only at the M2 subtype (3.5), suggesting that the M2 subtype in the mPRF modulates the amount of REM sleep. This study also tested the null hypothesis that sleep-dependent respiratory depression evoked by mPRF cholinomimetics would not be antagonized by pretreatment of the mPRF with muscarinic antagonists. Neither methoctramine nor 4-DAMP antagonized the bethanechol-induced decrease in respiratory rate.