<i>Background and Purpose:</i> The modified Rankin Scale (mRS), designed as a measure of disability in the community, has increasingly been administered in the acute stroke setting but has been poorly studied within the hospital environment. We prospectively studied the interrater reliability of the mRS in acutely hospitalised stroke patients and examined the effect of prior experience with the scale and use of a decision tool on the interrater agreement of trained raters. <i>Methods:</i> Patients <4 days after stroke were recruited. Individuals from 3 trained rater groups (experienced, inexperienced and inexperienced with decision tool) independently scored each patient within 6 h of each other. Agreement was measured with the intraclass correlation (ICC) and the weighted kappa statistic (ĸ_w), with systematic bias evaluated using the bias index. <i>Results:</i> Twelve raters scored 56 patients with overall agreement of ICC = 0.675. Agreement of ĸ_w = 0.686 was found between experienced and inexperienced raters but a modest systematic bias was present. Experience in rating patients appeared to play some role in affecting agreement but the decision tool did not improve the performance of inexperienced raters. <i>Conclusions:</i> Trained raters were found to have good interrater agreement overall when the mRS was scored in acute stroke patients but obvious problems with the interpretation and relevancy of the scale in this setting raise concerns about validity. The use of the mRS to rate disability in the acute hospital environment should be questioned.