Seasonal to decadal predictions are inevitably uncertain, depending on the size of the predictable signal relative to unpredictable chaos. Uncertainties can be accounted for using ensemble techniques, permitting quantitative probabilistic forecasts. In a perfect system, each ensemble member would represent a potential realization of the true evolution of the climate system, and the predictable components in models and reality would be equal. However, we show that the predictable component is sometimes lower in models than observations, especially for seasonal forecasts of the North Atlantic Oscillation and multi-year forecasts of North Atlantic temperature and pressure. In these cases the forecasts are under-confident, with each ensemble member containing too much noise. Consequently, most deterministic and probabilistic measures under-estimate potential skill and idealized model experiments under-estimate predictability. However, skilful and reliable predictions may be achieved using a large ensemble to reduce noise and adjusting the forecast variance through a post-processing technique proposed here.