Cardiac surgery and subsequent treatment in the intensive care unit (ICU) has been shown to be associated with functional decline, especially in elderly patients. Due to the different assessment tools and assessment periods, it remains yet unclear what parameters determine unfavorable outcomes. This study sought to identify risk factors during the entire perioperative period and focused on the decline in activity of daily living (ADL) half a year after cardiac surgery. Follow-ups of 125 patients were available. It was found that in the majority of patients (60%), the mean ADL declined by 4.9 points (95% CI, −6.4 to −3.5; p < 0.000). In the “No decline” -group, the ADL rose by 3.3 points (2.0 to 4.6; p < 0.001). A multiple regression analysis revealed that preoperative cognitive impairment (MMSE ≤ 26; Exp(B) 2.862 (95%CI, 1.192–6.872); p = 0.019) and duration of postoperative delirium ≥ 2 days (Exp(B) 3.534 (1.094–11.411); p = 0.035) was independently associated with ADL decline half a year after the operation and ICU. Of note, preoperative ADL per se was neither associated with baseline cognitive function nor a risk factor for functional decline. We conclude that the preoperative assessment of cognitive function, rather than functional assessments, should be part of risk stratification when planning complex cardiosurgical procedures.