Our first impression of others is highly influenced by their facial appearance. However, the perception and evaluation of faces is not only guided by internal features such as facial expressions, but also highly dependent on contextual information such as secondhand information (verbal descriptions) about the target person. To investigate the time course of contextual influences on cortical face processing, event-related brain potentials were investigated in response to neutral faces, which were preceded by brief verbal descriptions containing cues of affective valence (negative, neutral, positive) and self-reference (self-related vs. other-related). ERP analysis demonstrated that early and late stages of face processing are enhanced by negative and positive as well as self-relevant descriptions, although faces per se did not differ perceptually. Affective ratings of the faces confirmed these findings. Altogether, these results demonstrate for the first time both on an electrocortical and behavioral level how contextual information modifies early visual perception in a top-down manner.