. To detect emotional user behavior, particularly anger, can be very useful for successful automatic dialog processing. We present databases and prosodic classifiers implemented for the recognition of emotion in Verbmobil. Using a prosodic feature vector alone is, however, not sufficient for the modelling of emotional user behavior. Therefore, a module is described that combines several knowledge sources within an integrated classification of trouble in communication. 1 Introduction Research on the automatic processing of speech and language in the last ten years showed the beginning integration of prosody and a broadening of the view towards more sophisticated dialog systems. It might well be that for the next decade to come, the integration of emotion into such systems will be one of the pivotal topics. Applications that can be imagined are: automatic dialog systems, especially within call centers, operation of machines, especially car navigation, monitoring of pilots, interactive ...