Semantic analysis of multimedia data, such as images, image databases and videos, has attracted much attention in the past years. The extraction of human centered descriptions, matching end users cognition, and specifically the detection and identification of events in videos is a particularly challenging problem, due to the volume and diversity of both the low-level features that can be extracted from multimedia data, and also of the corresponding high-level information conveyed. Numerous efforts have begun, attempting to bridge the semantic gap between low-level data and higher level descriptions, often resorting to domainspecific learning-based approaches. The advent, however of the Semantic Web paved a new direction in knowledge management and utilization, advancing aspects such as knowledge sharing and reuse. In this paper we present a novel, generally applicable approach, for hierarchical semantic analysis of spatiotemporal video features (trajectories) in order to localize and detect events of interest. Dynamically changing trajectories are extracted by processing the optical flow, based on its statistics. The temporal evolution of the trajectories' geometrical and spatiotemporal characteristics forms the basis on which event detection is performed. This is based on the exploitation of prior knowledge, which provides the formal conceptualization needed to enable the automatic inference of high level event descriptions. To enhance extensibility, a modular approach has been followed, employing different ontologies for the individual types of knowledge required, namely geometrical, spatiotemporal and domain related. Thus, knowledge and consequently derived descriptions, propagate along successively higher abstraction levels. Experimental results with a variety of surveillance videos are presented to exemplify the usability and effectiveness of the proposed system.