Registration of 3D facial surfaces means establishing point-to-point correspondence between two 3D facial surfaces. Difficulties typical for the registration of 3D facial surfaces are varying illumination, pose or viewpoint changes, varying facial expressions, and different appearance of individuals. In this work we propose to use a covariance matrix as descriptor for the neighborhood of a salient point in a face. It encodes the variance of the channels, such as red, green, blue, depth, etc., their correlations with each other, and spatial layout, while filtering out the influence of the disturbing effects mentioned above. A pyramidal approach is applied where first the location of a corresponding point is computed roughly and then the position is gradually refined. The method does not require any training. Particle Swarm Optimization makes the search for corresponding points more efficient. Results with a challenging dataset confirm that the approach works greatly for a variety of disturbing effects.