We attempt to encode an image in a fashion that is only weakly dependent on rotation of objects within the image, as an expansion of Roger Grosse's work on Translation Invariant sparse coding. Our approach is to specify only a small set of basis images, from which some reasonably large number rotated bases are calculated. The image is trained to this set of rotated bases, so that ultimately the spare code representation is given in terms of rotations of a small set of vectors. We develop an image representation scheme and several algorithms suited to this problem, particularly gradient descent methods for sparse coding and an investigation of rotation invariant Principal Components Analysis. Our experimental results suggest that PCA is significantly more fruitful, unless better algorithms for the sparse coding side can be developed in the future.