Recent observations opened up a new window on the inflationary model building. As it was firstly reported by the WMAP data, there may be some indications of statistical anisotropy on the CMB map, although the statistical significance of these findings are under debate. Motivated by these observations, people begun considering new inflationary models which may lead to statistical anisotropy. The simplest possible way to construct anisotropic inflation is to introduce vector fields. During the course of this thesis, we study models of anisotropic inflation and their observational implications such as power spectrum, bispectrum etc. Firstly we build a new model, which contains the gauge field which breaks the conformal invariance while preserving the gauge invariance. We show that in these kind of models, there can be an attractor phase in the evolution of the system when the back-reaction of the gauge field becomes important in the evolution of the inflaton field. We then study the cosmological perturbation theory in these kind of models. More specifically, we calculate the anisotropic corrections due to the presence of the vector field. We then generalize the separate universe formalism to our anisotropic set up and use it in some specific examples of anisotropic inflation. Finally, we connect the primordial anisotropies to the specific examples and to CMB observations. We calculate the TT, TE, TB, EB and BB correlation in the model of charged scalar field model and look for the unique signatures that the anisotropic inflation can have on the CMB map. Any future detection of these statistical anisotropies would rule out the isotropic FRW models. ; Comment: 269 pages, 33 figures. The materials of this PhD thesis are based on the results from arXiv:1010.5495, arXiv:1111.1919, arXiv:1301.1219, arXiv:1302.6986, arXiv:1311.0493, arXiv:1401.4406, arXiv:1404.5112, arXiv:1408.2096