The advent of chemical vapor deposition (CVD) grown graphene has allowed researchers to investigate large area graphene/n-silicon Schottky barrier solar cells. Using chemically doped graphene, efficiencies of nearly 10% can be achieved for devices without anti-reflective coatings. However, many devices reported in past literature often exhibit a distinctive s-shaped kink in the measured I/V curves under illumination resulting in poor fill factor. This behavior is especially prevalent for devices with pristine (not chemically doped) graphene but can be seen in some cases for doped graphene as well. In this work, we show that the native oxide on the silicon presents a transport barrier for photo-generated holes and causes recombination current, which is responsible for causing the kink. We experimentally verify our hypothesis and propose a simple semiconductor physics model that qualitatively captures the effect. Furthermore, we offer an additional optimization to graphene/n-silicon devices: by choosing the optimal oxide thickness, we can increase the efficiency of our devices to 12.4% after chemical doping and to a new record of 15.6% after applying an anti-reflective coating.