We investigate the effect of active galactic nucleus (AGN) variability on the observed connection between star formation and black hole accretion in extragalactic surveys. Recent studies have reported relatively weak correlations between observed AGN luminosities and the properties of AGN hosts, which has been interpreted to imply that there is no direct connection between AGN activity and star formation. However, AGNs may be expected to vary significantly on a wide range of timescales (from hours to Myr) that are far shorter than the typical timescale for star formation (>~100 Myr). This variability can have important consequences for observed correlations. We present a simple model in which all star-forming galaxies host an AGN when averaged over ~100 Myr timescales, with long-term average AGN accretion rates that are perfectly correlated with the star formation rate (SFR). We show that reasonable prescriptions for AGN variability reproduce the observed weak correlations between SFR and L_AGN in typical AGN host galaxies, as well as the general trends in the observed AGN luminosity functions, merger fractions, and measurements of the average AGN luminosity as a function of SFR. These results imply there may be a tight connection between AGN activity and SFR over galaxy evolution timescales, and that the apparent similarities in rest-frame colors, merger rates, and clustering of AGNs compared to "inactive" galaxies may be due primarily to AGN variability. The results provide motivation for future deep, wide extragalactic surveys that can measure the distribution of AGN accretion rates as a function of SFR. ; Comment: 11 pages, 7 figures, ApJ in press; v3: minor revisions; model output available at http://www.dartmouth.edu/~hickox/sfagn.php