Nanoscale antennas sandwiched between two graphene monolayers yield a photodetector that efficiently converts visible and near-infrared photons into electrons with an 800% enhancement of the photocurrent relative to the antennaless graphene device. The antenna contributes to the photocurrent in two ways: by the transfer of hot electrons generated in the antenna structure upon plasmon decay, as well as by direct plasmon-enhanced excitation of intrinsic graphene electrons due to the antenna near field. This results in a graphene-based photodetector achieving up to 20% internal quantum efficiency in the visible and near-infrared regions of the spectrum. This device can serve as a model for merging the light-harvesting characteristics of optical frequency antennas with the highly attractive transport properties of graphene in new optoelectronic devices.