Reverse transcription-quantitative PCR (RT-qPCR)-based tests are widely used to diagnose coronavirus disease 2019 (COVID-19). As a result that these tests cannot be done in local clinics where RT-qPCR testing capability is lacking, rapid antigen tests (RATs) for COVID-19 based on lateral flow immunoassays are used for rapid diagnosis. However, their sensitivity compared with each other and with RT-qPCR and infectious virus isolation has not been examined. Here, we compared the sensitivity among four RATs by using severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) isolates and several types of COVID-19 patient specimens and compared their sensitivity with that of RT-qPCR and infectious virus isolation. Although the RATs read the samples containing large amounts of virus as positive, even the most sensitive RAT read the samples containing small amounts of virus as negative. Moreover, all RATs tested failed to detect viral antigens in several specimens from which the virus was isolated. The current RATs will likely miss some COVID-19 patients who are shedding infectious SARS-CoV-2.