X-ray speckle visibility spectroscopy (XSVS) is a method for studying dynamics in disordered systems. This method was originally developed in the visible-light region, in which an intense laser can be used. When applied in the X-ray region, where the number of photons is much smaller than in the visible-light region, it suffers from photon statistics. In this paper, we quantitatively discuss the effect of photon shot noise on XSVS analyses. The effect is experimentally confirmed using sequential speckle patterns from Brownian polystyrene nanospheres in glycerol.