X-ray flares were discovered in the afterglow phase of gamma-ray bursts (GRBs) by the {\em Swift} satellite a decade ago and known as a canonical component in GRB X-ray afterglows. In this paper, we constrain the Lorentz factors of GRB X-ray flares using two different methods. For the first method, we estimate the lower limit on the bulk Lorentz factor with the flare duration and jet break time. In the second method, the upper limit on the Lorentz factor is derived by assuming that the X-ray flare jet has undergone saturated acceleration. We also re-estimate the initial Lorentz factor with GRB afterglow onsets, and find the coefficient of the theoretical Lorentz factor is 1.67 rather than the commonly used 2 for interstellar medium (ISM) and 1.44 for the wind case. We find that the correlation between the limited Lorentz factor and the isotropic radiation energy of X-ray flares in the ISM case is more consistent with that of prompt emission than the wind case in a statistical sense. For a comparison, the lower limit on Lorentz factor is statistically larger than the extrapolation from prompt bursts in the wind case. Our results indicate that X-ray flares and prompt bursts are produced by the same physical mechanism. ; Comment: 16 pages, 6 figures, 1 table, accepted for publication in ApJ