A novel critical vector (CV) regression algorithm is proposed in the paper based on our previous work and PRESS statistics. The proposed regularized CV algorithm finds critical vectors in a successive greedy process in which, compared to the classical OLS algorithm, the orthogonalization has been removed from the algorithm. The performance of the proposed algorithm is comparable to the OLS algorithm while it saves a lot of time complexities in implementing orthogonalization needed in the OLS algorithm.