In this study, we demonstrate that introducing of rare-earth elements, $R$ = La or Pr, into the Bi-O charge reservoir layer of BiCuSeO leads to an increase of both, the charge carrier concentration and the effective mass. Although the charge carrier mobility slightly decreases upon Bi$^{3+}$ to $R^{3+}$ substitution, the electronic transport properties are significantly improved in a broad temperature range from 100 K to 800 K. In particular, the electrical resistivity decreases by two times, while the Seebeck coefficient drops from 323 $μ$V K$^{-1}$ to 238 $μ$V K$^{-1}$ at 800 K. Thus, a power factor of nearly 3 $μ$W cm$^{-1}$ K$^{-2}$ is achieved for Bi$_{0.92}$La$_{0.08}$CuSeO sample at 800 K. Meanwhile, a noticeable decrease of the lattice thermal conductivity is observed for the substituted samples, which can be attributed to the enhanced point defect scattering mostly originated from atomic mass fluctuations between $R$ and Bi. Ultimately, a maximum $zT$ value of nearly 0.34 at 800 K is obtained for the Bi$_{0.92}$La$_{0.08}$CuSeO sample, which is ~30% higher than that of pristine BiCuSeO.