A thorough study of the RbTiOPO4(RTP) crystallization in its self-flux and WO3-containing fluxes (10, 20, and 30 mol % WO3) has been performed. The composition regions and isotherms of crystallization were obtained, and most of the crystallized neighboring phases were identified. Afterward, the possibilities of doping and codoping RTP crystals with Er3+ and Nb5+ were studied. Adding Nb2O5 substituting TiO2 in the solution increases the distribution coefficient of Er3+ but changes the crystal morphology toward thin plates significantly. This means it is difficult to grow crystals of sufficient quality and size for research and applications. To optimize the crystal growth process, the conditions for growing doped and codoped RTP single crystals with Er3+ and Nb3+ by the top seeded solution growth technique (TSSG) were studied. For crystal growth from self-flux, stirring the solution with an immersed platinum turbine significantly increased the efficiency of the crystal growth process. These conditions allow achieving 0.65 × 1020 atom·cm-3 as an Er3+ dopant concentration in the crystal. The Judd−Ofelt parameters for Er3+ in RTP:Nb determined from the 300 K optical absorption spectra are Ω2 = 5.99 × 10-20, Ω4 = 0.54 × 10-20, and Ω6 = 0.37 × 10-20 cm2. Finally, the second harmonic generation (SHG) efficiency of RTP:Nb single crystals increased as the concentration of Nb increased up to a 4 atom % of Ti4+ substitution, after which the SHG efficiency decreased.