The magnetic ground state of CeRhGe3 has been investigated using magnetic susceptibility, heat capacity, neutron diffraction, muon spin relaxation (μSR), and inelastic neutron scattering (INS) techniques. Our μSR study clearly reveals the presence of two frequencies below TN2 = 7 K and three frequencies between 7 K and TN1 = 14.5 K, indicating long-range magnetic ordering of the Ce3+ moment. The temperature dependence of the highest frequency follows a mean-field order parameter. Our powder neutron diffraction study at 1.5 K reveals the presence of magnetic Bragg peaks, indexed by the propagation vector k = (0, 0, 3/4) with the Ce3+ magnetic moment ∼0.45(9) μB along the c axis. INS studies at 18 K (i.e., above TN1) show the presence of two well-defined crystal-field (CEF) excitations at 7.5 and 18 meV. At 10 and 4.5 K, a very small increase has been observed in the CEF excitation energies. At 100 K, both CEF excitations broaden and a broad quasielastic component has also been observed. Further, the low-energy INS study reveals the presence of a nearly temperature-independent quasielastic linewidth between 16 and 60 K, which indicates a Kondo temperature TK = 12.6(3) K. The presence of well-defined CEF excitations in CeRhGe3 suggests local moment magnetism and may explain the absence of pressure-induced superconductivity. Analyzing the INS data based on a CEF model, we have evaluated the CEF ground-state wave functions and ground-state moment. The observed small value of the ordered moment along the c axis, deduced from the neutron diffraction data, contrasts with the ab-plane moment direction predicted by the single-ion CEF anisotropy and indicates the presence of two-ion anisotropic magnetic exchange interactions, which govern the direction of the moment.