We report a series of quantum-chemical calculations for the ground and some of the low-lying excited states of an isolated LiYb molecule by the spin-orbit multistate complete active space second-order perturbation theory (SO-MS-CASPT2). Potential energy curves, spectroscopic constants, and transition dipole moments (TDMs) at both spin-free and spin-orbit levels are obtained. Large spin-orbit effects especially in the TDMs of the molecular states dissociating to Yb((3)P(0,1,2)) excited states are found. To ensure the reliability of our calculations, we test five types of incremental basis sets and study their effect on the equilibrium distance and dissociation energy of the ground state. We also compare CASPT2 and CCSD(T) results for the ground state spectroscopic constants at the spin-free relativistic level. The discrepancies between the CASPT2 and CCSD(T) results are only 0.01 Å in equilibrium bond distance (R(e)) and 200 cm(-1) in dissociation energy (D(e)). Our CASPT2 calculation in the supermolecular state (R=100 a.u.) with the largest basis set reproduces experimental atomic excitation energies within 3% error. Transition dipole moments of the super molecular state (R=100 a.u.) dissociating to Li((2)P) excited states are quite close to experimental atomic TDMs as compared to the Yb((3)P) and Yb((1)P) excited states. The information obtained from this work would be useful for ultracold photoassociation experiments on LiYb.