Systematic variations in the structural and electronic properties of BiFeO3 with A-site substitutions were studied using first-principles density functional theory calculations. It is found that the ferroelectric distortion of BiFeO3 with group IIIA element (Sc3+, Y3+, and La3+) substitutions is significantly affected by the hybridization between substitute d states and oxygen 2p states, while that with group VB element (Sb3+) substitution is stabilized by the s2 lone pair electrons. For both groups, the substitute with smaller ionic size and larger electronegativity causes more significant off-center displacement and narrower band gap.