Traditionally, inertial measurement units- (IMU) based human joint angle estimation requires a priori knowledge about sensor alignment or specific calibration motions. Furthermore, magnetometer measurements can become unreliable indoors. Without magnetometers, however, IMUs lack a heading reference, which leads to unobservability issues. This paper proposes a magnetometer-free estimation method, which provides desirable observability qualities under joint kinematics that sufficiently excite the lower body degrees of freedom. The proposed lower body model expands on the current self-calibrating human-IMU estimation literature and demonstrates a novel knee hinge model, the inclusion of segment length anthropometry, segment cross-leg length discrepancy, and the relationship between the knee axis and femur/tibia segment. The maximum a posteriori problem is formulated as a factor graph and inference is performed via post-hoc, on-manifold global optimization. The method is evaluated (N = 12) for a prescribed human motion profile task. Accuracy of derived knee flexion/extension angle (4.34∘ root mean square error (RMSE)) without magnetometers is similar to current state-of-the-art with magnetometer use. The developed framework can be expanded for modeling additional joints and constraints.