Background: Prior studies have reported inconsistent results or less well-explored associations between sex hormones and non-alcoholic fatty liver disease (NAFLD). Here, we aimed to investigate the associations of NAFLD with sex steroids and sex hormone-binding globulin (SHBG) in the population-based study and conduct a comprehensive systematic review and meta-analysis of all published observational studies. Methods: Analyses included 755 men and 1109 women with available data on sex steroids, SHBG, and ultrasound-based NAFLD from the Rotterdam Study. Multivariable regression models were used to examine the associations. Additionally, we searched five databases from inception to 1 April 2022 and performed a systematic review and meta-analysis. Random-effects (DerSimonian-Laird) method was used for meta-analysis, odds ratios (ORs) were calculated for the effect estimate, subgroup and leave-one-out sensitivity analyses were conducted, and meta-regression was performed to explore the pooled statistics with high heterogeneity. Results: In the Rotterdam Study, lower levels of SHBG were associated with NAFLD in both sexes, while lower testosterone was associated with NAFLD only among women. Similarly, the meta-analysis of 16 studies indicated no sex-specific association between SHBG and NAFLD (men: OR = 0.37, 95%CI 0.21–0.53; women: OR = 0.40, 95%CI 0.21–0.60), yet there was a sex-specific association between testosterone and NAFLD (men: OR = 0.59, 95%CI 0.42–0.76; women: OR = 1.06, 95%CI 0.68–1.44). Moreover, men with NAFLD had lower estradiol levels than those without NAFLD. Conclusions: Lower SHBG levels were associated with NAFLD in both sexes, but testosterone levels were associated in a sex-specific manner. In addition, our results showed estradiol with the potential as a protective factor for NAFLD in healthy men.