Southern Tuscany (Italy) is an important metallogenic district that hosts relevant S-polymetallic deposits that have intensely been exploited for centuries. Consequently, potential toxic elements, such as Hg and As, are widely distributed in the surrounding environment. In this paper, an extensive sedimentological, mineralogical and geochemical study of two Late Quaternary sediment profiles, partially outcropping along the coast of southern Tuscany (Ansedonia area), was carried out to evaluate the contents and mobility of Hg and As with the aims to contribute to the definition of the geochemical baseline of southern Tuscany before the human intervention and evaluate the potential dispersion of these harmful elements. The sedimentological, mineralogical and geochemical (major elements) features revealed that the studied profiles are mostly related to the local geological characteristics and the Quaternary geological history of the area. The concentrations and the normalized patterns of trace and rare earth elements highlighted the absence of any anthropogenic activity. This implies that the studied samples are to be regarded as good proxies for evaluating the geochemical baseline of southern Tuscany before the intense mining activity. The enrichment factors (EF) of most trace elements were indeed lower or close to 2, indicating a variability close to the average concentration of the Upper Continental Crust (UCC), while other elements slightly enriched, such as Pb, were in agreement with the natural baseline reported for southern Tuscany. Mercury and As displayed EF values >40 when compared to the average contents of UCC, although they decrease down to 4 when compared to the suggested baseline for southern Tuscany. The higher Hg and As contents detected in this study, inferred to natural sources, evidenced (i) the great natural variability occurring in largely mineralized areas and (ii) the importance of estimating reference environmental parameters in order to avoid misleading interpretations of the detected anomalies. Moreover, the results of leaching test on sediment samples denoted a relatively low mobility of Hg and As, suggesting that these elements are preferentially mobilized by transport of clastic sediments and such anomalies may be preserved for relatively long times in Quaternary sediments. However, leachable Hg (0.6–9.7 μg/L) and As (2.1–42.2 μg/L) concentrations are significantly high when compared to those of the Italian limit for groundwater (1 µg/L for Hg and 10 µg/L for As). Quaternary sediments from southern Tuscany could then be a potential, though natural, source of Hg and As to groundwater systems.