Well-defined silver nanoparticles were doped into bio-based amorphous silica (Ag-b-SiO2) with different silver contents (from 2 to 20 wt%) by a solvent-free procedure. The four as-synthetized samples were hydrogenated at 300 °C to ensure the formation of zero-valent Ag nanoparticles. The prepared samples were characterized by X-ray powder diffraction (XRD), elemental analysis, N2 sorption measurements, scanning electron microscopy (SEM), X-ray photoelectron spectroscopy (XPS), and high-resolution transmission electron microscopy (HR-TEM). The characterization data confirmed the formation of well-defined zero-valent silver nanoparticles in the range of 3–10 nm in the low-loading samples, while in high-loading samples, bulky particles of silver in the range of 200–500 nm were formed. The in vitro cytotoxic activities of the Ag-b-SiO2 samples were tested against the tumor cell lines of breast (MCF-7), liver (HepG2), and colon (HCT 116) over a concentration range of 0.01 to 1000 g. The prepared samples exhibited a wide range of cytotoxic activities against cancer cells. An inverse relationship was observed between the silver nanoparticles’ size and the cytotoxic activity, while a direct relationship between the silver nanoparticles’ size and the apoptotic cell death was noticed.