Carbon-oxide and carbon-sulfide nanocomposites have attracted tremendous interest as the anode materials for Li and Na ion batteries. Such composites are fascinating as they often show synergistic effect compared to their singular components. Carbon nanomaterials are often used as the matrix due to their high conductivity, tensile strength, and chemical stability under the battery condition. Metal oxides and sulfides are often used as active material fillers because of their large capacity. Numerous works have shown that by taking one step further into fabricating nanocomposites with rational structure design, much better performance can be achieved. The present review aims to present and discuss the development of carbon-based nanocomposite anodes in both Li ion batteries and Na ion batteries. The authors introduce the individual components in the composites, i.e., carbon matrices (e.g., carbon nanotube, graphene) and metal oxides/sulfides; followed by evaluating how advanced nanostructures benefit from the synergistic effect when put together. Particular attention is placed on strategies employed in fabricating such composites, with examples such as yolk–shell structure, layered-by-layered structure, and composite comprising one or more carbon matrices. Lastly, the authors conclude by highlighting challenges that still persist and their perspective on how to further develop the technologies.