The work on cellulose fiber composites is typically strictly divided into two separated research fields depending on the fiber origin, that is, from wood and from annual plants, representing the two different industries of forest and agriculture, respectively. The present paper evaluates in parallel wood fibers and plant fibers to highlight their similarities and differences regarding their use as reinforcement in composites and to enable mutual transfer of knowledge and technology between the two research fields. The paper gives an introduction to the morphology, chemistry, and ultrastructure of the fibers, the modeling of the mechanical properties of the fibers, the fiber preforms available for manufacturing of composites, the typical mechanical properties of the composites, the modeling of the mechanical properties with focus on composites having a random fiber orientation and a non-negligible porosity content, and finally, the moisture sensitivity of the composites. The performance of wood and plant fiber composites is compared to the synthetic glass and carbon fibers conventionally used for composites, and advantages and disadvantages of the different fibers are discussed.