Clothing and the enclosed air layers highly affect heat dissipation from the body and thus, are crucial factors when it comes to thermal comfort. The heat and moisture transfer is affected by the variation of the size and the shape of air gaps between the garment and the human body. In addition, the fabric and garment design properties can affect the amount of heat loss from different body parts. In this study, we investigated the effect of fabric properties (different raw materials and weave types) and the garment fit on the heat loss through the garment combinations (undershirt and shirt) for the different parts of the upper body (trunk, chest, and back) using a sweating thermal manikin. The undershirt fit and the raw material of the shirts showed strong effects on the dry thermal resistance of the garment combinations. Moreover, the undershirt properties affected the evaporative heat loss from garment combinations, and the magnitude of these effects varied over different body regions. Whilst the undershirt fit had a significant impact on the evaporative heat loss of the back region, the influence of the undershirt raw material was more important in the chest region. The findings of this study provide fundamental knowledge to improve the thermal comfort of garment combinations for office wear.