This article is based on interviews with 15 men whose children have, or have had, involvement with Norwegian child welfare services (CWS). The aim is to contribute to an increased knowledge of their experience of fatherhood, factors affecting their self-concept and the consequences this has for their fathering practice. The fathers’ relationship to their parents appears important – in terms of what they emphasise in their childcare, and the importance they ascribe themselves as fathers. The fathers constitute their fatherhood as in opposition to their own fathers, and to their children’s mothers. If fathers are uncertain of their role and importance for their children, while being faced with the CWS’ understanding of mothers as primary caregivers, this can cause them to withdraw. There is a need for a greater degree of reflection regarding different understandings of fatherhood, and how the CWS relate to gender roles and to fathers.