The effective on-site Coulomb interaction (Hubbard U) is an important parameter for studying strongly correlated systems. While U is determined empirically by fitting to bulk values, its value for a cluster with a finite number of atoms remains uncertain. Here, we choose Gd13 as a prototypical example of a strongly correlated cluster. Contrary to the well-known results in transition-metal clusters where magnetic moments of clusters are larger than their bulk, in Gd13 cluster the magnetic moment is smaller than its bulk value. Using density functional theory and the linear response approach, we determine U self-consistently for the cluster and apply it to explain the anomalous magnetic properties of Gd13. We demonstrate that the interaction between core and shell atoms of the Gd13 cluster strongly depends on the Hubbard U. For U =0 eV magnetism is governed by a direct f-f electron interaction between core and shell atoms, while for U =5.5 eV it is the indirect Ruderman-Kittel-Kasuya-Yosida interaction that prevails. We also demonstrate that the noncollinear spin arrangement of each atom in the cluster strongly depends on the Hubbard U. Monte Carlo calculations further confirm that magnetic moments decrease with temperature, thus addressing a long-standing disagreement in experimental results.