In this Letter we demonstrate that the phase segregated state observed in many rare-earth perovskites constitutes a sort of self-generated assembly of magnetic clusters in which magnetic interaction introduces collectivity among them. We show that the observed glassy behavior (memory, aging, etc.) can be perfectly understood taking into account only the intercluster interactions. We address the fundamental question about whether this state constitutes classical spin glass or if, on the other hand, a new universality class must be defined.