We use the GOODS-MUSIC sample, a catalog of ~3000 Ks-selected galaxies based on VLT and HST observation of the GOODS-South field with extended multi-wavelength coverage (from 0.3 to 8 micron) and accurate estimates of the photometric redshifts to select 179 DRGs with J-Ks>1.3 in an area of 135 sq. arcmin. We first show that the J-Ks>1.3 criterion selects a rather heterogeneous sample of galaxies, going from the targeted high-redshift luminous evolved systems, to a significant fraction of lower redshift (1<z<2) and less luminous dusty starbursts. These low-redshift DRGs are significantly less clustered than higher-z DRGs. With the aid of extreme and simplified theoretical models of clustering evolution we show that it is unlikely that the two samples are drawn from the same population observed at two different stages of evolution. High-z DRGs likely represent the progenitors of the more massive and more luminous galaxies in the local Universe and might mark the regions that will later evolve into structures of intermediate mass, like groups or small galaxy clusters. Low-z DRGs, on the other hand, will likely evolve into slightly less massive field galaxies. Comment: paper submitted to A&A, version revised accounting for all referee Comments, uses aa.cls, 12 pages, 10 figures