In order to deal with inhomogeneous groups of subjects in fMRI studies, we investigate several robust statistics to perform random-effect analysis on the mean population effect (sign statistic, Wilcoxon's signed rank statistic and empirical t statistic). The tests' significance levels are calibrated using the same sign permutation framework, allowing for exact specificity control under a mild symmetry assumption about the subjects' distribution. The benefit of the robust approach is illustrated on a speech listening experiment involving three month-old infants