Giant elliptical galaxies, believed to be built from the merger of lesser galaxies, are known to house a massive black hole at their center rather than a compact star cluster. If low- and intermediate-mass galaxies do indeed partake in the hierarchical merger scenario, then one needs to explain why their dense nuclear star clusters are not preserved in merger events. A valuable clue may the recent revelation that nuclear star clusters and massive black holes frequently co-exist in intermediate mass bulges and elliptical galaxies. In an effort to understand the physical mechanism responsible for the disappearance of nuclear star clusters, we have numerically investigated the evolution of merging star clusters with seed black holes. Using black holes that are 1-5% of their host nuclear cluster mass, we reveal how their binary coalescence during a merger dynamically heats the newly wed star cluster, expanding it, significantly lowering its central stellar density, and thus making it susceptible to tidal destruction during galaxy merging. Moreover, this mechanism provides a pathway to explain the observed reduction in the nucleus-to-galaxy stellar mass ratio as one proceeds from dwarf to giant elliptical galaxies. Comment: 15 page, 5 figures, accepted in ApJL