In Arabidopsis, multisubunit RNA polymerases IV and V orchestrate RNA-directed DNA methylation (RdDM) and transcriptional silencing, but what identifies the loci to be silenced is unclear. We show that heritable silent locus identity at a specific subset of RdDM targets requires HISTONE DEACETYLASE 6 (HDA6) acting upstream of Pol IV recruitment and siRNA biogenesis. At these loci, epigenetic memory conferring silent locus identity is erased in hda6 mutants such that restoration of HDA6 activity cannot restore siRNA biogenesis or silencing. Silent locus identity is similarly lost in mutants for the cytosine maintenance methyltransferase, MET1. By contrast, pol IV or pol V mutants disrupt silencing without erasing silent locus identity, allowing restoration of Pol IV or Pol V function to restore silencing. Collectively, these observations indicate that silent locus specification and silencing are separable steps that together account for epigenetic inheritance of the silenced state.