Highly sensitive and reproducible surface-enhanced Raman scattering (SERS) substrates are the main challenge for practical applications. In this work, an ordered and hierarchical Ag nanoparticles (Ag-NPs)-decorated Au nanotubes (Au-NTs) array was achieved based on a funnel-shaped pore anodic aluminum oxide (AAO) template-assisted strategy. First, funnel-pore-AAO templates were fabricated by further oxidation of conical-pore-AAO templates achieved by multistep anodization and etching. Then physical sputtering was used to assemble the Au-NTs and Ag-NPs using the as-prepared funnel-pore-AAO as sacrificial templates. SEM revealed abundant sub-10 nm neighboring gaps and sub-10 nm nanocavities at the bottom of the nanotubes because of the special shape of the AAO template, which resulted in abundant strong “hot spots” contributing to the sensitive SERS detection. The resultant hierarchical substrates manifested a SERS enhancement factor of 1.8 × 107 and reproducible response to 10−11 M rhodamine 6G and 10−8 M methyl parathion, showing potential in SERS-based rapid detection of trace pollutants in the environment.