Ta$_4$Pd$_3$Te$_{16}$ is a recently discovered superconductor that has been suggested to be an unconventional superconductor near magnetism. We report electronic structure calculations showing that in spite of the layered crystal structure the material is an anisotropic three dimensional metal. The Fermi surface contains prominent one and two dimensional features, including nested 1D sheets, a 2D cylindrical section and a 3D sheet. The electronic states that make up the Fermi surface are mostly derived from Te $p$ states, with small Ta $d$ and Pd $d$ contributions. This places the compound far from magnetic instabilities. The results are discussed in terms of multiband superconductivity.