To shed new light on the mechanism of superconductivity in sodium cobalt oxide bilayer-hydrate (BLH), we perform a density functional calculation with full structure optimization for BLH and its related nonsuperconducting phase, monolayer hydrate (MLH). We find that these hydrates have similar band structures, but a notable difference can be seen in the $a_{1g}$ band around the Fermi level. While its dispersion in the $z$ direction is negligibly small for BLH, it is of the order of 0.1 eV for MLH. This result implies that the three dimensional feature of the $a_{1g}$ band may be the origin for the absence of superconductivity in MLH. ; Comment: 5 pages, 7 figures, to be published in Phys. Rev. B