We reconsider the origin and processing of dust in elliptical galaxies. We theoretically formulate the evolution of grain size distribution, taking into account dust supply from asymptotic giant branch (AGB) stars and dust destruction by sputtering in the hot interstellar medium (ISM), whose temperature evolution is treated by including two cooling paths: gas emission and dust emission (i.e. gas cooling and dust cooling). With our new full treatment of grain size distribution, we confirm that dust destruction by sputtering is too efficient to explain the observed dust abundance even if AGB stars continue to supply dust grains, and that, except for the case where the initial dust-to-gas ratio in the hot gas is as high as $∼ 0.01$, dust cooling is negligible compared with gas cooling. However, we show that, contrary to previous expectations, cooling does not help to protect the dust; rather, the sputtering efficiency is raised by the gas compression as a result of cooling. We additionally consider grain growth after the gas cools down. Dust growth by the accretion of gas-phase metals in the cold medium increase the dust-to-gas ratio up to $∼ 10^{-3}$ if this process lasts >10/(n_H/10^3 cm^{-3}) Myr, where $n_\mathrm{H}$ is the number density of hydrogen nuclei. We show that the accretion of gas-phase metals is a viable mechanism of increasing the dust abundance in elliptical galaxies to a level consistent with observations, and that the steepness of observed extinction curves is better explained with grain growth by accretion. ; Comment: 15 pages, 11 figures, accepted for publication in MNRAS