We study the Casimir force on a single surface immersed in an inhomogeneous medium. Specifically we study the vacuum fluctuations of a scalar field with a spatially varying squared mass, $m^{2}+λΔ(x-a)+V(x)$, where $V$ is a smooth potential and $Δ(x)$ is a unit-area function sharply peaked around $x=0$. $Δ(x-a)$ represents a semi-penetrable thin plate placed at $x=a$. In the limits $\{Δ(x-a)\toδ(x-a), λ\to∞ \}$ the scalar field obeys a Dirichlet boundary condition, $ϕ=0$, at $x=a$. We formulate the problem in general and solve it in several approximations and specific cases. In all the cases we have studied we find that the Casimir force on the plate points in the direction opposite to the force on the quanta of $ϕ$: it pushes the plate toward higher potential, hence our use of the term buoyancy. We investigate Casimir buoyancy for weak, reflectionless, or smooth $V(x)$, and for several explicitly solvable examples. In the semiclassical approximation, which seems to be quite useful and accurate, the Casimir buoyancy is a local function of $V(a)$. We extend our analysis to the analogous problem in $n$-dimensions with $n-1$ translational symmetries, where Casimir divergences become more severe. We also extend the analysis to non-zero temperatures. ; Comment: 32 pages, 8 figures. Published version