Context. Planets have been detected around a variety of stars, including low-mass objects, such as brown dwarfs. However, such extreme cases are challenging for planet formation models. Recent sub-millimeter observations of disks around brown dwarf measured low spectral indices of the continuum emission that suggest that dust grains grow to mm-sizes even in these very low mass environments. Aims. To understand the first steps of planet formation in scaled-down versions of T-Tauri disks, we investigate the physical conditions that can theoretically explain the growth from interstellar dust to millimeter-sized grains in disks around brown dwarf. Methods. We modeled the evolution of dust particles under conditions of low-mass disks around brown dwarfs. We used coagulation, fragmentation, and disk-structure models to simulate the evolution of dust, with zero and non-zero radial drift. For the non-zero radial drift, we considered strong inhomogeneities in the gas surface density profile that mimic long-lived pressure bumps in the disk. We studied different scenarios that could lead to an agreement between theoretical models and the spectral slope found by millimeter observations. Results. We find that fragmentation is less likely and rapid inward drift is more significant for particles in brown dwarf disks than in T-Tauri disks. We present different scenarios that can nevertheless explain millimeter-sized grains. As an example, a model that combines the following parameters can fit the millimeter fluxes measured for brown dwarf disks: strong pressure inhomogeneities of ~40% of amplitude, a small radial extent ~15 AU, a moderate turbulence strength α_(turb) = 10^(-3), and average fragmentation velocities for ices v_f = 10 m s^(-1).