Solid electrolyte based-resistive memories have been considered to be a potential candidate for future information technology with applications in non-volatile memory, logic circuits and neuromorphic computing. A conductive filament model has been generally accepted to be the underlying mechanism for the resistive switching. However, the growth dynamics of such conductive filaments is still not fully understood. Here, we explore the controllability of filament growth by correlating observations of the filament growth with the electric field distribution and several other factors. The filament growth behavior has been recorded using in situ transmission electron microscopy. By studying the real-time recorded filament growth behavior and morphologies, we have been able to simulate the electric field distribution in accordance with our observations. Other factors have also been shown to affect the filament growth, such as Joule heating and electrolyte infrastructure. This work provides insight into the controllable growth of conductive filaments and will help guide research into further functionalities of nanoionic resistive memories.