Graphene nanoplate (GNP) is a two-dimensional plate-like carbon material with high aspect ratio and excellent electrical conductivity. It is one of the most commonly used fillers for conductive polymer composites (CPCs), which have potential applications in flexible electrodes and sensors. The electrical properties of the CPCs particularly depend on the microstructure of GNP networks. The electrical conductivity of the CPCs leaps in several magnitude levels when the graphene concentration reaches a critical value, which is defined as the percolation threshold. For ordinary isotropic CPCs, the percolation threshold is relatively high, which leads to inferior performance with poor mechanical and electrical properties. Aligning the graphene plates is an effective method to reduce the percolation threshold of the CPCs. Carbonyl iron particles (CIPs) are easily aligned to form chain-like structures when a magnetic field is applied. In this work, CIPs and GNPs are mixed with polydimethylsiloxane (PDMS), and the hybrid is cured under a magnetic field of 0.5 T. The alignment of CIPs induces the GNPs in the PDMS to orientate in a certain direction under the applied magnetic field generating anisotropic structures. Both isotropic and anisotropic structured GNPs/PDMS composites are prepared with various GNP concentrations. The microstructure and electrical conductivity of the GNPs/PDMS composites are investigated by experimental methods. It is found that anisotropic graphene networks are formed and the percolation threshold of the anisotropic composites is 0.15 vol%, compared to that of the isotropic composites which is 0.85 vol%. The alignment of GNPs significantly reduces the percolation threshold. Furthermore, a plate lattice model is proposed to reveal the effect of the alignment of GNPs on the formation of conductive networks. With the increase of the alignment degree of GNPs, the percolation threshold decreases significantly, which is consistent with the experimental results.