Phonon transmission across interfaces of dissimilar materials has been studied intensively in recent years by using atomistic simulation tools due to its importance in determining the effective thermal conductivity of nanostructured materials. The atomistic Green's function method with interatomic force constants from the first-principles calculations has evolved to be a promising approach to study phonon transmission in many material systems that have not been well studied. However, the direct first-principles calculation for interatomic force constants becomes infeasible when the system involves atomic disorder. Mass approximation is usually used, but its validity has not been tested. In this paper, we employ the higher-order force constant model to extract harmonic force constants from the first-principles calculations, which originates from the virtual crystal approximation but considers the local force-field difference. As a feasibility demonstration of the proposed method that integrates higher-order force constant model from the first-principles calculations with the atomistic Green's function, we study the phonon transmission in the Mg2Si/Mg2Si1−xSnx systems. When integrated with the atomistic Green's function, the widely used mass approximation is found to overpredict phonon transmission across the Mg2Si/Mg2Sn interface. The difference can be attributed to the absence of local strain field-induced scattering in the mass approximation, which makes the high-frequency phonons less scattered. The frequency-dependent phonon transmission across an interface between a crystal and an alloy, which often appears in high-efficiency “nanoparticle in alloy” thermoelectric materials, is studied. The interfacial thermal resistance across Mg2Si/Mg2Si1−xSnx interface is found to be weakly dependent on the composition of Sn when the composition of x is less than 40% but increases rapidly when it is larger than 40% due to the transition of high-frequency phonon DOS in Mg2Si1−xSnx alloys. This work could have a great impact on the design of novel nanostructures with tunable thermal properties.