Dibutyl phthalate (DBP) produced by Streptomyces sp. H11809 exerted inhibitory activity against human GSK-3β (Hs GSK-3β) and Plasmodiumfalciparum 3D7 (Pf 3D7) malaria parasites. The current study aimed to determine DBP’s plausible mode of action against Hs GSK-3β and Pf 3D7. Molecular docking analysis indicated that DBP has a higher binding affinity to the substrate-binding site (pocket 2; −6.9 kcal/mol) than the ATP-binding site (pocket 1; −6.1 kcal/mol) of Hs GSK-3β. It was suggested that the esters of DBP play a pivotal role in the inhibition of Hs GSK-3β through the formation of hydrogen bonds with Arg96/Glu97 amino acid residues in pocket 2. Subsequently, an in vitro Hs GSK-3β enzymatic assay revealed that DBP inhibits the activity of Hs GSK-3β via mixed inhibition inhibitory mechanisms, with a moderate IC50 of 2.0 µM. Furthermore, the decrease in Km value with an increasing DBP concentration suggested that DBP favors binding on free Hs GSK-3β over its substrate-bound state. However, the antimalarial mode of action of DBP remains unknown since the generation of a Pf 3D7 DBP-resistant clone was not successful. Thus, the molecular target of DBP might be indispensable for Pf survival. We also identified nocardamine as another active compound from Streptomyces sp. H11809 chloroform extract. It showed potent antimalarial activity with an IC50 of 1.5 μM, which is ~10-fold more potent than DBP, but with no effect on Hs GSK-3β. The addition of ≥12.5 µM ferric ions into the Pf culture reduced nocardamine antimalarial activity by 90% under in vitro settings. Hence, the iron-chelating ability of nocardamine was shown to starve the parasites from their iron source, eventually inhibiting their growth.