The mutualism of ants and extrafloral nectary (EFN)-bearing plants is known to reduce rates of herbivory. However, ants may have negative impacts on other mutualisms such as pollination, constituting an indirect cost of a facultative mutualism. For instance, when foraging on or close to reproductive plant parts ants might attack pollinators or inhibit their visits. We tested the hypothesis that ants on EFN-bearing plants may negatively influence pollinator behavior, ultimately reducing plant fitness (fruit set). The study was done in a reserve at Brazilian savannah using the EFN-bearing plant Banisteriopsis malifolia (Malpighiaceae). The experimental manipulation was carried out with four groups: control (free visitation of ants), without ants (ant-free branches), artificial ants (isolated branches with artificial ants on flowers) and plastic circles (isolated branches with plastic circles on flowers). We made observations on flower visitors and their interactions, and measured fruit formation as a proxy for plant fitness. Our results showed that pollinators hesitated to visit flowers with artificial ants, negatively affecting pollination, but did not hesitate to visit flowers with plastic circles, suggesting that they recognize the specific morphology of the ants. Pollinators spent more time per flower on the ant-free branches, and the fruiting rate was lower in the group with artificial ants. Our results confirm an indirect cost in this facultative mutualism, where the balance between these negative and positive effects of ants on EFN-bearing plants are not well known.