Danny Jordaan, South Africa's World Cup organising chief, said the vuvuzela might be banned from inside stadiums after complaints from broadcasters and supporters.
The constant sound of the horn-like instrument has so far drowned out much of the atmosphere-generating singing usually associated with soccer matches.
And Jordaan, when asked if he would get rid of them, told BBC Sport: "If there are grounds to do so, yes.
"We did say that if any land on the pitch in anger we will take action."
France captain Patrice Evra blamed the vuvuzelas for his side's poor performance in their group opener against Uruguay.
He said: "We can't sleep at night because of the vuvuzelas. People start playing them from 6am. We can't hear one another out on the pitch because of them."
Jordaan said the local organizers were doing all they could to minimise the impact.
"We've tried to get some order," he said. "We have asked for no vuvuzelas during national anthems or stadium announcements. It's difficult but we're trying to manage the best we can."