The sources said eight of the 16 Cuban players had deserted, while team member Roger Aguilera put the number at seven, just the latest in a rash of Cuban defections across several sports.
"Everyone knows what happened to our team, we have seven of them in the United States," said Aguilera, after the decimated Cubans were hammered 13-0 by Trinidad and Tobago.
Short of manpower, Cuba could only field eight players instead of the standard 11 plus five substitutes.
They are not the first Cubans to defect during the Pan Am Games, after four rowers disappeared last week, including silver medalist Orlando Sotolongo.
It is not uncommon for Cuban athletes to defect, with baseball, boxing and volleyball -- sports where Cuban stars can command big salaries internationally -- particularly hard hit.
At the 1999 Pan American Games in Winnipeg, eight members of the Cuban delegation defected.
In January 2013, Cuba eliminated legal barriers to foreign travel, but that did not end the defections of artists and athletes, who remain susceptible to the lure of higher pay abroad.
In an attempt to stem the losses, Cuba raised salaries of athletes, and began allowing them to sign contracts with foreign clubs.
But the defections have continued, including at this month's football Gold Cup, despite a historic thaw in ties between communist Cuba and the United States, AFP reported.