The episode took a sharp political turn when presumptive Republican presidential nominee Mitt Romney said he would fire the agents involved.
Romney told radio host Laura Ingraham on Wednesday that "I'd clean house" at the Secret Service.
"The right thing to do is to remove people who have violated the public trust and have put their play time and their personal interests ahead of the interests of the nation," Romney said.
While Romney suggested to Ingraham that a leadership problem led to the scandal, he told a Columbus, Ohio, radio station earlier that he has confidence in Sullivan, the head of the agency.
"I believe the right corrective action will be taken there and obviously everyone is very, very disappointed," Romney said. "I think it will be dealt with (in) as aggressive a way as is possible given the requirements of the law."
When asked, the Romney campaign would not say whether he had been briefed on the situation or was relying upon media reports for details.
At least 10 military personnel who were staying at the same hotel are also being investigated for misconduct.
Two U.S. military officials have said they include five Army Green Berets. One of the officials said the group also includes two Navy Explosive Ordinance Disposal technicians, two Marine dog handlers and an Air Force airman. The officials spoke on condition of anonymity because the investigation is still under way.
Secret Service's Office of Professional Responsibility, which handles that agency's internal affairs, is investigating, and the Homeland Security Department's inspector general also has been notified.
Sullivan, who this week has briefed lawmakers behind closed doors, said he has referred to the case to an independent government investigator.
Secret Service investigators have interviewed all of the hotel's maids and cleaning ladies as part of its investigation, according to a person familiar with the investigation. The person, who spoke on the condition of anonymity because the person was not authorized to speak publicly about the ongoing probe, said investigators have not found any drugs or drug paraphernalia in the agents' rooms.
King said the agency was "reasonably confident" that drug use was not an issue with the three agents forced out on Wednesday. But he said Secret Service investigators would continue to look into whether drugs played a role in the incident as it continues talking to the other eight agents involved.
"Everything is on the table," the congressman said.
Col. Scott Malcom, a spokesman of U.S. Southern Command, which organized the military team assigned to support the Secret Service's mission in Cartagena, said Wednesday that an Air Force colonel is leading the military investigation and arrived in Colombia with a military lawyer Tuesday morning.
The troops are suspected of violating curfews set by their commanders.
"They were either not in their room or they showed up to their room late while all this was going on or they were in their room with somebody who shouldn't be there," Malcom said.
Lawmakers have called for a thorough investigation and have suggested they would hold oversight hearings, though none has yet been scheduled. The incident is expected to come up next week on Wednesday when Homeland Security Secretary Janet Napolitano appears before the Senate Judiciary Committee for a previously scheduled oversight hearing.
House Speaker John Boehner, R-Ohio, said that for now, he is interested in what actually happened. He did not address how much responsibility Obama should bear for the scandal or whether Congress should hold hearings on it.