Jack Conger and Gunnar Bentz were taken off a flight leaving Rio de Janeiro by authorities investigating doubts over their claim of having been mugged.
That came hours after top European Olympic official Patrick Hickey was arrested in an investigation into black market Olympic ticket sales.
Conger and Bentz along with star US swimmer Ryan Lochte and squad member James Feigen said they were held up at gunpoint in the early hours of Sunday.
However Brazilian judge Keyla Blank ordered the passports of all four to be seized to prevent them leaving the country. Lochte had already gone home.
The Globo media organisation posted a video online showing Conger and Bentz walking into a police station on Wednesday.
European Olympic official Patrick Hickey was arrested on Wednesday.
The G1 news site, which is part of Globo, reported that Bentz, 20, and Conger, 21, were detained for questioning and that Feigen, 26, was still in the country.
Bentz and Conger left the airport in the middle of the night in a car that passed by a crowd of journalists, according to an AFP reporter. They were scheduled to appear before Rio tourism police on Thursday.
The four made headlines with their terrifying account while returning from a party.
Lochte said they were held up by muggers posing as police as they left a party just after 4:00am. In interviews, he described how he had a hand gun put to his head and handed over his money and wallet.
Victims laughed
However, the judge said their story was full of inconsistencies.
US television channel NBC quoted witnesses as saying the swimmers were at the party until daybreak.
And Britain's Daily Mail obtained video surveillance showing the four laughing as they returned to the athletes' village at 6.56am on Sunday.
Hickey, an IOC executive member, head of the European Olympic Committees and the Irish national committee, spent Wednesday night in hospital after being detained at his luxury hotel.
Accused of ticket scalping, ambush marketing and conspiracy after tickets were seized from another Irish businessman, Hickey has "temporarily" stood down from his posts.
Following a raid similar to the arrest of seven top FIFA officials at a Zurich hotel last year, Hickey was shown in media video answering his hotel room door unclothed before putting on a bath robe.
Police said they suspect he had changed rooms to try to evade arrest.
Hickey's wife was in another room and at first told them Hickey had left the country, police told reporters.
Brazilian police have launched a widening investigation after seizing more than 1,000 premier tickets for Games events, including the opening ceremony.
Tickets with a face value of about $1,000 were sold for $8,000, in a racket prosecutors said yielded profits of $3.0mn.
The head of a sports ticketing firm, THG Sports, was detained with the tickets on August 5.
Hickey's son once worked for THG, but the sports powerbroker denied any wrongdoing in an interview with Irish television last week as the scandal mounted.
The arrests diverted attention from the sporting action as Usain Bolt reached the final of the 200m -- the second of his targeted three-gold-medal sweep of the Olympic sprint titles.
Gatlin fails
While Bolt ran the season's best 19.78 sec, his American arch-rival Justin Gatlin, 34, failed to reach the final.
Gatlin, who won 100m silver behind Bolt at the weekend, said he had not fully recovered from an ankle injury suffered in November.
"I definitely think I can try for the world record, I definitely feel that," Bolt said.
Drug-tainted Russia's only competitor in the track and field, Darya Klishina, failed to reach the final medal battle in the long jump. The 25-year-old blamed the turmoil of the doping scandal for her disappointment.
The IAAF gave her the green light for Rio and then banned her because of "new" doping information. She went to the sports tribunal to reverse the ban.
Brazil's suffering sports fans got some good news when football star Neymar scored twice -- including the fastest Olympic goal after 15 seconds -- in a 6-0 win over Honduras in the Olympic semi-final.
In the final they will take on Germany in a revenge repeat of their disastrous 7-1 loss in the World Cup semi-final two years ago.