After regaining control of Anbar provincial capital Ramadi from the Islamic State group earlier this year, Iraqi forces have been advancing up the Euphrates Valley in recent weeks.
Officials are vowing to launch a final operation to retake Hit, a key hub along the Euphrates, in the coming days.
Police colonel Fadhel al-Nimrawi said thousands of families had recently fled Hit to Al-Baghdadi, a town to the northwest, and other locations in Anbar where displaced civilians are gathered.
"At least 120 families arrived in Al-Baghdadi yesterday but there are thousands of families still in there," he told AFP.
He said most of the civilians had gathered in the Jamaiya and Al-Omal neighbourhoods near the main market of Hit, a city that lies around 145 kilometres (90 miles) west of Baghdad.
Some IS fighters fled the town on Sunday and Saturday, including some top foreign leaders who had been based in Hit, according to several senior security officials.
"It is clear however that some of them remain, they are mostly deployed in defensive positions around the city," Nimrawi said.
Yahya Rasool, the spokesman for the Joint Operations Command coordinating the fight against IS in Iraq, said an operation in Hit would come soon.
"Unfortunately, Hit is still under Daesh control but it will be retaken in the coming days," he told AFP, using an Arabic acronym for IS.
Hit residents had hoped when IS fighters pulled out on Sunday that government forces and allied tribal fighters would swoop in to seal the recapture of the city.
But while some IS fighters left, others rotated in and civilians in Hit fear being trapped in the town for the final battle.
"So far, there are no security forces in Hit, the inhabitants are scared because they know there will be a big military operation," said Naim al-Kaoud, the leader of the Nimr tribe.
Over the weekend, IS fighters also abandoned the town of Rutba -- deep in the Anbar desert -- only to move back in 24 hours later, according to security officials.
They still control the town but their foreign leaders did not return and are believed to have moved towards Al-Qaim, a town on the border with Syria.
Iraqi forces, backed by a US-led coalition, have been battling to regain ground from IS since the jihadists seized control of large parts of Iraq and neighbouring Syria in mid-2014.
