The evacuation comes after a deal struck by President Bashar al-Assad's government and opposition forces in the town, which is close to the capital Damascus and was one of the first to rise up against his regime.
On Friday morning, ambulances and Syrian Red Crescent vehicles entered the devastated town, which has been surrounded by loyalist troops since 2012 and suffered constant regime bombardment.
Inside the town, residents bid farewell to each other, as well as their homes and neighbourhoods, a local rebel fighter told AFP.
"This is the hardest moment, everyone is crying, young and old," he said, speaking on condition of anonymity.
The deal to evacuate the town was announced on Thursday by Syrian state news agency SANA.
"Seven hundred armed men with their personal weapons will leave Daraya to head to the (rebel-controlled) city of Idlib, while thousands of men and women with their families will be taken to reception centres," SANA said.
The rebels would have to surrender other armaments to the army.
The rebels who have controlled the town belong to two Islamist groups: Ajnad al-Sham and the Martyrs of Islam.
A Syrian source on the ground said Thursday the evacuation could take four days to complete and a military source said the army would enter Daraya.
"The civilians will go to regions under regime control around Damascus, rebels will go to Idlib "or sort out their situation with the regime", a rebel official told AFP.
Weeping residents bid farewell
Daraya's local council said on its official Facebook page that civilians would be taken to the government-held town of Hrajela in Western Ghouta, outside the capital.
"From there they will continue to the areas they wish to go to," it said.
The council said that fighters and their families would be taken to northern Syria, escorted by the Red Crescent.
Daraya is located just 15 minutes drive from the capital and lies even closer to the government's vital Mazzeh air base.
It was one of the first towns to erupt in anti-government protests in March 2011 and the evacuation has provoked anger and bitterness among opposition supporters.
The local rebel fighter said residents were weeping as they prepared to leave.
"People are saying goodbye to one another, children are bidding their schools farewell, mothers are saying goodbye to the martyrs in the graves," he said.
"People are gathering their memories and the few possessions they have left to preserve the memory of the four years of siege, hunger and shelling, and as a reminder to the international community that let them down without any guilt," he said.
Daraya 'completely destroyed'
He said the decision to evacuate the town had been taken because of deteriorating humanitarian conditions.
"The town is no longer habitable, it has been completely destroyed," he said.
Daraya has been seen as a symbolic bastion of the uprising that began with peaceful protests against Assad's government, before devolving into a war that has killed over 290,000 people.
Just one food aid convoy entered the town since late 2012, in June, shortly after a single convoy carrying medicine entered.
The arrival of the food aid was followed by heavy regime bombardment that residents said stalled distribution.
According to the United Nations, nearly 600,000 live under siege across Syria, most surrounded by government forces, although rebels and jihadists also use the tactic.
In several places, lengthy government sieges have prompted rebels to agree evacuation deals with the regime, prompting activists to accuse Damascus of using "starve or surrender" tactics.
In Geneva on Friday, US Secretary of State John Kerry and Russian Foreign Minister Sergei Lavrov met to discuss efforts to resume Syrian peace talks.
The conflict has been further complicated by a Turkish military incursion into northern Syria this week to fight the Islamic State jihadist group and halt advancing Kurdish fighters.