Griveaux, 41, said he opted for retreat to protect his family after a website published the video with messages addressed to a woman, which was said to have come from the former government spokesman.
It then spread on social media.
"A website and social networks have launched vile attacks concerning my private life. My family does not deserve this. No one should ever be subjected to such abuse," Griveaux said at AFP headquarters in Paris.
"For more than a year, my family and I have been subjected to defamatory remarks, lies, rumours, anonymous attacks, the revelation of stolen private conversations and death threats," he said.
"As if this was not enough, yesterday a new level was reached," said Griveaux, adding he was not willing to expose his family any further.
"This is going too far."
The ambitious Griveaux helped found the Republic on the Move (LREM) party, but recent polls showed him trailing in third place in Paris, which Macron won comfortably in the 2017 presidential election.
Macron tapped Griveaux to be the LREM's Paris mayoral candidate, causing a rift in the party that led to the president asking rival Cedric Villani to step aside.
Villani refused, triggering his exclusion from the ruling party.
Wresting control of the capital would be a huge prize for Macron as he seeks to consolidate his base ahead of an expected re-election bid in 2022.
It would also help offset heavy losses he is expected to face in rural areas, where his policies have met fierce resistance and fired up the "yellow vest" protest movement.
One of Macron's closest political allies, Griveaux resigned as a junior minister and government spokesman in March last year to run for mayor.