Mertens had lost all five of her previous meetings with Gauff, but produced a composed display to claim her first victory over the fourth seed.
Gauff struggled to find her rhythm, committing 34 unforced errors and eight double faults. Mertens broke four times, including in the final game of the opening set, before moving 4-1 ahead in the second.
Gauff appeared increasingly drained and told her team she had no energy and felt as though she was going to pass out. Mertens remained composed, particularly after previously letting significant leads slip against Gauff.
The Belgian then won 10 of her final 11 service points before sealing the match with her second ace.
The victory is Mertens' 10th career win over a top-five opponent and her second of the season after beating Elena Rybakina at Wimbledon. It also takes her into her first China Open quarter-final in seven appearances.
Mertens had already battled through three-set matches against Katie Volynets and Jelena Ostapenko to reach the fourth round.
She will now face eighth seed Iga Swiatek, who defeated American teenager Iva Jovic 6-1, 6-3. Swiatek has won both of her previous meetings with Mertens.
Gauff's exit came after a difficult few days in Beijing. The 2024 champion had received an "insane" amount of racist abuse online following a controversial incident during her third-round win over home player Sun Xinran.
Gauff had stopped a point after noticing a flickering light around the court, with the chair umpire ordering it to be replayed. She later said the abuse had been draining and that she had dealt with it by switching off her phone and moving on.