Ulster started superbly as captain Nick Timoney crossed for an early try after several penalty advantages, with Nathan Doak converting.
But Montpellier quickly hit back: Zac Ward missed a tackle, allowing Auguste Cadot to release Gabriel N'Gandebe, who scored, and Domingo Miotti levelled the game from the tee.
Mistakes continued to haunt Ulster, including a botched line-out and a lost ball over the line, giving Montpellier space to surge ahead through Taofifenua.
Cormac Izuchukwu brought Ulster back within reach with a well-taken try in the 27th minute, but Billy Vunipola and Alex Becognee pushed Montpellier's lead to 26-12 by half-time.
The French side's dominance grew in the second half: Taofifenua acrobatically scored in the corner, followed by Lenni Nouchi powering over, putting the game out of reach.
Ulster's Robert Baloucoune scored his first try since the Six Nations, offering a brief glimmer of hope, but Montpellier quickly responded with Erdocio and Hounkpatin crossing in succession.
Ulster continued to fight, with Michael Lowry scoring a late consolation try, but the French side's control never wavered.
Arthur Vincent sealed the emphatic victory with a try in the final minutes as Montpellier lifted the Challenge Cup for a third time.
The Irish province ended a brutal eight days on a low note, after also missing out on a spot in the URC play-offs by the narrowest of margins.