The Proteas reached 386-7 at stumps, still 14 runs behind Bangladesh's first-innings total of 400, with Ryan Rickelton unbeaten on 56 after resuming the day on 287-4.
Bavuma had looked comfortable alongside Rickelton as South Africa moved to 358-4, but his innings ended on 49 off 105 deliveries after he struggled with discomfort in his right calf.
The captain had received treatment on the field before eventually retiring hurt in the 84th over, having been reduced to hobbling between the wickets.
His departure was followed by a sudden collapse as Bangladesh made the most of the opening.
Taskin Ahmed struck in the next over, removing Wiaan Mulder for a duck after the all-rounder edged a wide delivery to wicketkeeper Litton Das. Marco Jansen followed for eight, also caught behind off Taskin.
Hasan Mahmud then dismissed Simon Harmer for one, with Litton completing his sixth catch of the innings.
The wicketkeeper's performance was particularly notable, having already scored a century earlier in Bangladesh's innings. He became only the third player in Test history to register a hundred and six dismissals in the same innings.
South Africa's innings had earlier received a boost from Rickelton, who capitalised on two dropped catches to reach his half-century. Mahfijul Islam put him down on 30, while Litton had also missed an opportunity when the left-hander was yet to score.
Rickelton was 56 not out at the close, with Kagiso Rabada on five, while Mahmud finished as Bangladesh's leading wicket-taker with 4-72. Taskin claimed 2-88.
Both sides also had injury concerns to contend with. Mahmud left the field early in the morning with a right elbow problem but returned wearing heavy strapping and went on to claim another wicket.
Cricket South Africa confirmed during the lunch interval that Bavuma was being assessed for discomfort in his right calf and was expected to return to bat.
However, that opportunity never came as lightning in the vicinity of the Wanderers initially delayed the resumption after lunch, before heavy rain and hail forced the abandonment of the remaining sessions.
With more than 90 minutes of scheduled play still remaining, the umpires eventually called stumps, leaving South Africa facing a 14-run deficit heading into the fourth day.