MPs of the Economic Freedom Fighters (EFF) were kicked out of Parliament during a joint sitting, Parliament sources confirmed on Thursday.
This came after the session was plunged into chaos on Wednesday night during a Presidency budget vote speech by President Jacob Zuma, the sources said on condition of anonymity.
EFF MPs requested for a debate on the removal of Zuma, but the request was rejected by Mbete.
It was the president's first appearance in Parliament since the Constitutional Court ruled in late March that he had failed to uphold the Constitution regarding upgrades to his Nkandla homestead, a project that allegedly cost public funding worth 246 million rand (about 17 million U.S. dollars).
The EFF MPs refused to let Zuma speak, plunged Parliament into chaos.
Under Mbete's order, the EFF MPs were forcibly removed from Parliament.
During the session, MPs of the ruling African National Congress (ANC) rallied behind Zuma.
ANC chief whip Jackson Mthembu warned EFF leader Julius Malema and his supporters that the fate of the right-wing Boeremag would soon be theirs.
Mthembu likened the actions of EFF leader Julius Malema, who recently said in a televised interview that his party was prepared to remove the current government "through the barrel of a gun", to those of the Boeremag, the right-wing group which plotted to overthrow the government.
Parliament sessions have previoulsy been plunged into chaos by EFF MPs who insist that Zuma be held accountable for alleged corruption.