"It's actually terrorism, and whether it's domestic or worldwide terrorism, with the moral authority of the presidency, you have to call that stuff out", Scaramucci added.
In his first interview since being deposed as White House Communications Director after 10 days on the job, Scaramucci appeared on ABC's "This Week With George Stephanopoulos" and started by being critical of President Donald Trump's response to the terrorist attack in Charlottesville on Saturday.
"He likes doing the opposite of what the media thinks he's going to do", Scaramucci said.
"I think the president knows what he's going to do with Steve Bannon". "I think he needed to be much harsher as it related to the white supremacists and the nature of that", Scaramucci told George Stephanopoulos, referring to his former boss's broad and vague condemnation of hatred and violence.
Scaramucci acknowledged that he made a mistake and was sacked for it by giving a profanity-laced interview to The New Yorker's Ryan Lizza during his short time working at the White House.
"The fact that [Steve Bannon] is sitting there inside the White House probably makes people think that [Breitbart] has an influence, and I think that's reason enough to probably make some changes".
He added that it's not serving the President's interests to still have that wing around. Bannon was formerly with the alt-right Breitbart News.
"He's got to move more to the mainstream", Scaramucci said.
Studies have revealed the President's path to victory past year was paved in the most part by four groups, including free marketeers, anti-elites, American preservationists, and staunch conservatives.
In the interview Mr Scaramucci also criticised his former boss' handling of violent protests involving white supremacists in Charlottesville in Virginia. The Washington Post reported that the move was at the request of new White House Chief of Staff John F. Kelly.
"Can you and Steve Bannon still work together in this White House or not?"
He called Lizza's decision to record the call " a very deceitful thing that he did".
He admitted he "absolutely" made mistakes but said "I went in there with my heart and soul".