The manosphere influencer has lost the president's support — but he could pose a threat to Barron