"No. 1, the Cavs are the biggest losers in this situation,” Perkins, who teamed with James on the Cavaliers and also in AAU basketball, said Saturday