Some stars made no secret of their dislike of the convicted rapist over the past 20 years