Critics accused the actor of promoting a 'privileged perspective' while supporters said he was making 'too much sense'