The controversial Love Is Blind UK star has set the record straight following viewer backlash