Bake Off fans were 'gutted' as one star had to leave the tent for good