‘She has got to keep control of a certain person,’ former judge quipped, referencing Paul Hollywood