'I don’t think the public fully comprehend quite how much influence Brown has,' says a senior Whitehall source