Tories face an uphill struggle despite now being seen as the main opposition to Labour