The US President has previously called the Prime Minister 'extremely liberal' and dismissed him as the 'mayor of a town'