Women and equalities secretary Bridget Phillipson has rejected the invitation from her Tory shadow Joy Morrissey accusing the opposition of ‘political point scoring’