Residents have accused the council of taking "no meaningful action" against the travellers who are "living illegally in plain sight".