Gap between those who would rejoin and those who want to stay out is widening