The agency's growing unpopularity may make it easier to abolish it.