Residents in Piddington voted in a symbolic referendum after the Government announced plans to convert a former military base into an asylum seeker camp.