Swimmers hit out at ‘restrictive’ measures they say are changing who can enjoy the popular London swimming spot