The former foreign secretary has told Kemi Badenoch he will leave her front bench.