The area where Princess Kate grew up could be destroyed by Angela Rayner's plans to build more homes, it has been claimed.