Sir Joseph Ayloffe, Baronet, resided at Framfield where he was born and where he built a small mansion upon his estate called Wharton's. He received his preliminary education at Westminster school and was admitted to Lincoln's Inn in 1724. In 1731 he became Fellow of the Society of Antiquaries and subsequently a Vice-President and he was also a leading member of the Royal Society. He was involved in the building of Westminster Bridge and published and revised many important works of literature.