The large inland estate, which includes main and secondary houses and 4,000 lychee trees, served as a beloved multigenerational property