Designed by Martyn Lawrence Bullard, the Mediterranean-style manse is an ideal backdrop for a collection inherited from Rivers’s late mother, Joan