On a recent trip to London, I tempted fate and dragged my children along for a visit to the sumptous Georgian mansion that is Jasper Conran's flagship