While in-person sales have stalled, the art world tries its hand at e-commerce—but not without trial and error