The art, fearfully esoteric and influenced by its creator’s séances and spiritualism, matches a present mood of restless searching, Peter Schjeldahl writes.