The distorted Character Heads of the 18th-century Viennese sculptor have long perplexed critics – but they catch the mood of Vienna during the Enlightenment