Joseph Harker: From Stuart Hall to north Wales, the issue won't go away. If you think I'm being ridiculous, read to the end of my argument