Aside from the obvious, what makes this mysterious man so irresistible? Some of his female costars – and paramours – weigh in