The appearance of the detective in real life and in fiction seem to me to be entwined. The autobiography of the great French thief taker Eugene Francois Vidocq seems to have been an influence on both the establishment of the first detective unit in Britain and in Edgar Allen Poe's creation of what many argue is the first literary detective, Auguste Dupin in Murders in the Rue Morgue, published in 1841. The first detective department in Britain was created in 1843.
Detectives quickly started popping up everywhere – Dickens was a big fan of the real life and fictional detectives, and packed his monthly magazine All the Year Round with fictions and features on the new agents of order.
Something we crave from genre fiction, I think, is a validation of human agency. In perhaps the grandest genre of them all – the classical tragedy – the goal of the exercise is to show a man brought down by hubris, that is the man that challenges the world, that refuses to accept things as they are and takes his life in his own hands. It is this exercise of agency that will eventually lead to his destruction.
A crime story is usually a tragedy, too, with the same causes as the classical tragedies – envy, desire, greed, and vengeance – but the detective story introduces a counterpoint to the theme of the protagonist brought down by their own arrogance. I suppose it's a non-supernatural incarnation of Nemesis (no, not that one) pursuing the evil doer to their comeuppance, but they also form a force of order. Among other forms of transgression, a crime is a the breaking of taboo, and the detective is the one that ferrets out the source of the disturbance of the natural order and sets things to rights.
The Murder at Roadhill House features a lot of taboo breaking – relationships between the classes, the possible breaking of the marriage vow, the family bond. They are the sorts of tensions that might have bubbled up into expressions of supernatural as we see in historical cases of demonic possession or folk tales of ghosts and hauntings. The detective is a kind of rootless, stateless type able to cross the barriers imposed by class and propriety to ask the questions people would rather not address.
I suppose it's no surprise that these genres have mixed together in the form of the paranormal noir that is so popular these days. The breaking of taboo is extended into the metaphorical realm as it always has been in folklore, but we impose the same taboo-fixing powers to detective, now an almost blatantly shamanic figure.