Perhaps the strangest such novel is Harry Stephen Keeler's The Murdered Mathematician. Keeler (1890 - 1967) was an eccentric novelist many of whose (decidedly unusual) works are, wonderfully, available from Ramble House Press (http://www.ramblehouse.com/) whose service is excellent. There is an entertaining Wikipedia article.
In The Murdered Mathematician the victim is an eccentric professor, "Radical Luke" (whose radicalness is exemplified by his refusal to use Greek letters in doing mathematics). The murder is solved by Quiribus Brown, who is 7 foot 6 tall, and has been taught higher mathematics by his father. The book contains an exam paper of Radical Luke's, and Brown uses some fairly sophisticated mathematics to identify the murderer. It is certainly unlike any other book I have ever read! I'm now reading the further adventures of Quiribus Brown in The Case of the Flying Hands.