From mermen and rabbit-men to body horror and alternate realities, these literary novels prove the genre can get very weird, very fast.