That's always my problem with these huge lists. I feel like any recommendation is so much more useful if it's accompanied by at least a bit of context about why it's being recommended. Whenever I ask for recommendations on reddit or elsewhere, I prefer 2 or 3 answers with information over one of the 20 item list dumps. This site does actually have some snippets of context, you just have to basically click on one (I guess at random) to get to them. Some categorization or other discovery feature might help greatly I think.

Anyway, I will second the other reply's recommendation of Borges also with the caveat that he didn't write novels. His short stories are phenomenal. They are also a bit more philosophical than "mathematical", but some of them (like The Library of Babylon) deal a bit with the intersection of mathematical ideas (combinatorics and infinity in that one's case) with the way that playing with them would impact people and society.

The best advice for reading him would be, assuming you are reading in English, to pick up Penguin's Collected Fictions edition, containing a good translation of a lot of his short stories (and poetry as well). Skip past the first collection of short stories ("A Universal History of Iniquity") as it's pretty good but not his best. Start with the collection called "Fictions". Its first story is Tlön, Uqbar, Orbis Tertius, which gives you a good taste of how he writes.