Well, one can argue that Metafont fonts are "niche". No font library supports them. But from what I know they could easily be technically superior to, say, Type 1 fonts. Of course a technology will be niche if it is treated as a poor relative. XSLT could be an alternative to CSS, for example. It is definitely more powerful than CSS, because it actually transforms the document, not just alters the appearance and sprinkles some automatic content here and there. And is actually used this way in XSL-FO, which, I think, powers a substantial share of technical publishing.
XSLT as an alternative to CSS sounds like a nightmare, though to be fair, I felt the same way about XSLT as an alternative to DSSSL, which to me felt more like a satirical response to the "XML everywhere" zeitgeist in the spirit of INTERCAL and eating babies than a serious design proposal.