> X11 is also not a display server it is a protocol

With a reference implementation. Xsgi, Xsun, Xquartz, and all the rest used the reference implementation with platform driver glue.