It does not matter, X11 is also not a display server it is a protocol. That distinction changes nothing.

For what it's worth Xorg is the display server, before that it was XFree86, there was also Xsgi and Xsun and for mac Xquartz, I even saw a neat project once where the Xserver was in javascript, I am trying to find it again but our modern search engines are a bit shit. The point being all of these can interoperate with one another.

  Around computers it is difficult to find the correct unit of time to
  measure progress.  Some cathedrals took a century to complete.  Can you
  imagine the grandeur and scope of a program that would take as long?
                -- Epigrams in Programming, ACM SIGPLAN Sept. 1982

We have our cathedrals, only we are such barbarians as to call them worthless garbage and unfit for purpose and do our best to tear our legacy down.