It's not terribly far from the truth. IRC is a pretty thin layer on top of a generic TCP server. On your classic ircds the notion of users and channels is about the minimum possible, and everything's tightly coupled to the network and DNS. Single-core servers back in the day could and did handle 10,000 active users.
Bandwidth was most commonly the issue. Monash University in Australia ran a part time EFnet server back in the mid 90s. It had to be turned off during the day because it would saturate the university's internet connection (and this is even before DoS's became all the rage).