> Maybe the motherboard? Could that have a speaker built into it? That must be terrible for acoustics, but maybe useful for a little beep when something is wrong?

Yes, it was called the PC Speaker, and that's pretty much exactly what it was used for. https://en.wikipedia.org/wiki/PC_speaker

It was standard equipment through the (mid?) 90s, and completely independent of the (optional) PCM sound card.

Now PCM sound is built in to motherboards and the PC Speaker long ago faded into irrelevance. Modern motherboards don't even have headers to connect a PC Speaker. Some motherboards will emulate the PC Speaker over the built in sound output, but of course you need speakers plugged in and on to hear those beeps.