Ok, digging into the schematic, I think I see the problem.

Almost all of the normally free GPIO is eaten up by a DPI (Display Parallel Interface) connection to the screen.

The screen should instead use the currently unused MIPI pins if you're already at the level of sophistication of laying out PCIe and USB3 traces.

That gives you back nearly all of your GPIO to use for stuff like I2S, and you can then even expose more non USB externally than just that one stemma port.

On top of that, for a project like this, I would disagree with the quality you get out of the pwm as audio thing. It's not audiophile by any means, but neither is some cheap Bluetooth receiver chip, and it's certainly good enough for the speakers that weren't designed for this case.