> Can't we have a system that is optimized for the notes that are actually played in a song rather than the hypothetical set? And what if the optimization is done per note rather than over an entire song?
You can. It’s called adaptive tuning, or dynamic just intonation, and it happens naturally for singers with no accompanying instruments.
It’s impractical on a real instrument, but there’s a commercial synthesiser implementation called hermode tuning.
You’re trading one problem for another, though. No matter how you do this, you will either have occasional mis-tuning or else your notes will drift.
In addition to singers, adaptive tuning is something which happens naturally for fretless stringed instruments (violin, etc), brass instruments with slides (most prominently the slide trombone but in fact many (most?) others), woodwind instruments where the pitch can be bent like saxophone, and so on.
I used to play fretless bass in a garage hip hop troupe that played with heavily manipulated samples that were all over the place in terms of tuning instead of locked to A440, forcing adaptations like "this section is a minor chord a little above C#".
Adaptive tuning is hard to do on a guitar because the frets are fixed. String bending doesn't help much because the biggest issue is that major thirds are too wide in equal temperament and string bending the third makes pitch go up and exacerbates the problem.
You can do a teeny little bit using lateral pressure (along the string) to move something flat. It's very difficult to make adaptations in chords though. A studio musician trick is to retune the guitar slightly for certain sections, though this can screw with everybody else in the ensemble.
Attempts to experiment with temperament using squiggly frets make it clear how challenging this problem is: https://stringjoy.com/true-temperament-frets-explained/
Played trombone many years ago, but never well enough to ever adjust that finely (at least not consciously?). The tuning slide on the third valve on a trumpet usually has a finger fork/loop so that it can be tuned in realtime. I believe the first valve on higher end trumpets similarly has a thumb fork for the same reason.
I played trombone in high school, never very well, but I definitely adjusted like this. Actually, although it was a slide trombone, I'm talking about adjusting automatically with embouchure. Someone would play the reference note, I'd match (in 1st position) but bend my pitch to match. The band teacher once complimented me on the adjustment. Which was stupid, because (1) I wasn't doing it intentionally, and (2) the adjustment only lasted during tuning; as soon as we started playing, I was right back out of tune. I never did learn to suppress the adjustment so I could actually fix the tuning.
But with the way I played, I'm not even sure how much it mattered. The best tool for enhancing my playing would've been a mute. (And it would have been most effective lodged in my windpipe.)