> would an ML optimization loop have discovered transformers?

I think that’s exactly the kind of problem this group is looking to solve. You make a compelling intuitive argument, but that’s not the same thing as a proof