The architecture changes and systems improvements being brought into LLMs is so awesome to see. It really feels like this is now a systems problem where a defined goal is set then systems optimizations are made around the model architecture to solve it.

Underlying it all is that any architecture can be trained to the same convergence just difference in compute utilization both in training and inference