Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
Evidence for g in LLMs: https://en.wikipedia.org/wiki/G_factor_(psychometrics)
Unsure, or as LLMs say, I’d gently push back on that. In my opinion, the jaggedness has increased. While this type of capability has improved, I still see longer running Astra agents making the assumptions that any employable person at any level wouldn't make. Also, generally, all models feel more robotic and less pleasant to talk to (subjective) while still lacking judgement.
So I have a hunch about this: I think it makes sense that they become less pleasant as they become more capable, because LLMs inherently do not have feelings, or any faculties/organs that could generate them. The pleasant tone was just an artificially encouraged artifact of the fact that LLMs approximate reasoning probabilistically, with mimicry. Of course, if you mimic things thoroughly enough, with sufficient granularity, you start to approximate the real thing and become robot-like, but maybe then there's also less room for more pleasant disclosure that was the result of less rigid mimicry.
So it almost seems like a situation where you cannot really have your cake and eat it, at least not in the way it currently works.
I am very interested in benchmark correlations.
Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.
I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.
Might want to reach out to the author of the above benchmark. I think having a working group of multiple distinct benchmark owners might help quickly evaluate new model developments
Is this meaningful? Aren't frontier models MoE anyway? If you issue a StarCraft query they will probably just route to their StarCraft optimized model.
MoE is a misleading term, there is no "starcraft expert". There's not even a python expert. MoE just means a subset of blocks of weights are activated per query, nothing more. In so far as they might have legible functions they're like "balancing parens"
MoE doesn't work that way. It's not about increasing performance of a task, but speeding it up. To get a faster model, you need to make it smaller. If you make it smaller, the less it can remember. MoE tries to get faster responses without reducing the capabilities by running a smaller part of the model, and constantly swapping the parts used, so that the whole model gets to influence the output collectively, but each token is created by a smaller set of weights.
Anyone else feels like they can tell when the underlying model is MoE? Dense models are more concise in their answers. MoE seems to stall for time, writing vaugely until a right set of weights gets picked and commits to an answer. MoE also backtracks a whole lot more.