Haven't gone through full PDF as its very detailed, few things have resonated with me so far.
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
Love your analogy.
Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc).
Maps well onto the LLM surface.
Anyone that's into auto racing as a hobby will tell you the engine is the last thing you should modify unless your whole goal is to get into the guts of engine mechanics.
But if you just want to improve lap times it's way less of a priority than brakes, tires, suspension setup, cooling to keep everything happy, etc.
I'm not a car guy but would it be fair to see that your maximum possible performance is bounded by the engine, but the average performance gains are from tuning the rest of the vehicle?
Kind of like if your RAM has to constantly page, you would see much more improvement from upgrading your RAM than overclocking your CPU?
so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness