> I would love to see an Opus-4.8-level local model but TBH I just haven't got there yet. The models I've tried so far _are_ good but they aren't able to solve tough technical challenges, regardless of harness/prompting/etc.
Agree. It doesn’t even have to be local, using models in this size class through OpenRouter will reveal their limits if you work side by side with Opus level models regularly.
There are a lot of social media posts about people cancelling their Anthropic or ChatGPT subscriptions after installing a local LLM. I’ve used local LLMs a lot and I spend a lot of time with frontier models and the difference is still huge. As far as I can tell, the social media posts about local LLMs replacing frontier models are either wishful thinking, engagement bait, or people who must be working on much simpler projects with a much higher tolerance for slop than I have.
I have exactly the same opinion
Over the last couple years I’ve had to learn sales and understand the thought process behind this better, and I think I’m beginning to understand it
The psychology is that most people aren’t really trying to optimize for productivity (even most people who think they are) on an ROI basis, because their compensation is too decoupled from their actual raw output, and more closely coupled to how differentiated their marginal contribution is to peers. They’re much more incentivized to spend their personal/work time optimizing for being more skilled or acquiring some kind of competitive advantage relative to baseline.
Most people don’t consciously run the numbers of “I get paid $X/hr to add $Y of value” or model pay at work as something with variable inputs (eg something that can be increased with high performance), so it makes sense to them to spend 20 hours of time to save $100 or to make themselves 5% less efficient to take home 0.5% more or avoid doing something they don’t want to start doing.
NOT saying this always happens or that they’re stupid for doing so. I didn’t even realize how much I had been doing it myself until I started recognizing it, and shifted to having my own comp/performance fully aligned with the company’s P/L.
It actually makes a lot of sense IF you can accurately estimate incremental upside (which is much harder and more diffuse than modeling downside if you’re salaried a employee) or if the upfront skill/knowledge investment that looks like bikeshedding pays off in the long run.
These are great points. It's a little off topic but what you bring up is why i advise new grads to spend the first couple years of their career in small eat-what-you-kill companies. I think software devs who start out in large companies get this distorted view that their twice a month direct deposit is just magic and comes from the ether no matter what they do. The whole industry would be better off if everyone started out in a "you don't deliver, you don't eat" company and grew from there.
Strong agree, but I also think some roles in big companies (for me, infrastructure) or in certain industries (eg trading/finance) can help build the same understanding without as much of the variance/raw exposure to bottom line.
Now that the role of the ticket-cruncher is on the path towards full commoditization, and individuals can move much more quickly (and even more carelessly!), I think product roles will probably shift towards one where developers are more deeply embedded in the product/business process so that they own/understand what to build without as much separation between the decision-making and prioritization of what to build. Or at least, they should.
It was eye opening to me to run the math of "should X people work for Y months on this project to save Z per year?" and realize that in so many cases, the time and effort it would cost to stop "wasting" money on things is WAY more than you could actually save on it. Even "small" projects can very quickly become $1M+ investments in time and resources, and the diminishing returns add up quickly (but also a good way to justify the value of your contributions, when done). But the job only exists if it saves money or makes money...
While starting my career in the late 2000s in web design agencies wasn't good for my stress levels, it absolutely gave me an appreciation for some things that working at a ~15k employee corporate just doesn't. I can tell exactly which of my coworkers come from the "outside world" vs those who joined here as a grad and have only ever worked here haha
This is really insightful, thank you.
Can you share a bit more about how you shifted to be more aligned with P/L? And how to accurately estimate incremental upside?
I'm an early PhD student with interest in ibdustrial research/R&D, and currently struggling to understand how to think about how to navigate through my career.
To be fair, there is no 3 turns that I don't have to jump in into what Opus 5 is doing. There is either some regression or my prompting skills are so much worse now. Flash is not perfect and honestly some things depend on how big context do you keep. So I'm keeping like a really short context with my flash, but it works okay, even though it has a tendency to overthink, and yeah, I run it always in max effort mode.
Use Opus 4.8. 5 is absolute garbage.
Don't use DS4 Flash in max effort mode. It's just spinning its wheels, in my experience (I have a harness for testing models with 25 real bugs/features/etc from my real projects that I measure outcomes against) DS4 flash does _worse_ with max effort. It will literally have the right approach and reason itself away from it.
I don’t know how 5 can do so much better in benchmarks but absolutely suck to use in practice compared to 4.X. Fable feels better, Kimi and GLM also feel better sometimes but tbh all of them make plenty of annoying mistakes.
The prompts in most of the benchmarks match what you have to write to obtain good performance from Opus 5. Reading benchmarks is very revealing.
i have heard that for max effort in flash and it can be true, but overall it still performs better then the high, i run a mixed q2q4 quant.