My take is the opposite: considering how good AI is at coding right now I'm eager to see what comes next. I don't know what kind of tasks you've tried using it for but I'm surprised to hear someone think that it's not even "barely usable". Personally, I can't imagine going back to programming without a coding assistant.
I've seen them all over the place.
The best are shockingly good… so long as their context doesn't expire and they forget e.g. the Vector class they just created has methods `.mul(…)` rather than `.multiply(…)` or similar. Even the longer context windows are still too short to really take over our jobs (for now), the haystack tests seem to be over-estimating their quality in this regard.
The worst LLM's that I've seen (one of the downloadable run-locally models but I forget which) — one of my standard tests is that I ask them to "write Tetris as a web app", and it started off doing something a little bit wrong (square grid), before giving up on that task entirely and switching from JavaScript to python and continuing by writing a script to train a new machine learning model (and people still ask how these things will "get out of the box" :P)
People who see more of the latter? I can empathise with them dismissing the whole thing as "just autocomplete on steroids".
I've been playing with it recently, and I find unless there are very clear patterns in surrounding code or on the Internet, it does quite terribly. Even for well-seasoned libraries like V8 and libuv, it can't reliably not make up APIs that don't exist and it very regularly spits out nonsense code. Sometimes it writes code that works and does the wrong thing, it can't reliably make good decisions around undefined behavior. The worst is when I've asked for it to refactor code, and it actually subtly changes the behavior in the process.
I imagine it's great for CRUD apps and generating unit tests, but for anything reliable where I work, it's not even close to being useful at all, let alone a game changer. It's a shame, because it's not like I really enjoy fiddling with memory buffers and painstakingly avoiding UB, but I still have to do it (I love Rust, but it's not an option for me because I have to support AIX. V8 in Rust also sounds like a nightmare, to be honest. It's a very C++ API).
> but I'm surprised to hear someone think that it's not even "barely usable".
write performance oriented and memory safe C++ code. Current coding assistants are glorified autocomplete for unit tests or short api endpoints or what have you but if you have to write any safety oriented code or you have to think about what the hardware does it's unusable.
I tried using several of the assistants and they write broken or non-performant code so regularly it's irresponsible to use them.
I've also had trouble having assistants help with CSS, which is ostensibly easier than performance oriented and memory safe C++
Isn't this a good reward function for RL? Take a codebase's test suite. Rip out a function, let the LLM rewrite the function, benchmark it and then RL it using the benchmark results.