> programming has a clear reward function.

If you’re the most junior level, sure.

Anything above that, things get fuzzy, requirements change, biz goals shift.

I don’t really see this current wave of AI giving us anything much better than incremental improvement over copilot.

A small example of what I mean:

These systems are statistically based, so there’s no probability. Because of that, I wouldn’t even gain anything from having it write my tests since tests are easily built wrong in subtle ways.

I’d need to verify the test by reviewing it and, imo, writing the test would be less time than coaxing a correct one, reviewing, re-coaxing, repeat.