Do you have any research, empirical data, or other hard evidence that shows that a model (especially a non-SOTA model) makes engineers even "a little faster"? I'm aware of anecdotes, but nothing more.
Do you have any research, empirical data, or other hard evidence that shows that a model (especially a non-SOTA model) makes engineers even "a little faster"? I'm aware of anecdotes, but nothing more.
The cleanest positive study is Cui et al., because it used randomized rollout in real companies. Developers at Microsoft, Accenture, and a Fortune 100 electronics manufacturer were randomly given access to an AI coding assistant. Pooled across 4,867 developers, the authors estimate a 26.08% increase in completed tasks among users. They also report increases in commits and builds.
While that research was published in 2026, it analyzed the 2023 period. Very much NOT sota models!
https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535
Thank you!