Apparently, the noise around the AI policy came from Bun's developers saying that policy blocks upstreaming their performance PR. But the real reason seems to be that PR's code itself isn't in great shape, and introduces unhealthy complexity https://ziggit.dev/t/bun-s-zig-fork-got-4x-faster-compilatio...
> Parallel semantic analysis has been an explicitly planned feature of the Zig compiler for a long time, and it has heavily influenced the design of the self-hosted Zig compiler. However, implementing this feature correctly has implications not only for the compiler implementation, but for the Zig language itself! Therefore, to implement this feature without an avalanche of bugs and inconsistencies, we need to make language changes.
Yes, that reply provides convincing arguments for not merging the Bun fork, as it interferes with Zig's own roadmap for achieving even better results, while continuing to improve the whole language.
Not only this, but also:
Bun's fork will exhibit indeterministic behavior.
A single PR for a 3000-line addition would, in all likelihood, be rejected anyway.
Really depends the author and context. Large PRs are often justified for compiler work, you have a lot of pieces to touch at the same time
Doubt it: https://github.com/ziglang/zig/pull/24536
When somebody comments PR with “Incredible work, Jacob. It is an honor to call you my colleague.” then it's safe to assume it's out of the ordinary contribution. Pretty much falling outside of the “in all likelyhood”.
3000 line LLM commit is not that.
Also 95% of those 30k lines changed are fully self-contained inside of the aarch64 directory and of the remaining changes it looks like the majority is just adding "aarch64" as another item into an existing list. There are a few core changes that to me look like they could be done in their own PRs, but also core maintainers get to decide if they want to apply bureaucracy to their own work.
No description provided. I love this PR. But yeah, try being anyone besides Jacob and submitting that!
> In successful open source projects you eventually reach a point where you start getting more PRs than what you’re capable of processing. Given what I mentioned so far, it would make sense to stop accepting imperfect PRs in order to maximize ROI from your work, but that’s not what we do in the Zig project. Instead, we try our best to help new contributors to get their work in, even if they need some help getting there. We don’t do this just because it’s the “right” thing to do, but also because it’s the smart thing to do.
I feel like if their goal is to prioritize contributors over contributions, it'd also logically follow that they should try to have descriptions where possible? Just to make exploring any set of changes and learning easier? Looked it over briefly, no Markdown or similar doc changes there either.
I mean the changes can be amazing, it's just that adding some description of what they are in more detail, alongside the considerations during development, for new folks or anyone wanting to learn from good code would also be due diligence.
How would you differentiate a 3000 line LLM commit made by the best models and good AI processes from a 3000 line commit made by the best human developer?
edit Okay, I set the bar too high here with "best human developer" and vague "good AI processes". My bad. Yes, LLM is not quite there yet.
A personal relationship and trust, as seems to be the case here?
By using my brain.
Don't be ridiculous! We don't do that anymore.
Read it?
The post that inspired this post [0] says:
> So while one could in theory be a valid contributor that makes use of LLMs, from the perspective of contributor poker it’s simply irrational for us to bet on LLM users while there’s a huge pool of other contributors that don’t present this risk factor.
> The people who remarked on how it’s impossible to know if a contribution comes from an LLM or not have completely missed the point of this policy and are clearly unaware of contributor poker.
The point isn't about the 3000 line PR, it's about do we think the submitter is going to stick around.
[0] https://kristoff.it/blog/contributor-poker-and-ai/
It seems to be trivially easy for everyone but people heavily invested into LLM to spot LLM slop
It's still fairly obvious just by skimming the code. The best AI models are still quite far from the best human developers in ability and especially in code quality.
When the best AI models are the same or better than the best[1] human developers, what then?
We're already at the point talking about best vs. best.
If that happens and we have a way of reliably knowing if some code is produced to that high quality, then I think we probably can accept that AI coding is the only sensible option.
We definitely are not close to that point though and it's unclear if/when we will get there.
It seems to me that people might be arguing from conflicting hidden premises here. "AI Coding" is a spectrum that could mean something as simple as letting the LLM proofread your changes and then act on those with your own human brain, or it could mean just telling the agent what you want and let it rip and tear until it is done.
If I do the latter and submit a PR to something like Zig, I'll be certainly caught doing it and rightfully chastised. If I do the former, my PR will be better without anybody besides myself having any way of knowing how it got better. Probably I do something in between when I contribute to open-source these days.
Blanket banning all of these seems like a bad idea to me. It actively gates people like myself from contributing, because I respect these people and projects that much. It feels like I would be doing something they find disgusting if my work has touched an LLM and I obviously don't want to do that to people I respect. But it's fine, there are plenty of things to do in the world even when some doors are closed.
I do not presume to have any say on Zig project's well argued decisions[0] -- I'm not really even their user let alone someone important like a contributor. Their point of preferring human contact is superb, frankly. Probably a different kind of problem in an open-source project staffed with a lot of remote working people, where human contact is scarce.
https://kristoff.it/blog/contributor-poker-and-ai/
How can AI possibly be better than “the best” when the corpus of training data now includes its own slop in addition to all the code by new devs/lazy devs/bad devs scattered all over the internet? Law of averages applies here.
Because LLM models are obviously much more than the sum of their parts.
Oh, which parts are those? Do tell!
Jacob is part of the core team, not a random outside contributor.
Very different context: that PR is from a maintainer, and trusted member of Zig, which surely discussed the implementation/design internally as well
What’s the point in debating the PR quality? The policy explicitly forbids all LLM code, so that policy is of course the “real reason”.
> What’s the point in debating the PR quality?
Because the pro-group are whining that the policy is preventing the merge, when in actual fact even if the policy did not exist, the PR is crap anyway.
I don’t see how it could be that bad (incorrect, specifically), considering bun is probably the most widely-used production use case of zig. But regardless, let’s say it’s a bad PR for the sake of argument - it’s beside the point. It cannot be merged no matter how good it is, due to the strict no-LLM policy.
> I don’t see how it could be that bad (incorrect, specifically), considering bun is probably the most widely-used production use case of zig.
That may be the case, but the bun project only needs zig to correctly compile bun. The zig project needs to be able to correctly compile all existing and possible zig programs.
I haven't reviewed things, but it's possible and even likely (at least based on my own experience with LLMs) that the validation is mostly focused on bun compilation.
Do you think they skipped the main zig test suite or something? Only tested bun compilation? That seems unlikely to me
They didn't take into account the long-run impacts of the changes on future development, etc.
I recommend reading the explanation given by one of the Zig devs, as it's a very clear and solid one.
This is the most common issue I see with LLM authored PRs. Yes it does fix the issue _right now_ but as a maintainer I need to consider how it affects the project in the future. But “contributors” get mad if you reject for those reasons. So I can understand having a blanket policy.
> I don’t see how it could be that bad (incorrect, specifically), considering bun is probably the most widely-used production use case of zig.
The PR is probably fine for bun’s purposes. That doesn’t make it a good PR for Zig’s purposes, and could very well paint Zig into a weird corner.
> It cannot be merged no matter how good it is, due to the strict no-LLM policy.
This is about meta-discourse. Of course it’s against the policy. That’s the point of discussing the PR: to get Zig to change the policy, or at least provide an exception in this case. Or to argue the opposite.
Of course the policy is preventing the merge. That’s literally the point of the policy…
> Of course the policy is preventing the merge. That’s literally the point of the policy…
In this case it isn't the blocker - the fact that the dev took the time to read the PR in detail, comment on it, and provide reasons why it could not be merged makes it very clear to me that the policy wasn't the blocker.
If they were going to enforce the policy for this PR, they wouldn't have bothered to read it. The only reason to read it is to see if the policy is waived for this specific PR.
OTOH why bother to polish the PR if it won't get accepted anyway?
> OTOH why bother to polish the PR if it won't get accepted anyway?
As the Zig maintainer so patiently explained, no amount of "polish" can fix the PR because it is misaligned to the correctness that they require.
IOW, that PR is so far off the reservation, unless it is completely rewritten, it won't be accepted.
it could have been rewritten, rewriting PRs is cheap today, but that isn't the question. the question is, would it have been accepted had it met all the quality and engineering standards and full disclosure that it was 90%+ LLM generated?
> it could have been rewritten, rewriting PRs is cheap today
Rewriting PRs with LLMs is cheap, but often the output is no better than the previous revision (fixing one issue only to cause another one is very common IME). And reviewing each revision of the PR is not cheap.
I've had good experiences with people submitting AI generated PRs who then actually take the time to understand what's going on and fix issues (either by hand or with a targeted LLM generated fix) that are brought up in review. But it's incredibly frustrating when you spend an hour reviewing something only to have someone throw your review comments directly back at the LLM and have it generate something new that requires another hour of review.
> it could have been rewritten, rewriting PRs is cheap today, but that isn't the question. the question is, would it have been accepted had it met all the quality and engineering standards and full disclosure that it was 90%+ LLM generated?
In this case it looks like the answer is "Yes"; the PR was not dismissed immediately, it was first examined in great detail!
Why would the maintainer expend effort on something that was going to be rejected anyway?
because the policy is clearly 'reject' and yet significant time has been spent - either effort was wasted or policy is at best 'not implemented'.
> either effort was wasted or policy is at best 'not implemented'.
I don't understand this PoV - have you ever come across a policy in any environment that wasn't subject to case-by-case exceptions?
Even in highly regulated environments (banking/fintech, Insurance, Medical, etc), policies are subject to exceptions and exemptions, done on a case-by-case basis.
The notion, in this specific case, that "well they rejected it because of policy" is clearly nonsense and I don't understand why people are pushing this so hard when the explanation of why an exemption can't be made for this specific PR is public, accessible and, I feel, already public knowledge.
No amount of rewriting will help you if you, fundamentally, wrote the wrong thing, as is the case here.
People forget that LLM code cannot be covered by copyright. So LLM code cannot be placed under an open source license
This is overstated. Not all LLM code is produced the same way. Code produced through substantial human creative input still falls under copyright, at least the way things are now. Besides, nothing legally prevents placing code under a license. Enforceability is the question, not permission.
It's a bit like saying speed limits don't apply on private property, therefore you can't have any traffic rules on your private racetrack.
Because it's Bun. Which is practically the use case testimonial of Zig.