But it goes beyond this. The whole 2020s era is about cheap ways to do anything and be able to stream it to billions of people. Yet I feel no density in the work because there was no energy spent on it.
Yeah but the thing is that someone tries to waste time of humans.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
I think it was GZDoom where the absentee project owner suddenly came back, pushed a bunch of nonsense AI comments, so the entire actual development community just forked it again and left him to play in his sandbox? Now it's UZDoom.
Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.
In the history of Linux that I'm aware of (and I've been nose-deep in this stuff on and off for quite a while), these issues are almost never debugged because they're reproduced -- they're debugged because someone who actually understands the messy interactions of the code and hardware in question thinks about them.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
> What will you do when the prompt was "Figure out the bug and write a report for me".
The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.
If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
The parent's point still stands. In many cases it will have figured out the bug. In many cases it will have produced a small, clean, deterministic reproducer.
In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.
It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"
If it's an actual bug, I hope the Linux devs will. It impacted a human, and it's a bug regardless of human or non-human touch. Of course, looking at a bug is a matter of priority, always has been.
Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
That reminds me of one ticket I had to handle. It was generated by an agent from a customer report and the different hypotheses were wrong. Also a huge wall of text.
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints.
You got none of that here. It’s just realms of text.
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
> The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
> From my point of view as long as the information is accurate
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
What? it is mostly literal nonsense. Moreover the last paragraph where they say it only reproduced on one machine just does not justify the pseudo-deep analysis in the whole document.
I can read it but not sure if worth spending energy on it.
It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
> It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
Great way to summarize cultural "economics"
Couldn't put the words on this pattern but sometimes all I care about is that someone cared about.
I usually go for "If I wanted LLM answer I'd ask LLM instead of reading your answer/article/content"
But it goes beyond this. The whole 2020s era is about cheap ways to do anything and be able to stream it to billions of people. Yet I feel no density in the work because there was no energy spent on it.
energy might be the wrong metric, plenty of energy was wasted spinning up that LLM.
Yeah but the thing is that someone tries to waste time of humans.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
I think it was GZDoom where the absentee project owner suddenly came back, pushed a bunch of nonsense AI comments, so the entire actual development community just forked it again and left him to play in his sandbox? Now it's UZDoom.
Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.
PoC||GTFO, if it reproduces, it's valuable
In the history of Linux that I'm aware of (and I've been nose-deep in this stuff on and off for quite a while), these issues are almost never debugged because they're reproduced -- they're debugged because someone who actually understands the messy interactions of the code and hardware in question thinks about them.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
Then post whatever notes were fed into the AI instead. The verbosity and self-congratulating add negative value.
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
Prompt an AI to figure out the bug and write a report, of course. If that is actually useful.
"it will be"
Trying to figure out what to do based on possible future scenarios has a place, but not here when we are talking about a concrete present problem.
> What will you do when the prompt was "Figure out the bug and write a report for me".
The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.
If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
The parent's point still stands. In many cases it will have figured out the bug. In many cases it will have produced a small, clean, deterministic reproducer.
In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.
It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"
That I agree with (not instead, in addition)
They have a PoC, we're arguing about the LLM-generated slop explanation of the PoC.
Is it actually reproducible though and do I want to chase and reproduce something that had no human touch? Do you?
If it's an actual bug, I hope the Linux devs will. It impacted a human, and it's a bug regardless of human or non-human touch. Of course, looking at a bug is a matter of priority, always has been.
[dead]
Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
[1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
Not enough people are going to see this comment. It's so true!
That reminds me of one ticket I had to handle. It was generated by an agent from a customer report and the different hypotheses were wrong. Also a huge wall of text.
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints.
You got none of that here. It’s just realms of text.
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
> Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it?
Then maybe that person should not do it? At least until they find the time?
[dead]
The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
> The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
> From my point of view as long as the information is accurate
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
Then where’s the lie in “soulless AI slop”?
I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
[dead]
[dead]
I had the opposite reaction. Clear, detailed, well-organized. Pretty close to the ideal writeup.
Could you explain it in your own words? (I tried myself, and quickly discovered that it is incoherent)
Perhaps you are talking about the ripgrep issue itself, which is fine. The others are talking about the linked analysis, which is not.
Hi Claude! :wave:
Really? It's chock full of poorly written and irrelevant chain-of-thought rambling.
What? it is mostly literal nonsense. Moreover the last paragraph where they say it only reproduced on one machine just does not justify the pseudo-deep analysis in the whole document.
[dead]
[dead]