I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same.
If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?"
> If a nuclear engineer enabled and instructed him, would there not be liability for the hazard?
Should the library where he read books about physics also be liable?
A difference of degree is a difference of kind here. If something previously required years to full-time study to learn, but now you can kind of somewhat stumble your way through it and get somewhat close to the result, you should not disregard that with a snarky one-liner IMO.
E.g. look at programming - people who don't know how what a compiler is, are making things that I could only make after a few years into my programming journey.
You obviously get the same results in chemistry or nuclear physics or whatever, the models are heavily trained on code in particular, but if there's a chance that we've reduced the ease of committing certain kinds of crime that were previously gate-kept by knowledge, we should know about it.
I read a high-school chemistry book describing the synthesis of nitroglycerine, it's not complicated. I would not recommend to try the synthesis in any significant amount.
> If a nuclear engineer enabled and instructed him, would there not be liability for the hazard?
I bet the professional would be able to sate the kid's curiosity safely without creating excessive risks.
I've come across detailed instructions on how to synthesize sarin gas on the internet. Anyone who follows those instructions will probably die horribly. I still thought it was pretty interesting.
I agree LLMs can be harmful and that the companies behind them should be held liable to some extent, for example the recent news with Google being held responsible for their AI's defamation.[1]
This is a pretty different argument though. The comment that started this thread was talking about LLMs making potentially dangerous knowledge more available to bad actors, now we're talking about LLMs giving personally harmful advice.
You asked:
>If he had the help of Claude at the time, how much more dangerous would his bumbling have been?
Probably less? Even if you removed all the guardrails from Claude it would've likely told him his reactor plan wouldn't work and that he would have a high chance of poisoning himself and the environment.
[1]https://news.ycombinator.com/item?id=48470248