I agree LLMs can be harmful and that the companies behind them should be held liable to some extent, for example the recent news with Google being held responsible for their AI's defamation.[1]
This is a pretty different argument though. The comment that started this thread was talking about LLMs making potentially dangerous knowledge more available to bad actors, now we're talking about LLMs giving personally harmful advice.
You asked:
>If he had the help of Claude at the time, how much more dangerous would his bumbling have been?
Probably less? Even if you removed all the guardrails from Claude it would've likely told him his reactor plan wouldn't work and that he would have a high chance of poisoning himself and the environment.