Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
[dead]