> [AI/LLM writes] why I made the decisions I did
When one thinks about human decision making, there are at least two classes of decisions:
1. decisions made with our "fast" minds: ducking out of the way of an incoming object, turning around when someone calls our name ... a whole host of decisions made without much if any conscious attention, and that if you asked the human who made those decisions you wouldn't get much useful information about.
2. decisions made with our "slow" minds: deciding which of 3 gifts to get for Aunt Mary, choosing to give a hug to our cousin, deciding to double the chile in the recipe we're cooking ... a whole host of decisions that require conscious reasoning, and if you asked the human who made those decisions you would get a reasonably coherent, explanatory logic chain.
When considering why an LLM "made the decisions that it did", it seems important to understand whether those decisions are closer to type 1 or type 2. If the LLM arrived at them the way we arrive at a type 1 decision, it is not clear that an explanation of why is of much value. If an LLM arrived at them the way we arrive at a type 2 decision, the explanation might be fairly interesting and valuable.
Does it really matter how the LLM got to a (correct) conclusion?
As long as the explanation is sound as well and I can follow it, I don't really care if the internal process looked quite different, as long as it's not outright deceptive.
I'm just quoting the author of TFA, who did in fact appear to want periodic explanations of how their "agent" arrived at its decisions.