>2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained.
When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of subjective opinions of what is appropriate. To go beyond broad demographic guessing the model really needs to 'get to know you' to know what it means when you specifically request an action. Without that information about you it has to weigh your words against the level of sophistication it expects a standard user is able to express.
Maybe you mean that an expert will use more specific language which in turn triggers the model to give a response that more closely matches the "expert distribution"
Anthropic published a study showing that Claude does more work for the expert user, and experts have a higher rate of "successful sessions" than novices.
https://www.anthropic.com/research/claude-code-expertise
> has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do.
I guess OP should have told it more explicitly to “find all errors without missing anything.”
> Thinking. I know this user well, they don't actually want me to find all errors.
> Thinking.. But I found a smoking gun of an error with this SPICE model, maybe I should inform the user.
> Thinking... Hm, but again, I know this human well, they likely don't care about this error. That's absolutely right - it's not an assistant's job to decide this, it's the user's.
Well if you want it go go off and try and validate the spice simulator and the kernel of the operating system that it's running on then that might be an approach to use.