Yes, it's not that people know specifically what they don't know, it's that they develop the wisdom to know those boundaries and anticipate them and reduce their likelihood and impact.
For example, if I use the language of my expertise for a familiar project then the boundaries where the challenges might lie are known. If I start learning a new language for the project I won't know which areas might produce unknowns.
The LLM will happily give you code in a language it's not trained well on. With the same confidence as using any other language.
Sorry for copypasting from another comment but this is relevant
I can tell you what skills I possess. For example: programming, writing music. A LLM can not do this unless it's told what it knows. I could also tell you whether I studied thing X or attempted to do it and what success I had. So I'm pretty good at assessing my abilities. A LLM has no idea
This is interesting, because I wouldn't be too sure about that. Whether I am able to play chess well exclusively depends on the strengths of my opponents, because there is no absolute baseline. If society somehow decided that what you were writing wasn't "music" anymore, you would be the only person left stating you had that skill.
I believe that most claims about one's own skills do come from outward judgement and interpretation of one's actions, not introspection. The only thing humans have is, to radically appropriate the jargon, a way longer context window (spanning over an entire lifetime!), together with many ways to compress it.
Yes, it's not that people know specifically what they don't know, it's that they develop the wisdom to know those boundaries and anticipate them and reduce their likelihood and impact.
For example, if I use the language of my expertise for a familiar project then the boundaries where the challenges might lie are known. If I start learning a new language for the project I won't know which areas might produce unknowns.
The LLM will happily give you code in a language it's not trained well on. With the same confidence as using any other language.
Sorry for copypasting from another comment but this is relevant
I can tell you what skills I possess. For example: programming, writing music. A LLM can not do this unless it's told what it knows. I could also tell you whether I studied thing X or attempted to do it and what success I had. So I'm pretty good at assessing my abilities. A LLM has no idea
This is interesting, because I wouldn't be too sure about that. Whether I am able to play chess well exclusively depends on the strengths of my opponents, because there is no absolute baseline. If society somehow decided that what you were writing wasn't "music" anymore, you would be the only person left stating you had that skill.
I believe that most claims about one's own skills do come from outward judgement and interpretation of one's actions, not introspection. The only thing humans have is, to radically appropriate the jargon, a way longer context window (spanning over an entire lifetime!), together with many ways to compress it.