Talking to an LLM is not a skill, just like using Google is not a skill.
Why? One, the companies like Google or Anthropic or OpenAI are working hard for it not to be a skill. That's the whole point. Second, these system are opaque, so there is no understanding to happen, only superstition, which might be wrong or change tomorrow.
> just like using Google is not a skill
I beg to differ. It is a simple skill that a great many have, but that doesn't make it “not a skill” - there are certainly many that don't have it, or don't want to practise it. Though I wouldn't name it specifically for Google, it is the more general “finding information online” skill which feels more specific because for many people these days it doesn't extend much beyond using Google or whatever their browser's default search service is.
People without the skill are quite evident: many of the closed duplicates on SO and similar sites are due to people lacking the skill to find information in old answers and effectively just asking others to look things up for them, the same for this week's 20th+ “my first layer has these bumps and gaps, what is going on?” question on any 3D printing forum (facebook groups etc.) that could be answered by scrolling down a few posts, and I'm sure the equivalent happens in groups serving any other plaything/hobby/skill/whatever.
I disagree, both of those are skills.
Neither are skills that a large portion of users of those services pursue to any meaningful extent, I'll grant you that. They also certainly are not synonymous with the term "soft skills" as I know it. So I think I am on your side of the fence on that part any way.
I feel like if they were skills under a reasonable definition, you should be able to name an expert in these skills, and how do we know they are an expert.
But I think you can't. It seems to me, instead, one is better at googling/prompting the better they are in a particular domain, but it only applies in that domain. Like knowing a jargon is not a skill, knowing the domain is.
Jason Wei's chain-of-thought prompting techniques literally revolutionized llms themselves.
https://arxiv.org/abs/2201.11903
> you should be able to name an expert in these skills
There are certainly local “finding information online” experts in many families and social groups.
> and how do we know they are an expert
They are the ones who get mentioned a lot in conversations in the manner “I'll have to ask [name]” with the implication that [name] will look up the information or know it from previous occasions people (possibly this specific person and [name] is getting sick of them asking and not remembering simple answers!) have asked.
Soft skills include: handling change under ambiguity, critical thinking under pressure, self-awareness, prioritizing, motivating and guiding others without relying on authority, navigating disagreement constructively.
Johnny Long is the godfather of google dorking.
https://www.youtube.com/watch?v=fo1BR9itwOY
https://blackhat.com/presentations/bh-usa-05/bh-us-05-long.p...
https://en.wikipedia.org/wiki/Offensive_Security
And of course if you point any of this out people go “Well many people are good at it, people getting bad results are just bad at it.” The constant refrain of “you’re just using it wrong” has become very tiresome.
If you can always say “just use it right” every time a critique comes up, then we aren’t having an honest conversation about the limitations of these tools.
It is impossible to duplicate results with an LLM. This strikes me as a serious barrier to calling it a proper skill. If you can’t even somewhat replicate the results you can’t really improve the input consistently. You can’t become “skilled” at it if you can’t even reproduce what you did.
If you enter the same prompt 3 times the results are of pretty significantly different quality. ChatGPT literally has you A/B test for them sometimes. They’re right to call it superstition - it feels like we’re making incantations and hoping for the best a lot of the time.
Prompting LLM’s still feels like a constant game of guess and check. At best you can argue it’s an educated guess. I don’t know about you but I didn’t learn math by guessing and checking, I frequently had to work backwards and review where I went wrong and/or I had the answer given to me with the work shown so I can learn. I can’t do that with a prompt. When I get bad results (which we all frequently do) I just guess what it didn’t like, try again, and pray for a better result.
Me typing into emacs is also not replicable, until I git push. Me with emacs and a good coding LLM can generate systems that I see as good, in my extensive professional judgement. For coding, for production grade code, the LLM will mostly be a tool in the hands of the professional. There is a new category of disposable code, which I think will be useful for many adhoc investigations; in neither case is replicability a serious requirement.
I do a lot of guessing and validating in learning maths. It is pretty efficient way to build that conceptual understanding. I even try to predict the next big theorem as I am listening to a lecture or reading a text book. It's engaging.
And honestly, I am enjoying learning this new way to make code I am pleased with. Using the LLM effectively and for quality deliverables is different from typing in many surface ways, but modularity of thought, iterative design and implementation, simplicity and generality, documentation, all still pay off.
These conversations always veer into “well I find it useful so I disagree.” I’m not saying anything about utility, I just think that system prompting is still very much hopes and prayers. We’re having a discussion about whether or not it’s a hard skill one can learn, I think that’s incredibly debatable
> It is impossible to duplicate results with an LLM
But that's the fundamental property of it - it is stochastic by nature. The skill is to learn how to sandwich deterministic logic between layers of randomness. Determinism doesn't live in the model. It lives in the harness you build around it. You can't make the model deterministic (it simply cannot be), so you make the system deterministic instead. Validation before, validation after, the randomness stays contained in the middle.
Talking to an LLM is not a skill, having a meaningful conversation leading to practical outcomes is.
> It is impossible to duplicate results with an LLM.
Humans are called experts in many domains whereby they cannot achieve duplicate results. In fact, in what domain can an expert do that?
Nobody crosses the same river twice.
If you ask me to write an email, all 3 variations will be more or less the same, except it will almost always improve with each iteration. 3 cracks at a prompt is 3 separate, unrelated attempts. None of them informs the other. In fact, you run the risk of making it worse if you include previous versions.
> 3 cracks at a prompt is 3 separate, unrelated attempts. None of them informs the other.
Thats where the "skill" part comes in. Like your 3 attempts at email that `almost always improve`. This is where you input the "previous versions", not directly back into the model. It's the "soft skill" of being flexible and adjusting based on how an entity responds to the input. Not learning how to adjust the input (by using previous attempts to inform the next) to more optimally direct the output given the state of the llm (chat? agent? model, effort lvl etc) it will seem chaotic.
The skills wont make an llm* deterministic, same as applying these soft skills to people. Give the same input to a person in `3 times separate, unrelated attempts` you are likely to get 3 different seemly chaotic outcomes. The "skill" is in being able to take what you received as output the first time, and make adjustments based on previous attempts while accounting for the state of the entity for the next attempt.
* In no way am I saying they are conscious beings or whatever nonsense by using people in the analogy. There are, however, parallels in how a set of soft skills (and this is why they are "soft" skills) can be used to get more optimal results from an entity that should never be expected to act as a pure function.