I am honestly asking and not trying to be a smartass:
What are the advantages of LLM summaries over man pages/google searches/stackoverflow threads?
I could maybe empathize if an engineer only had 30 seconds to understand a code snippet or a large shell command, but how often is that really the case? Is the time gained worth the risk of hallucination or incompleteness?
Or perhaps engineers are not the target audience?
I can provide a use case, although I don't use this site specifically, I do use LLMs (generally Claude) to get command line syntax fairly regularly.
The answer is it's faster and (in my experience) generally accurate for common command syntax. I don't remember all the params for a variety of commands but Claude can get them quickly and easily. Where (IME) LLMs fall down is for more obscure commands but then those are harder to find via traditional searches anyway :D
Googling is very hit or miss these days you end up with a load of sponsored results and then have to try and find an example with ideally the exact syntax you're looking for.
> The answer is it's faster and (in my experience) generally accurate for common command syntax.
If you don't mind, around what level of complexity are you querying it for? Are the queries along the lines of "how do I find a keyword in a directory in UNIX?" or more along the lines of "write a bash script that will do foo recursively and treat edge case bar by doing baz"?
In my experience, if the queries are closer to the latter, it takes less time to "man page" the holes in my knowledge than to identify and to fix whatever is wrong with the LLM's initial guess. If you more often than not receive correct answers, even in non-trivial cases, could you please provide the LLM and any custom parameters if they exist? I'd be happy to be proven wrong and learn something in the process.
> Googling is very hit or miss these days you end up with a load of sponsored results and then have to try and find an example with ideally the exact syntax you're looking for.
May age be a factor here? I grew up as search engines were becoming a thing and ignoring "fake" results has become second nature.
Kind of both. Simple things like the switches on ffmpeg to rotate a video, is one example. I do that infrequently and Claude got it right in one. Quicker than reading the ffmpeg man page for that information by a long way. Also basic utility scripting something like "give me a golang program that connects to a Docker instance and returns information in this format". I could write it but a lot slower than an LLM
As to the age thing, well I doubt it. I started on the Internet in 1995, and I've been on it ever since. Sure I can skim by fake results but it slows it down as the good results are ever fewer on the page. I can do it but LLMs are faster and avoid all of that.
Also by using them more, I get a better sense for what will work and what won't work, improving my velocity again. It's easy to spot that they're good for things that are widely done, so more common and widely used commands and languages.
They fail hard on novel or new things, for example OTel code is one place where they're bad, and also deeper parts of things like k8s.
I tried ChatGPT 4 for some time and went back to just using manpages after several wrong results. Sometimes results were bad even after providing information about my platform.
Have there been posts already from people accidentally deleting files caused by trusting LLMs too much?