It's much better to feed the book to the LLM and ask questions as you read along, instead of asking the LLM to basically write a custom book for you from scratch.
It's much better to feed the book to the LLM and ask questions as you read along, instead of asking the LLM to basically write a custom book for you from scratch.
I have to agree with this. Completely relying on LLM for all your learning needs is a disaster. But being absolutely against use of LLMs isn't doing you any favors. This is where you don't have a formula but rely on you judgement and evidence of your having learnt something.
For example having an LLM summarize a dense topic and to find books so that you can filter faster and spend time reading those books works way better than having the LLM summarize the books or the topic (or even relying on second hand information). Another one is having the LLM quiz you on your topics of interest. With questions tailored to attack specific areas that you struggle with. Its wonderful at this, nothing I've used comes close to what an LLM can do here.
You define for yourself what your goals are, slowly refining them as you learn more, and use LLM as a tool. This ,I find works best for learning.
Make it your goal to teach a room full of other humans that topic. I guarantee you will know that material cold. I've done lots of technical training in my career and after teaching a class two or three times I find myself to be very competent in the topic.
It's long been the case that the best way to learn something is to teach something.
> It's long been the case that the best way to learn something is to teach something
Which is pretty unfortunate for those that want to learn. I used to enjoy writing documentation at work, it was my favorite part of the job. And it did feel like it benefited me more than it benefited all the people that were (or weren't) reading my documentation. Now I can't really justify spending much time on docmentation when LLM's can do it in a fraction of the time and it's "good enough"
I got started as a software engineer working in the nuclear industry in the 80s. We measured our documentation in inches not pages, and it was all written by hand. And I'll bet you the documentation in the nuclear industry is still written by hand and not by LLMs.
I sure hope those docs aren’t written by LLMs for many reasons!
i wonder if people who use llm's for business critical documentation would have the same opinion, and if so why?
I have used LLMs for safety sensitive tech writing and so have people I know. It’s pretty good at it, and you usually have a 95% ready to go product at the end of it. The last 5% can pretty easily be filled in by technical experts with way less time spent battling the type writer.
Is it perfect? No. But it suffices most of the time in a pinch.
Can’t speak to the nuke industry, but it’s pretty good at aviation related things.
There is still value in experts distilling knowledge and crafting it to the audience. I'm a consultant in cybersecurity and someone asked me "give me a best practice framework for good policy hygiene". I'm sure an LLM could spit out some tips, but I've been on the industry 15 years and can write in 5 pages what an LLM wouldn't conceive of in that space.
> Another one is having the LLM quiz you on your topics of interest
This is a great idea. I'm going to try it.
Even with the latest models today, the hallucination rate is absurdly high on anything deeper than surface level knowledge or something that can be directly scraped from reddit.
And you notice when it's a topic you know well or something like software where you can immediately tell the options it's giving you don't exist on the page. Leading to the amusing statement "LLMs are bad at what I do but great at everything else".
I've (elsewhere) written about this diminishing return effect on LLM utility in relation to increasing expertise.
The question: what's the net positive gain of turning people who know nothing in a given field into sub-novices, while weighing actual experts down with work slop and marginal returns?
And I wonder what the true cost is of arming so many novices with that level of dangerous knowledge.
The fact that some people get genuine value from LLMs when learning doesn’t contradict the fact that they’re Dunning-Krueger “expertise” generators. The fact that the person learning from them is in charge of ensuring they aren’t full of shit, which they frequently are, is an inescapable flaw in this process. I honestly think that reduces the value of these things to just above what you can find out with a search engine with most topics. Hey, great. An improvement is an improvement right? Is it an improvement worth trillions of dollars and screwing over writers and artists worldwide? Fuck no.
Sounds about right.
Tangentially, but related: I'm old enough to remember when the spirit of your comment was pervasive on HN.
It wasn’t even all that long ago. I noticed around the time ChatGPT 3.5 was released I saw a significant change in tone regarding LLMs and diffusion models. I think that’s when some serious astroturfing started.
I’m old enough that I worked my first IT summer job the same year slashdot was founded. I’ve seen a lot of tech tribalism form and dissipate, and this one didn’t feel organic. My gut says a lot of the us-vs-them tension originated in a deliberate campaign to cast AI boosters as the tech industry in-crowd, and ‘other’ the people not on-board. Who knows.
1000% astroturfing—and across social media. Too many posts of the same quality at the same time(s). These things ran in cycles, spinning up and dissipating just as suddenly.
E.g. the entire framing to combat complaints about shortcomings was, "It's not the tech. It's you. You're just not doing it right. Wrong setup, wrong workflow, add this to your .MD, use loops, etc". Every complaint was immediately met with this same treatment by a swarm of vague bro-bots that materialized from the ether. The core message? Always the human's fault.
And, don't get me started on the waves of newly minted expert AI creators, making recommendations without showing a single example of what they'd supposedly built.
I'm sure some bandwagon organic creators tried to cash in on the genre, but encouraging that was also part of the point.
I think the true cost will be some catastrophic failures.
Just hoping folks don’t get hurt due to people not understanding what they’re doing with these things but believing they’re competent.
If that is the case, I seriously doubt the blame would ever touch the people that oversold this shit.
I came across the socratic method recently, and have used it to learn a couple of topics that I was having trouble getting to stick. There are some SKILL.md's available for it. It works for concepts as opposed to facts, and causes the model to guide you to answers through your own reasoning, which is both much more engaging than reading a wall of LLM text and helps the information stick.
> I came across the socratic method recently
This statement would out you as someone who didn't attend an elite school.
Can you link the Skills.md?
The "Socratic Method" (aka maieutic) skills annoy me, precisely because when you read them they are the kind of low-effort, low-expertise crap someone who over relies on AI would naively come up with when tasked with the problem of coming up with skills for learning. "Hey the Platonic dialogues are pretty cool and smart, let's do that".
The body of literature on learning theory, and beyond that on specific types of learning and specific mediums such as learning from text is so rich there are way more useful models to draw from. Believe it or not, prellm, researchers in the textual learning field had already demonstrated you can achieve performance equal or better than novice tutors using pretty basic computer aids that follow specific hint/pump interaction structures. Guiding an LLM to use these findings has evidence backing it and is way better than telling it "i guess be like socrates". The problem is, to realize there might be richer more effective and highly researched ways of tackling the problem beyond the first fart of a thought you had one afternoon requires the deep respect for expertise and specialization that precisely basically everyone in the AI space right now fundamentally lacks.
Can you provide some references to the research you're referring to? Sounds interesting but you haven't really provided enough information to find it.
That is the point – you can't just provide the references in a short post here.
You need much more time and guidance.
Ironically, LLMs are pretty good at finding stuff like that from vague descriptions. With that, I think the parent commenter meant Art Graesser's AutoTutor work based on the terminology used (hint/pump).
TLDR: Actual human tutoring sessions were recorded and analyzed and Socratic questioning was barely used at all. Instead the following pattern was observed:
Pump — "Uh huh?" "What else?" Costs nothing, so try it first.
Hint — points at the region of the answer. "What about the pumpkin's motion sideways?"
Prompt — fishes for one specific word, with the sentence frame supplied. "The pumpkin keeps moving forward at the same ___?"
Assertion — just says it. "It keeps the runner's horizontal velocity."
I was so annoyed I made a Socratic wrapper based on predefined curriculum:
https://adaptive.bounded.cc
Trying to diagrams/animations didn't yield good results even with frontier models. But pure text, any model does a decent job.
I didn't expect this to be hit by hundreds of request per minute.
So it may be very slow or become unavailable, back end can't handle that, no caching whatsoever.
what a time to be alive! "if you're having trouble understanding what your robot tutor is trying to teach you, you can ask it to guide you to the concepts using your own reasoning. This is both much more engaging than reading a wall of the robot's text and helps the information stick."
Whats funny about LLMs is they are trained from books, but they are also trained to not output books, so how much of an LLM skews its output because a perfectly normal sentence could be a quote in like 300 different books?
i threw the entire sanderson cosmere into a RAG graph sorta deal just to see how it would do if i questioned an mcp server for it about a universe i know decently well. it was actually astoundingly good. was able to find easter eggs acrossed different books and answer dumb questions like "why is kaladin emo"
If you don't know why kaladin is emo, did you truly read the books lol. Every character has to deal with the stresses of war and most don't come equipped with good mental health to begin with, they're just normal people
i was more or less testing to see how complete of an answer it could come up by scanning the whole semantic graph so i threw questions at it that spanned multiple books
Sanderson is 100% in the training data
damn it good point
I also agree with this. LLMs are a great companion when reading a book to clarify things and dive into specific topics.
I'd imagine an application that uses LLMs will be created that better manages learning. It's just not clear what that UX is yet- it's obviously not just a chatbot
dumb question, but what is the best way to feed the book to the llm?
i run into context window limits, or practical limitations of digitizing the book
You need a non DRMd copy of the book. You don't have to feed it all at once, although with a 1M context limit it is doable. A few chapters at a time is enough, in my experience. An easy alternative is using NotebookLM (now Gemini Notebook), and that has worked brilliantly for me, but I haven't tested it for technical topics (for that I like the LLM to create graphs and e.g. interact with Mathematica, so I haven't tried it).
Hoping you get answer to this. I have the same question.
I prefer to have the LLM ask me the questions
This. Ask it to quiz you if you are feeling it.
LLM is still too verbose, a real person Socratic conversation can interact a couple sentences at a time, not spew 1-3 windowfuls of low density bullet points.
I even wonder if this behavior is due to next-token prediction architectures, somehow.
I don't understand some previous complaints. It's dense and verbose seem at ends to me.
I know you probably don't consider it dense but wondering if someone can shed insight.
I find them like empty calories, like programming youtube tutorials. They maximize for feeling learnt instead of steady progress
There are ways to ground an LLM to be concise, and Socratic (method of inquiry, back and forth dialogue)
This. This is exactly how I use them and I have had no issues so far. I read the book myself, then I point the LLM at it to ask questions about notions I might be struggling with.
I'm sorry, but why not just read the fucking book if you're interested?
[dead]
This is the exact opposite of what the parent comments are talking about.
I agree, I too like to read authored books. But there are certain topics, especially the new ones doesn't have good books yet. My post was to show that you can create books based on your interest, in a way that you like w.r.t to the content/tone/layout etc. It's a new idea made possible by LLM's. It might become the norm in few years I believe, where each one will be having a personlized library of books that they curate. So the point was to persuade the parent commentator that some books are better created this way.