Somewhat tangential, but this reminds me of when I took a certain Microbiology midterm in college. I hadn't put much effort into that class leading up to it, as I had some tough CS classes that consumed my focus that semester (also was admittedly not a great student in general). So when I finally went to take the midterm, after cramming like mad before the exam, I quickly realized I basically didn't know the answer to most of the questions, even what I crammed was only a small portion of the material.
The exam was huge, at least over 10 pages, and even when we technically ran out of time, the professor was kind enough to move the remaining exam takers to the neighboring lecture hall to continue taking it. I recall I spent a total of 2 hours on that exam.
Now mind you it was mostly short answer or multiple choice questions. The multiple choice questions were pretty sharp too, lots of traps and false but sounds right answers mixed in. But if it had been purely essay questions, I would have been screwed.
However with such a huge corpus of information in front of me, I ended up basically learning all the material on the spot. I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
In the end I got B+ in the exam (which was curved to an A), and walked out understanding the material better than I did walking in.
Reflecting in the experience years later, I've wondered if a hypothetical LLM which was ignorant of microbiology could do the same thing if fed that exam. In some respects the traps they placed in the multiple choice questions actually were what helped me refine my understanding the most. Made me appreciate information theory more.
> I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
This strategy saved my ass on so many tests from high school through college that I did not study properly for.
I think it would certainly be possible to write a test such that this strategy doesn’t work at all, but lucky for me no one did in any meaningful way.
There was a couple famous papers from a few years ago asking why computer science students, sorted by ability, seemed to form two clusters.
In one of the studies they got incoming CS students to fill out a multiple choice exam on programming, before the class had even started. Then they looked for hints in their exam results to see if they predicted the end of semester grade.
Turned out following this strategy was an incredibly strong predictor of success in computer science. Some students somehow got 0 (or close to it) in the multiple choice test. But the researchers looked at their answers, and could see that the student was forming hypotheses about the questions. Even if they got all the answers wrong, they answered all of them with a consistent idea of what the computer programs might have meant.
Weak students just answered each question individually based on vibe, without using what they learned (or figured out) from other questions.