on
Learning math with LLMs
I am sure that much has been said on this topic somewhere, but actually I haven’t seen a lot of discussion of this precise point.
People seem generally to agree that LLMs have the potential to revolutionize education somehow, and a few people are already productizing this belief in limited ways. In my view, an idealized form of education is complicated because so much depends on what you are trying to optimize for, and the child’s natural inclinations, but I don’t want to get sidetracked by a discussion for that now. I just want to make this narrower claim:
For a sufficiently motivated individual, LLM chatbots have made it tremendously easier to obtain expertise on new topics of interest, particularly in math and science.
This is a pretty intuitive statement on a couple of levels. With a little bit of imagination, we can make the analogy to aristocratic tutoring, which historically is how a bunch of the greatest minds ever were taught, and in the modern day is one of the only things that people can agree actually improves educational outcomes, because of course it would. It’s hard to enumerate the number of material constraints placed on even the most capable and well-meaning of teachers, when they have to teach multiple students at the same time, even the most capable and enthusiastic of students, even with the benefits of modern technology. By contrast, the main problem with aristocratic tutoring is that it’s limited by the pool of available human teaching capital, and is too resource intensive to apply at scale, but you know what isn’t? Yeah.
Here are some specific observations about the advantages of learning from LLMs:
Obviously, LLMs have an extremely broad knowledge base, so they are able to make connections bridging whichever disparate concepts you care to name (at least, if such a connection exists in its training corpus, or can be easily reasoned towards).
The startup costs are minimal, because chatbots are so technically easy to use, and already provide the most natural possible interface for inquiry-based learning. (This is disproportionately valuable for me because I have dragged my feet on doing things that would improve my life because of their perceived complexity.)
They never tire or become impatient, and will always make a good-faith attempt to answer your question, even if it is confused. Asking a well-defined question and providing as much information as is necessary to solve it is a skill which a lot of people do not have. Historically, StackOverflow was the place to get expert advice about narrow problems that could not be solved by search engine queries. However, it was human experts creating the answers; as a result, the barriers to entry were high to maintain site quality and reduce demand on the experts, and, more plainly, sometimes people are assholes. By contrast, LLMs’ patience and energy capabilities exceed those of even an ideally-incentivized team of aristocratic tutors.
As a narrow example of the above, even among people who are generally skilled at asking questions and learning new things, sometimes when you’re new to a field you just don’t know enough about how something works to ask coherent questions. Sometimes you might even have a clear concept in mind and simply not know the term for it, and have no means of finding it (even in the age of search engines). I emphasize this because in my estimation, this was a serious problem before. Sometimes your line of inquiry would die right there if the conditions weren’t right. Sometimes it would be too huge of a distraction to take the effort to pursue it, so you let it die.
Their comparative “thinking” and “speaking” speed are superior to a human’s. They can provide as many illustrative examples as you ask for without having to think hard or spend time constructing them, and you can absorb the “non-dense” parts of their speech as fast as you can read, which is faster than people can speak.
Closely related, with a human teacher, even if they are dedicated to teaching just you, they are not always perfectly aware of what you do and don’t know/remember, so a lot of their effort/exposition is wasted recapitulating things that you already know. With LLMs you can kind of just “fast-forward”/skim over these things in a way that would be impractically rude with a human teacher.
The returns to meta-learning are also comparatively higher, because you’re not limited by existing resources and references, because LLMs are generative. This is to say, you can reflect on what materials would help you learn best, and then just ask the LLM to generate those materials. You are more in charge of your learning than ever before.
Obvious caveats apply about misinformation, hallucinations and the like. I assume that these are real problems that go quite deep, and it’s fun/harrowing to imagine how to best fit these tools to the ideal education of a young child. (For example, regarding that point about asking good questions being a hard skill, asking good questions still provides returns to the clarity of your own thought, and even increases the capabilities you can extract from talking to an LLM, but maybe having a helpful LLM around disincentivizes you from ever learning this skill in the first place.) But for adults who already know a bit about their topic of choice, or are just used to the way learning broadly works, these downsides can be mitigated effectively, and are incredibly small compared to the upsides.
Reflections
I reflect constantly on the current state of affairs with awe because when I was a child, I spent a lot of time looking for good answers to subtle inquiries, and being frustrated that I couldn’t find them (on the internet or from the people around me). I also found that I learned a lot of things better in dialogue, and that dialogue kept me enthusiastic and focused about learning about various things, so I thought that it would be best if it would be possible if I had some kind of outlet I could speak to about what was on my mind and have an engaging conversation with, but ideally on-demand and without the limitations of human communities (they take time and energy to coordinate with and travel to, they often lack expertise except sometimes on the narrow domain of their interest, even when they have the expertise, they don’t always think on the same wavelength as you, etc. etc.) I spent a long time coming around to coping with the fact that I didn’t live in the idealized world I imagined, and that the resources I was looking about didn’t exist. Then, they started to, which felt a little bit like the universe playing a joke on me.
Another thing is, I have spent a lot of time internally investing in becoming a better teacher– motivating ideas and explaining things clearly from multiple perspectives. This is something I think of as unique and valuable in myself, so as with artists and other creatives I don’t like the notion that the effective (outward facing, “economic”) value of being this way is decreasing, and I’m afraid that it losing its extrinsic value will eventually cause people to lose track of its intrinsic value.
For now, human experts still have the following advantages over LLM/AI teachers:
Things coupled to being human, like being able to read facial expressions, gauge interest, emote, apply social pressure, and generally take a more active role in shaping student motivation.
Superior ability to actively suggest an entire known path of inquiry, as opposed to answering requests passively (e.g, “this topic is important, and here is what there is to learn in it”)..
The bottom line about all of this is that if there is something that you have always wanted to know or understand (and it can be taught in a purely text-based format), it has never been easier to do so, and this statement is more and more true the more that you were previously limited because the topic was too hard or too nuanced, or its materials were too dense or obscure. The psychological implications are also substantial- for one, it can feel like time that was previously spent learning things “the hard way”, searching for the right word to fit your query, etc. was wasted (God, I’m glad to not know what research, or learning in general, was like before the internet); for another, another limitation has been removed, our individual potentials have all gotten higher as a result, and so too are the corresponding expectations.
At the beginning of this post I gave the caveat that a person must be “sufficiently motivated”. There aren’t a lot of clear examples of people who have given themselves world-class educations with LLMs, even in the kind of narrow sense that would be achievable in the time since LLMs came out. I think that the main reason for this is that committing to learning something big is genuinely hard, even with all of the advantages that LLMs bring, and I think it’s likely that actually being able to muster the motivation to see oneself all the way to expertise is now the limiting factor in the education of a lot of people. For this, traditional social structures and obligations, such as schools, still have their place as an external source of motivation, and old-fashioned references and collections of knowledge, such as textbooks, can be helpful as scaffolding or jumping-off points.