# LLMs Memorize Content Transformations If you train a ML system on one task, and then it becomes able to perform another task you did not anticipate, that's emergence. Many people interpret "emergence" as something wondrous and magical -- "it's alive!" But it's actually banal and has been going on for a long time.  Emergent learning happens because information space is not random. It's highly organized. So that if you learn one aspect of its organization, you will pick up other aspects as a by-product. It's especially prevalent with self-supervised learning.  Every self-supervised system ever developed has displayed emergent properties of some kind. When Mikolov trained word vectors in 2013, he noticed that some vectors in the resulting space encoded useful semantic transformations, like "plural(x)" or "gender(x)". That's emergence.  He had not designed or trained the system to do this. It was picked up as a by-product of learning word co-occurrences. Learn to organize words in a vector space, and your space will turn out to have interesting properties you did not anticipate.  LLMs display emergent learning all over the place -- not surprising, since they're pretty much a deep, at-scale version of word vectors. But why that happens and what it means is typically misinterpreted.  The extent to which it happens is also misunderstood -- most of what LLMs are good at is tasks *they were explicitly fine-tuned to do*. They may have had certain skills emerge from self-supervised training, but to turn them into something useful required explicit supervision.  People who say "LLMs don't memorize: look, it can do X, which it had never seen in its training data" misunderstand what memorization means. LLMs don't just memorize *content*. They memorize *content transformations*. Functions over content space -- higher-level versions of… [twitter.com/i/web/status/1…](https://twitter.com/i/web/status/1689703571493896194) And most of these content transformations don't need to be learned explicitly -- just like with 2013 word vectors. They just emerge as a by-product of self-supervised training.  But they're still functions that were memorized from the training data! To be able to do something, the model needs to have seen enough examples of the underlying pattern to be able to fit a function to it. And the function it learns might not be very good at all, in… [twitter.com/i/web/status/1…](https://twitter.com/i/web/status/1689704390897319936) What's the difference between memorization and reasoning, you ask? In short, it's the ability to synthesize new functions on the fly from very few examples -- what humans and many animals do.  If you need to be trained on many examples of a pattern (even if it's implicit) in order to learn a reusable representation for it, you're memorizing. If you can adapt on the fly to a new task by synthesizing a new brand function for it (from bits and pieces you previously… [twitter.com/i/web/status/1…](https://twitter.com/i/web/status/1689706474996645888) Further, the *generalizability* of the function getting learned/synthesized is also at stake. Memorizing systems (e.g. any system based on fitting vector functions via gradient descent) tend to learn functions via pointwise mapping, which does not lead to good generalization.… [twitter.com/i/web/status/1…](https://twitter.com/i/web/status/1689707069535043584) A good example of that: multiplying digits. LLMs have learned functions to perform multiplication, but 1. it took humongous amounts of data to learn them, 2. those functions generalize quite poorly. They're tied to specific number distributions, and even when operating… [twitter.com/i/web/status/1…](https://twitter.com/i/web/status/1689707778817024000) Meanwhile, a program synthesis engine with a basic DSL that contains `add` and `for` can use just a handful of training examples to synthesize a multiplication function that will generalize *exactly*, to any new input -- not even tied to a specific input distribution. --- Date: 20230813 Links to: Tags: References: * [Post / X](https://twitter.com/fchollet/status/1689700406824419334)