# Intelligence and Generalization This references Chollet’s original talk [Abstraction and Reasoning in AI Systems](Abstraction%20and%20Reasoning%20in%20AI%20Systems.md). Key ideas from his follow up talk on ML street talk: We will *combine* type 1 and type 2 reasoning into a **hybrid** system. We can use things like deep learning to provide *guidance* to the discrete search process. Consider the space of *all possible programs*. This may be *interpretable*, meaning that a deep learning system could provide *direction* to the discrete search system!  The purpose of [Abstraction (Computer Science)](Abstraction%20(Computer%20Science).md) is to erase the irrelevant differences between a thing and focus on the commonalities that actually matter. For instance, consider if we have a line with a small squiggle in it. A proper abstraction for a line should abstract away the squiggle because it is irrelevant, leaving us with only a line.  We can write Chollet’s main point as: > **Program based abstraction** is more powerful than **geometric based abstraction** because topology is more robust to small perturbations. We actually have *analogies*. We have *functions* and *abstractions* in our mind that will take away all of the irrelevant differences so we are left with only what is salient and generalizable. Just as you can embed discrete programs in a smooth manifold (e.g. think of embedding a discrete graph in $\mathbb{R}^n$), you can also do the reverse. You can encode an approximation of a geometric space using discrete constructs. This is exactly what you are doing if you have ever done linear algebra on a computer! Under the hood of a computer we are simply dealing with $1$‘s and $0$‘s! Yet somehow we are able to have vectors of seemingly continuous numbers, you can compute distance between two vectors and so on. All of this is an approximation that is grounded in discrete programs. Chollet thinks that the brain may be doing type 1 and type 2 reasoning using a discrete system. This is because it is actually much easier to do type 1 via an approximation of a geometric space is encoded in a discrete structure than it is to do the reverse. Deep learning is a class of continuous parametric models (in fact they are smooth because they are differentiable) trained via gradient descent. AI in the future will have *two engines*: the first being **NN’s trained via gradient descent** and the second being **discrete search** Chollet states that there are *three types of priors*: 1. Low-level Sensorimotor priors 2. Meta-Learning Strategies 3. High-Level Knowledge Chollet also makes the case in his paper that: > Intelligence is the *efficiency* with which we transform prior information and experience into task solutions. Why are we even trying to define and measure intelligence in the first place? Well it is useful because it is *actionable*! A good definition should be actionable. It should help you think, it should help you find solutions, it should help you make progress. In particular, a good definition will help you highlight the key challenges and help you think about it, and a good measure will give you an actionable feedback signal towards building the right kind of system. Chollet states that intelligence is embodied. You have a brain inside a body interacting with an environment. 1. **Intelligence** is **generalization** 2. Deep learning is really good for **value centric abstraction**. This is due to the manifold hypothesis (lots of natural data has some kind of manifold that you can interpolate on). However, lots of discrete problems do *not* have that. 3. Instead of thinking about individual brains we should think about systems and bottlenecks in systems. ### On the Measure of Intelligence * DL systems learn shortcuts and don't learn the kind of rules that humans would learn with respect to the same task. --- Date: 20221211 Links to: [Abstraction and Reasoning in AI Systems](Abstraction%20and%20Reasoning%20in%20AI%20Systems.md) Tags: #review References: * [#51 FRANCOIS CHOLLET - Intelligence and Generalisation - YouTube](https://www.youtube.com/watch?v=J0p_thJJnoo) * [On “On the Measure of Intelligence” by F. Chollet (2019) - Rob’s Homepage](https://roberttlange.github.io/posts/2020/02/on-the-measure-of-intelligence/) * [[1911.01547] On the Measure of Intelligence](https://arxiv.org/abs/1911.01547)