# Deep Learning The story of deep learning is one of representations (/posts/see [Big Idea Representation](Big%20Idea%20Representation.md)). If we think about language translation models, in general we may think of translating from English to French. We can think about this as the sentence (spoken in English) being the consequence of some *thought* that the speaker had. This thought is in a sense language agnostic. It is the semantic content of that phrase. If you can figure out the thought (a language agnostic representation of the underlying meaning) and then imagine what it would look like for a french speaker to produce words corresponding to that same thought. Then, you would have a very powerful way of doing language translation! We can summarize this big idea as follows: > Our goal is to learn an **intermediary representation** that is in a sense universal and [Invariant](Invariant.md) to language. In general when we think about [Machine-Learning](Machine-Learning.md) in the classic sense, it is a problem of predicting $y$ from $x$. It looks something like this:  But the story about machine translation gives a different view of what machine learning may actually be doing. In practice, $x$ is not just a number, it could be a sentence, an image, a sound, etc. These things are all far more complex than just a number, they contain far more structure inside of them. The big reason why techniques like DL are so powerful is that they can acquire representations of these inputs that are suitable for making meaningful complex predictions. The power of deep learning lies in its ability to **learn** such **representations** automatically. We can think of machine learning as basically a computer program with some free parameters. Traditionally, we may consider an approach where we manually create features and then from that let an algorithm (set of rules) make predictions. So traditional ML essentially has a set of manually created features, which then an algorithm learns some parameters on top of, followed by a prediction.  Deep learning on the other hand allows us to *learn* the features automatically. DL allows us to learn successive intermediary representations/features that become more and more *abstract* and *high level*. Why is this important? Because it allows us to learn an *invariant* representation! So we can summarize DL as follows: > Deep Learning is Machine Learning with **multiple layers** of **learned representations**. The **function** that represents the transformation from input to internal representation to output is usually a deep neural network. (Note: this a bit circular, because almost all *multi-layer parametric* functions with *learned parameters* can be called neural networks). The parameters for every layer are *usually* (but not always!) trained with respect to the overall task objective (e.g. **accuracy**). ### What makes deep learning work? Remember, a big part of the effectiveness of DL is it's ability to learn representations. Good representations are high level representations, they are abstract, they are invariant to nuisances, capture meaningful higher level features, objects, concepts, semantics, meaning and so on. With that said, DL requires: 1. **Big** models with many layers 2. **Large** datasets with many examples 3. Enough **compute** to handle all of the above ### 3b1b Series * A way to think about the gradient vector of our massive cost function, $-\nabla {C(\vec{W})}$, is that it encodes the relative importance of each weight and bias. In other words which of these parameters will give you the most bang for your buck if you change it. This really is just another way of thinking about direction. * Back propagation visual:  [Convolutional Neural Network](Convolutional%20Neural%20Network.md) --- Date: 20211118 Links to: [AI MOC](AI%20MOC.md) [Machine-Learning](Machine-Learning.md) [003-Data-Science-MOC](003-Data-Science-MOC.md) Tags: References: * [Good intro course](https://www.youtube.com/playlist?list=PL_iWQOsE6TfVmKkQHucjPAoRtIJYt8a5A)