# Singular Value Decomposition - Outline ### Key ideas I want to convey * Linearity is a constraint * It conveys a lot (or a little) information, depending on how you view it * Dot product has an implicit direction (i.e. dot product of v and u conveys how much of v is in direction of u, and how much of u is in direction of v. It is a scalar quantity denoting their similarity. There is no direction in the output, but it is implicitly tied to the input arguments) * inverse * [Matrix Transpose and the Four Fundamental Subspaces - YouTube](https://youtu.be/yfj8uMwAgrI?t=261) * can't get back to original vector, but can only get back to vector on red line   * For fundamental subspaces (and transpose) * Data vs maps * Jkun svd post 1 * Mechanics and better approximations * Jkun svd post 2 * Towards and exploratory medium post * Fundamental subspaces and matrix transpose: * [Matrix Transpose and the Four Fundamental Subspaces - YouTube](https://www.youtube.com/watch?v=yfj8uMwAgrI&t=287s) * Notability: feature column of AMS * Good notes matrix transform break down: * but in addition to projecting onto row vector, also consider eigenchris ([here](https://www.youtube.com/watch?v=LNoQ_Q5JQMY)) way of representing row vector as line (since they are indeed functions!) * See [here](https://youtu.be/LNoQ_Q5JQMY?t=411), there is a set of inputs that a row vector (dot product?) will map to a single value *  * Notice that if we draw the vector [2,1] here that the arrow will be orthogonal to all lines in the stack: *  * Example: Movie recommendation. We have users (vector), movies (vector), but are the recommendations themselves vectors? ### The Geometric Essence of SVD > This is the geometric essence of SVD. _Any_ linear transformation can be thought of as simply stretching or compressing or flipping a square, provided we are allowed to rotate it first. The transformed square or rectangle may have a new orientation after the transformation. > - [Singular Value Decomposition as Simply as Possible](https://gregorygundersen.com/blog/2018/12/10/svd/) ### Proof of SVD [Singular Value Decomposition as Simply as Possible](https://gregorygundersen.com/blog/2018/12/10/svd/) This is a good proof. Here is what I don't like/would improve: * Doesn't make it clear that the matrix $M$ corresponds to a linear map from a vector space $V$ to itself (where $V = \mathbb{R}^2$). * Doesn't specify what bases $M$ is tied to. Without that it makes it hard to know what we are changing *from* and going *to* * I think he made a mistake? I don't think that we should have said we are using a matrix, $M$, because we have never really picked bases. We should have just used $L$ the whole time. Let's try this. * Or maybe the idea is that *it doesn't matter what basis we are starting with*. It could be the standard basis, or something else. What matters is that $M$ is defined wrt some basis, and if we follow the correct rules we can *change its starting bases* to ones that allow it to have a new representation that is diagonal. * So maybe the key thing to point out is simply that $M$ has some basis we are starting with, it could be the standard basis, but it doesn't need to be. * I don't love that he is *not* specifying the basis of $x$ and $v_1$ / $v_2$ when performing the projection. My guess is that everything is just described wrt the standard basis, but that leads to a huge lack of clarity in this context. (99.9% certain we are in standard basis, or else dot product wouldn't make sense [Dot Product Same Basis](https://chat.openai.com/share/7507fe37-df5b-414c-a469-62dc6e794888)) ### Notes Great explanation by Chris olah [https://overcast.fm/+BBPwiIjQso/15:04](https://overcast.fm/+BBPwiIjQso/15:04) Everything is with respect to a basis! Including rotation. This is why we have to change basis at the end! We had a basis of v Then allly the stretching of A Then need to apply remainder of A but not stretch, hence divide by singular values (stretch of A). This changes basis back!!! --- Date: 20220612 Links to: Tags: #review References: * [Singular Value Decomposition](Singular%20Value%20Decomposition.md) * [Four Fundamental Subspaces](Four%20Fundamental%20Subspaces.md) * [svd visualization.excalidraw](svd%20visualization.excalidraw.md)