User:BenFrantzDale/Matrix decomposition

I don't like how I was taught eigendecomposition and SVD; it never made sense. Here's a stab at how I would do it...

Motivation: Linear algebra is useful for a startling number of applications...

[Describe the rules of matrix multiplication.]

Diagonal matrices

A diagonal matrix is one that has all its non-zero entries on the diagonal. That is,

$M_{ij}=0\forall i\neq j$.

This has a very practical meaning: It scales things only in axial directions. This is a little like the ellipse-select tool in Photoshop: you can make it wider (x) or taller (y) but you can't make it long and skinny along a diagonal. That is intuitively what it means for a matrix to be diagonal.

Rotation matrices

A rotation matrix is one that rotates vectors about the origin without scaling them. These matrices have a number of interesting properties:

1. The columns are all orthogonal.
2. The columns all have norm 1.
3. $R^{\top }=R^{-1}$.
4. The rows are all orthogonal.
5. The rows all have norm 1.

If a matrix rigidly rotates things, then for all orthogonal pairs of vectors, u, and v, they should still be orthogonal after rotation. That is: $u\cdot v=0=(Ru)\cdot (Rv)$ Suppose R has two non-orthogonal columns, column i and j. Then let u be all zeros except for the ith entry is 1 and v all zeros except the jth entry is 1. Clearly $u\cdot v=0$. Now Ru and Rv are just the ith and jth columns of R respectively, so by contradiction, if the columns were not orthogonal, we could always find orthogonal pairs of vectors that weren't rigidly rotated by R.

For 2, if a matrix doesn't scale vectors, only redirects them, then

$\|Rv\|=\|v\|\forall v$

Suppose c is the ith column of R and has norm not equal to 1. Then let u be the same vector, all zeros except the ith element is 1. Clearly $\|u\|=1$. But then

$Ru=c$ and so $\|Ru\|=\|c\|\neq \|u\|=1$.

So by contradiction, all columns must have norm 1.

Three is interesting: The transpose is the inverse. Consider again u being all zeros except for a 1 in the ith entry. Then Ru=c is the ith column of R. What should $R^{-1}$ look like? Well, to get $u=R^{-1}Ru=R^{-1}c$,the ith row of $R^{-1}$ will have to dot with c to get 1. The only vector that will do that is c (which, as we established, has a norm of 1). All of the other rows must be orthogonal to c, and this has to hold for all i. Clearly $R^{\top }$ has the property that the ith row equals the ith column of R. Similarly the other rows of $R^{\top }$ are the other columns of R so we know they are orthogonal!

Four and five follow from one through three: if the transpose is a rotation matrix, then any column properties apply to the rows too.

Symmetric matrices

Symmetric matrices are matrices for which $M_{ij}=M_{ji}$. These are extraordinarily common in science and engineering and have lots of nice properties. But aside from an array of numbers on a page, what does that mean?

Symmetric positive (semi-)definite matrices

Let's start with positive semi-definite matrices. These are those symmetric matrices that also have the following obtuse property:

$v^{\top }Mv\geq 0\forall v$.

What this really means is that M describes an oriented ellipsoid. Why isn't obvious.

Suppose there's a matrix, A, that scales and possibly turns or squashes v. Given a v, we could ask how much A changes its length. That is, if v is a unit vector, what is $\|Av\|$? Well, it's just

${\sqrt {(Av)^{\top }Av}}={\sqrt {v^{\top }A^{\top }Av}}$.