# 🌀 Eigenvectors and Eigen-Decomposition

By Better Wise

Last lesson we projected one vector onto another and used those projections to fit a least squares line, always treating a matrix as an action on the plane. Today we ask a sharper question about that action: are there directions the matrix refuses to rotate? Those directions turn out to be the skeleton of the matrix.

By the end of this lesson you can:

- Recognise an eigenvector as a direction the matrix does not rotate
- Interpret the eigenvalue as a stretch, shrink or flip factor
- Compute eigenpairs with `np.linalg.eig` and `np.linalg.eigh`, then verify them
- Explain why $A = P D P^{-1}$ means "diagonal in its own basis"

## Most vectors turn, a few stay on their line

Take the matrix

$$
A = \begin{bmatrix} 2 & 1 \\ 0 & 3 \end{bmatrix}
$$

and feed it a few vectors.

| Input $v$ | $A v$ | Still on the same line? |
| --- | --- | --- |
| $[1, 0]$ | $[2, 0]$ | yes, twice as long, so $\lambda = 2$ |
| $[1, 1]$ | $[3, 3]$ | yes, three times as long, so $\lambda = 3$ |
| $[1, 2]$ | $[4, 6]$ | no, it was knocked off its line |

Most vectors get turned. A few only get scaled. Those few have a name: $v$ is an **eigenvector** of $A$ when $v \neq 0$ and

$$
A v = \lambda v
$$

for some number $\lambda$, the **eigenvalue**. Nothing else is required: no unit length, no particular sign, no special coordinates.

```mermaid
flowchart TD
    V["➡️<br/>Vector v"] --> A["🔧<br/>Apply A"]
    A --> Q{"Still on the<br/>same line?"}
    Q -->|Yes| E["📏<br/>Eigenvector, Av = λv"]
    Q -->|No| N["🌀<br/>Direction changes"]
```

## What the eigenvalue number does

The eigenvalue is the whole story of *how* that direction is treated:

- $\lambda > 1$: stretch along that line
- $0 < \lambda < 1$: shrink along that line
- $\lambda < 0$: flip to the opposite side, then scale
- $\lambda = 0$: the direction collapses to zero, so the determinant is zero and $A$ has no inverse

For a triangular matrix, the eigenvalues sit right on the diagonal. That is why the example above gave 2 and 3 so easily.

**Try it.** Feed $A$ above the vector $[2, 1]$. Is it an eigenvector?

**Click to reveal the answer**

$A[2,1] = [2\cdot 2 + 1\cdot 1,\; 3\cdot 1] = [5, 3]$. If this were a multiple of $[2,1]$, the second coordinate would force $\lambda = 3$, but then the first coordinate would be 6, not 5. So no, it is not an eigenvector.

## Finding eigenpairs in NumPy

```python
import numpy as np

A = np.array([[2.0, 1.0], [0.0, 3.0]])
vals, vecs = np.linalg.eig(A)

print(vals)   # [2. 3.], the order is not guaranteed
print(vecs)   # eigenvectors live in the COLUMNS

for lam, v in zip(vals, vecs.T):
    print(np.allclose(A @ v, lam * v))   # True, True
```

Two habits to build immediately:

1. Eigenvectors come back as **columns** of `vecs`, not rows. Use `vecs[:, i]`.
2. Always verify with `A @ v` against `lam * v`. It costs one line and catches every mix-up.

For a real symmetric matrix, prefer `np.linalg.eigh`. It returns real eigenvalues in ascending order and orthonormal eigenvectors:

```python
S = np.array([[2.0, 1.0], [1.0, 2.0]])
vals, vecs = np.linalg.eigh(S)
print(vals)   # [1. 3.]
```

**Try it.** For $D = \begin{bmatrix} 4 & 0 \\ 0 & -1 \end{bmatrix}$, name the eigenpairs by eye, then confirm with NumPy. What does $\lambda = -1$ do to its eigenvector?

**Click to reveal the answer**

The coordinate axes are the eigenvectors: $[1,0]$ with $\lambda = 4$ (stretched four times) and $[0,1]$ with $\lambda = -1$ (same length, flipped to the opposite side). `np.linalg.eigh` returns `[-1., 4.]` with the matching axes as columns.

## The same matrix, seen from its eigenbasis

Put the eigenvectors into the columns of a matrix $P$, and the eigenvalues on the diagonal of $D$. Then

$$
A = P D P^{-1}
$$

This is the **eigen-decomposition**. Read it as a recipe, not as algebra:

1. Move into eigen-coordinates: `x_eig = P_inv @ x`
2. Stretch along the eigen-axes: `y_eig = D @ x_eig`
3. Return to standard coordinates: `y = P @ y_eig`

In its own basis, the matrix is diagonal. All the turning disappears and only stretching remains. This is exactly the change of basis from Module 2, applied to the directions the matrix already prefers.

Two cheap sanity checks on any result:

- $\text{trace}(A) = \sum \lambda_i$
- $\det(A) = \prod \lambda_i$

**Try it.** $A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$ has $\lambda = 3$ with eigenvector $[1,1]$ and $\lambda = 1$ with eigenvector $[1,-1]$. In eigen-coordinates, where does $A$ send the point $(2, 5)$?

**Click to reveal the answer**

To $(6, 5)$: multiply by the eigenvalue of each axis, $2 \cdot 3 = 6$ and $5 \cdot 1 = 5$. Notice how much less work that was than multiplying by $A$ directly.

Fill these in from memory before moving on:

An eigenvector of a matrix A is a nonzero vector whose direction is unchanged, so that A v equals _____ for some scalar λ.
The scalar λ is called the _____ and it says how much the eigenvector is stretched, shrunk, or flipped.
In NumPy, np.linalg.eig returns the eigenvectors as the _____ of its second output array.
For a symmetric matrix, np.linalg.eigh is preferable because it returns _____ eigenvalues in ascending order with orthonormal eigenvectors.
With P holding the eigenvectors as columns and D holding the eigenvalues on its diagonal, the decomposition is A = {{c5::P D P^{-1}}}.
The trace of A equals the _____ of its eigenvalues, and its determinant equals their product.

## Reading eigen-results without getting fooled

- **Columns, not rows.** Every eigenvector bug I see in office hours is a transposed `vecs` array.
- **Sign and scale are free.** NumPy normalises eigenvectors to length 1 and picks a sign arbitrarily. $-v$ is just as valid as $v$.
- **`eig` does not sort.** Pair `vals[i]` with `vecs[:, i]`, never with a sorted copy.
- **Repeated eigenvalues are a warning.** You may get fewer independent eigenvectors than the matrix size, and then $P^{-1}$ does not exist.
- **Some real matrices have no real eigenvectors.** A 90 degree rotation turns every vector, so `eig` returns complex numbers. Do not force them to be real.
- **Rule of thumb:** verify with `A @ v`, then check trace and determinant. Two checks, ten seconds, no silent errors.

## Check yourself before the checkpoint

Front: What is an eigenvector of A?
Back: A nonzero vector v with A v = λ v, so A does not change its direction.

Front: What does the eigenvalue tell you geometrically?
Back: How much that direction is scaled: above 1 stretches, between 0 and 1 shrinks, negative flips.

Front: What does λ = 0 mean for the matrix?
Back: That direction collapses to zero, det(A) = 0, and A has no inverse.

Front: Where are the eigenvectors in np.linalg.eig output?
Back: In the columns of the second returned array, so vecs[:, i].

Front: Which NumPy function suits a real symmetric matrix?
Back: np.linalg.eigh, which gives real eigenvalues ascending with orthonormal eigenvectors.

Front: Does np.linalg.eig sort the eigenvalues?
Back: No, the order is unspecified; eigh returns them ascending.

Front: One-line verification of an eigenpair (λ, v)?
Back: Check np.allclose(A @ v, lam * v).

Front: Is an eigenvector unique?
Back: No, any nonzero multiple is also one; NumPy returns length 1 with an arbitrary sign.

Front: What is A in terms of P and D?
Back: A = P D P^{-1}, with eigenvectors in the columns of P and eigenvalues on the diagonal of D.

Front: What does A look like in its own eigenbasis?
Back: A diagonal matrix: pure stretching along the eigenvector axes, no turning.

Front: Trace and determinant in terms of eigenvalues?
Back: Trace equals the sum of the eigenvalues, determinant equals their product.

Front: Can a real matrix lack real eigenvectors?
Back: Yes, a rotation such as [[0, -1], [1, 0]] turns every vector, so its eigenvalues are complex.

Front: Why is a repeated eigenvalue a warning?
Back: You may get too few independent eigenvectors, so P is not invertible and A cannot be diagonalised.

## What to carry forward

- An eigenvector is a direction $A$ only scales; the eigenvalue is the scale factor, including flips and collapses.
- `np.linalg.eig` puts eigenvectors in columns and does not sort them; `np.linalg.eigh` is the symmetric specialist.
- $A = P D P^{-1}$ says the matrix is diagonal once you look from its own basis, and trace and determinant let you check the result fast.

Eigen-decomposition only works cleanly when the matrix has a full set of independent eigenvectors. When it does not, the next lesson's tool still works: the SVD builds an orthonormal eigenbasis for any matrix, square or not, full rank or not.

**Where we go next:** in *A First Look at SVD* we factor any matrix as $A = U \Sigma V^T$ and read its singular values as the stretch factors of the directions $A$ cares about most, which is eigen-decomposition's idea generalised to every shape.

---

## Contents

Part of [🧮 Mathematics for Machine Learning 1: Linear Algebra Foundations](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations.md)

- **📐 Vectors, Norms and Similarity**
  - [➡️ Vectors as Arrows, Points, and NumPy Arrays](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/vectors-as-arrows-points-and-numpy-arrays.md)
  - [📏 Norms and Distances: Measuring Vector Size](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/norms-and-distances-measuring-vector-size.md) (sign-in required)
  - [🎯 Dot Product and Cosine Similarity](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/dot-product-and-cosine-similarity.md)
  - [✅ Checkpoint: Vectors and Similarity](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/checkpoint-vectors-and-similarity.md)
- **🔄 Matrices and Transformations**
  - [🧱 Matrices as Transformations and Matrix Multiplication as Composition](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/matrices-as-transformations-and-matrix-composition.md)
  - [🧭 Inverse, Rank, and Span](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/inverse-rank-and-span.md)
  - [🔀 Basis and Change of Basis](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/basis-and-change-of-basis.md)
  - [✅ Checkpoint: Matrices and Transformations](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/checkpoint-matrices-and-transformations.md)
- **🔆 Projections, Eigenvectors and SVD**
  - [📉 Projections and Least Squares Intuition](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/projections-and-least-squares-intuition.md)
  - [🌀 Eigenvectors and Eigen-Decomposition](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/eigenvectors-and-eigen-decomposition.md) — this lesson
  - [🔍 A First Look at SVD](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/first-look-at-svd.md) (premium content)
  - [✅ Checkpoint: Projections, Eigen and SVD](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/checkpoint-projections-eigen-and-svd.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/cheatsheet.md) (premium content)