# 📉 Projections and Least Squares Intuition

By Better Wise

## From reachable outputs to closest points

In Module 2 you learned to read a matrix as a transformation, with its columns spanning everything the transformation can reach: the column space. This lesson asks the practical follow-up question. What if the output you want is not reachable? The answer is one geometric move, dropping a perpendicular, and that single move is the whole idea behind least squares. You will lean on the dot product from Lesson 1.3, because it measures how much of one vector points along another.

By the end of this lesson you should be able to:

- compute the projection of one vector onto another, by hand and in NumPy
- read a projection as the shadow one arrow casts along another
- project a vector onto the column space of a matrix
- explain geometrically why the least squares solution is the closest reachable point

### Dropping a perpendicular: projecting one vector onto another

Draw two arrows, **a** and **b**, from the same origin. Now shine a light straight down onto the line through **b**. The shadow of **a** on that line is the **projection** of **a** onto **b**. The piece left over, **a** minus its shadow, is the **residual**, and it meets the line at a right angle.

| Quantity | Formula | What it means |
| --- | --- | --- |
| Scalar projection | $\frac{a \cdot b}{|b|}$ | signed length of the shadow |
| Vector projection | $\frac{a \cdot b}{b \cdot b}\, b$ | the shadow itself, as a vector |
| Residual | $a - \frac{a \cdot b}{b \cdot b}\, b$ | the part of **a** perpendicular to **b** |

The vector projection of **a** onto **b** is _____, and the leftover piece **a** minus that projection is called the _____.

**Worked example.** Take **a** = (3, 4) and **b** = (1, 1). Then **a**·**b** = 7 and **b**·**b** = 2, so the scale factor is 3.5 and the projection is (3.5, 3.5). The residual is (-0.5, 0.5), and its dot product with **b** is zero, which confirms the right angle.

```python
import numpy as np

a = np.array([3.0, 4.0])
b = np.array([1.0, 1.0])
scale = (a @ b) / (b @ b)      # 3.5
proj = scale * b               # [3.5 3.5]
resid = a - proj               # [-0.5 0.5]
print(proj, resid, resid @ b)  # the last value is ~0.0
```

> **Tip:** 
> Divide by `b @ b`, not by `np.linalg.norm(b)`. Dividing by the norm gives a length, not a vector. Using the dot product with itself also keeps square roots out of your arithmetic.

#### Practice

Find the projection of **a** = (2, 6) onto **b** = (3, 3), then check the residual.

**Click to reveal the answer**

**a**·**b** = 2·3 + 6·3 = 24 and **b**·**b** = 18, so the scale is 24/18 = 4/3. The projection is (4, 4). The residual is (2, 6) - (4, 4) = (-2, 2), and (-2, 2)·(3, 3) = -6 + 6 = 0.

### Projecting onto a whole column space

Now replace the single direction **b** with the columns of a matrix $A$. Solving $Ax = b$ exactly means landing on **b** itself, which only works when **b** already sits inside the column space. When it does not, the closest point you can reach is the projection of **b** onto that column space. Geometrically you drop a perpendicular from **b** down onto the plane or line spanned by the columns.

```mermaid
flowchart TD
    B["🎯<br/>Target b"] --> Q{"b inside the<br/>column space of A?"}
    Q -->|yes| E["✅<br/>Exact solution, residual zero"]
    Q -->|no| P["📐<br/>Drop a perpendicular, p is the projection"]
    P --> N["🧮<br/>Solve AᵀA x = Aᵀb"]
    N --> C["📍<br/>Closest reachable point"]
```

The projection matrix onto the column space of $A$ is _____. The vector $\hat{x}$ that produces it satisfies the **normal equations**:

$$
A^\top A \hat{x} = A^\top b
$$

Those equations simply say "the residual is orthogonal to every column", that is $A^\top (b - A\hat{x}) = 0$.

**Tiny example.** Let $A$ have the single column (1, 1) and let **b** = (3, 1). The scale along (1, 1) is (3 + 1)/2 = 2, so the projection is (2, 2) and the residual is (1, -1), which is perpendicular to (1, 1). In NumPy:

```python
A = np.array([[1.0], [1.0]])
b = np.array([3.0, 1.0])
x, *_ = np.linalg.lstsq(A, b, rcond=None)  # x = [2.]
p = A @ x                                   # [2. 2.]
print(A.T @ (b - p))                        # ~[0.]
```

#### Practice

Project **b** = (1, 2, 3) onto the span of the single column (1, 1, 1). Which familiar quantity appears?

**Click to reveal the answer**

The scale is (1 + 2 + 3)/3 = 2, so the projection is (2, 2, 2) and the residual is (-1, 0, 1). Projecting onto the all-ones direction returns the average of the entries, which is 2 here.

### Why the least squares answer is the closest reachable point

Least squares minimizes the squared length of the residual, $||Ax - b||^2$. Suppose the residual still had some component pointing along the column space. Then you could nudge $x$ in that direction and shorten the error. So the smallest possible error happens exactly when the residual has no component along the column space, meaning it is perpendicular to it. That perpendicular foot is the projection, and the coefficients that produce it are the least squares solution.

**Fitting a line.** For points (-1, 0), (0, 1), (1, 3) and a line $y = c + mt$, the design matrix has a column of ones and a column of $t$ values. The chosen $t$ values make $A^\top A$ diagonal, so the normal equations split apart:

```python
t = np.array([-1.0, 0.0, 1.0])
y = np.array([0.0, 1.0, 3.0])
A = np.column_stack([np.ones_like(t), t])
coef, *_ = np.linalg.lstsq(A, y, rcond=None)
print(coef)          # [1.33333333 1.5]
p = A @ coef         # fitted values
print(A.T @ (y - p)) # residuals orthogonal to both columns, ~[0. 0.]
```

The solution is $c = 4/3$ and $m = 1.5$. The fitted values are about (-0.167, 1.333, 2.833) and the residuals (0.167, -0.333, 0.167). Their dot products with the ones column and with the $t$ column are both zero, exactly as the geometry demands. Least squares minimizes _____, and the solution always satisfies the normal equations _____.

#### Practice

Fit a line to the points (-1, 1), (0, 1), (1, 2) using the same design matrix.

**Click to reveal the answer**

$A^\top A$ is still diagonal with entries 3 and 2. The right side is $A^\top y$ = (4, 1), so $3c = 4$ gives $c = 4/3$ and $2m = 1$ gives $m = 0.5$. The line is $y = 4/3 + 0.5t$. Check the residuals sum to zero, which they must.

Front: Vector projection of a onto b?
Back: (a·b / b·b) b

Front: Scalar projection of a onto b?
Back: a·b / |b|, the signed length of the shadow

Front: What is the residual of a projection?
Back: a minus its projection, the part perpendicular to the direction

Front: Why divide by b·b instead of |b| in the vector projection?
Back: Dividing by the norm gives a length, not a vector, and dot over dot avoids square roots

Front: How do you write the projection of b onto the column space of A?
Back: p = A(AᵀA)⁻¹Aᵀ b

Front: Defining property of the projection residual?
Back: It is orthogonal to every column of A, so Aᵀ(b - p) = 0

Front: What are the normal equations?
Back: AᵀA x̂ = Aᵀb

Front: What quantity does least squares minimize?
Back: The squared residual length ||Ax - b||²

Front: Geometric meaning of the least squares solution?
Back: The closest reachable point to b inside the column space of A

Front: What does np.linalg.lstsq return?
Back: The coefficient vector x̂, not the projection; compute A @ x̂ to get p

Front: When is the least squares solution unique?
Back: When A has full column rank, so AᵀA is invertible

Front: Which distances does a least squares line fit minimize?
Back: Vertical distances between the points and the line, not perpendicular distances

Front: If b already lies in the column space of A, what is the residual?
Back: Zero, so the projection equals b and Ax = b holds exactly

### Projection habits that save you from wrong answers

- Always compute the scale factor `(a @ b) / (b @ b)` in one line, then multiply by `b`. Splitting it wrongly is the most common beginner error.
- Run the free sanity check: `A.T @ (b - p)` should be a vector of zeros. If it is not, your projection is not the closest point.
- Remember that `lstsq` hands you coefficients. Students often print `x` and think they have the fitted values. Multiply by `A`.
- A zero direction vector has no line to project onto, so the formula divides by zero. Guard against it.
- Least squares fits a line by minimizing vertical gaps. If you care about perpendicular distances to the line, that is a different problem.
- If the columns of $A$ are dependent, $A^\top A$ is singular and the answer is not unique. Reach for `lstsq` rather than building the inverse by hand.

### What to carry forward

Three things are worth keeping. A projection is a shadow found by dropping a perpendicular. The residual of that shadow is orthogonal to everything you projected onto. Least squares is therefore not a mysterious formula but the name for the closest reachable point in a column space, with the normal equations as its arithmetic.

Next, in **Eigenvectors and Eigen-Decomposition**, you will look for the special directions that a transformation does not rotate at all, which turns the same geometric instinct toward the matrix itself rather than toward the vector it acts on.

---

## Contents

Part of [🧮 Mathematics for Machine Learning 1: Linear Algebra Foundations](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations.md)

- **📐 Vectors, Norms and Similarity**
  - [➡️ Vectors as Arrows, Points, and NumPy Arrays](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/vectors-as-arrows-points-and-numpy-arrays.md)
  - [📏 Norms and Distances: Measuring Vector Size](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/norms-and-distances-measuring-vector-size.md) (sign-in required)
  - [🎯 Dot Product and Cosine Similarity](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/dot-product-and-cosine-similarity.md)
  - [✅ Checkpoint: Vectors and Similarity](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/checkpoint-vectors-and-similarity.md)
- **🔄 Matrices and Transformations**
  - [🧱 Matrices as Transformations and Matrix Multiplication as Composition](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/matrices-as-transformations-and-matrix-composition.md)
  - [🧭 Inverse, Rank, and Span](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/inverse-rank-and-span.md)
  - [🔀 Basis and Change of Basis](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/basis-and-change-of-basis.md)
  - [✅ Checkpoint: Matrices and Transformations](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/checkpoint-matrices-and-transformations.md)
- **🔆 Projections, Eigenvectors and SVD**
  - [📉 Projections and Least Squares Intuition](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/projections-and-least-squares-intuition.md) — this lesson
  - [🌀 Eigenvectors and Eigen-Decomposition](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/eigenvectors-and-eigen-decomposition.md)
  - [🔍 A First Look at SVD](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/first-look-at-svd.md) (premium content)
  - [✅ Checkpoint: Projections, Eigen and SVD](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/checkpoint-projections-eigen-and-svd.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-1-linear-algebra-foundations/cheatsheet.md) (premium content)