# 🗄️ Capstone Step 2: Storage, Content Hashing and Integrity Checks

By Lê Nguyễn Tuân

The previous lesson ended with a video downloader that reads a manifest, picks a resolution and concatenates fragments into a byte stream. Your image downloader from 2.1 does the same, in chunks. Both now hand you bytes.

This step is about what happens to those bytes after the last chunk arrives. A crawler that saves files is a script. A crawler whose files you can still trust six months later, and which does not store the same photo 40 times, is a tool. That is the difference we build here.

## The storage contract your download stage must satisfy

Four guarantees, each one a bug you would otherwise meet later:

- **Deterministic path**: the same bytes land at the same path, on any machine, on any day. The path derives from the content hash, never from the URL, the timestamp, or the post ID.
- **One file per unique content**: if three Weibo accounts repost the same image, you keep one file and three references.
- **No partial files in the store**: a killed process or a timeout must never leave behind a file that looks complete.
- **Type and size verified after transfer**: a `200` response carrying an HTML login page is not a JPEG, even when the URL ends in `.jpg`.

```tree
data/
  tmp/
  media/
    image/
      3f/
        9a/
          3f9a...e1.jpg
  index.sqlite
```

The two-level shard uses the first two and next two hex characters of the hash. That gives 65,536 leaf directories, so millions of files never pile into one folder and slow down `readdir`.

### Naming: hash first, then name

You cannot name a file by its hash before you have its bytes, so the order is fixed.

1. Write the response to `data/tmp/<random>.part`, feeding every chunk to SHA-256 as it arrives.
2. Flush and `fsync`, then run size and magic-byte validation on the temp file.
3. Build the final path `media/<kind>/<h[0:2]>/<h[2:4]>/<hash>.<ext>`, with `ext` taken from the sniffed format.
4. If the final path already exists, delete the temp file and record only a reference.
5. Otherwise call `os.replace(temp, final)`, an atomic rename on the same filesystem.

```python
def store_stream(resp, kind, cfg):
    tmp = cfg.tmp_root / f"{uuid4().hex}.part"
    h, size = hashlib.sha256(), 0
    with open(tmp, "wb") as f:
        for chunk in resp.iter_content(1 << 16):
            h.update(chunk)
            f.write(chunk)
            size += len(chunk)
        f.flush()
        os.fsync(f.fileno())
    digest = h.hexdigest()
    ext = validate(tmp, kind, size, cfg)   # raises + deletes tmp on mismatch
    final = cfg.media_root / kind / digest[:2] / digest[2:4] / f"{digest}.{ext}"
    final.parent.mkdir(parents=True, exist_ok=True)
    os.replace(tmp, final)
    return digest, final
```

Hashing inside the write loop is free: one pass over the bytes. Hashing after the move means reading the whole file a second time, and letting unverified content into the store first.

| Check | Rule | Failure means |
| --- | --- | --- |
| Length | bytes written equals `Content-Length` when that header exists | truncated transfer |
| Bounds | size inside configured min/max for that media kind | placeholder or error page |
| Magic bytes | leading bytes match the claimed format | server returned HTML or JSON |
| Extension | derived from sniffed format, not from the URL | `.jpg` URL serving WebP |

Signatures worth hardcoding: JPEG `FF D8 FF`, PNG `89 50 4E 47 0D 0A 1A 0A`, GIF `47 49 46 38`, WebP `52 49 46 46` at offset 0 with `57 45 42 50` at offset 8, WebM `1A 45 DF A3`, and MP4 with `66 74 79 70` (`ftyp`) at offset 4.

### Dedupe at two moments

Before the transfer, a canonical URL to hash lookup can skip the download entirely. After the transfer, a hash that already exists means the content is new to this URL but not to your store, so you keep the existing file and add one row.

```sql
CREATE TABLE refs (
  source_url TEXT PRIMARY KEY,
  sha256     TEXT NOT NULL,
  kind       TEXT NOT NULL,
  bytes      INTEGER NOT NULL,
  fetched_at TEXT NOT NULL
);
```

That table is also the seed for the incremental sync work later: the URL is the natural key, the hash is the identity of the content itself.

**Exercise.** A response returns `200`, `Content-Type: image/jpeg`, `Content-Length: 12480`, and the body starts with `<!DOCTYPE html`. Which check catches it, and what should the code do?

**Check the answer**

The magic-byte sniff catches it: `3C 21 44 4F` is not `FF D8 FF`. `Content-Type` alone would have passed it, so treat the header as a hint and the bytes as truth. The stage should delete the temp file, log the URL with a `type_mismatch` reason, and count it as a failed task rather than retrying forever.

## Order of work for this step

1. Extend the config with `media_root`, `tmp_root`, `min_bytes` and `max_bytes` per kind, and the allowed formats list.
2. Write `validate(path, kind, size, cfg)`, returning the extension or raising a typed error.
3. Write `store_stream` as above, so images and videos share one path to disk.
4. Create the `refs` table and insert with conflict handling, so a repeated URL updates rather than explodes.
5. Wire both downloaders from 2.1 and 2.2 through `store_stream`, deleting their ad hoc `open(..., "wb")` calls.
6. Add a `verify-store` command that walks `media/`, re-hashes each file, and compares the result with the file name.

**Exercise.** After a second full crawl, `refs` holds 4,000 rows but only 3,900 distinct hashes. Is that a defect?

**Check the answer**

No. It means 100 URLs served content you already had, which is exactly the duplication the store is designed to absorb. The real signal to watch is a hash appearing under two incompatible kinds, or a file whose re-hash disagrees with its name, and `verify-store` is what reports both.

## How you know the step is done

- Interrupt a download mid-stream: `media/` contains no new file, only a `.part` that your cleanup sweep removes on the next run.
- Crawl the same feed twice: the second run downloads zero bytes for known URLs and adds no rows.
- Feed the stage a fake JPEG whose body is HTML: rejected, temp deleted, failure logged with URL and reason.
- Two different URLs serving one image: one file on disk, two rows in `refs`.
- `verify-store` reports zero mismatches, and rebuilding the index from disk reproduces the same paths.

## Traps that leave corrupt or duplicated files

- Naming by URL, post ID or timestamp: every crawl becomes a fresh copy, and storage grows without bound.
- Trusting the URL extension: WebP and AVIF are routinely served from `.jpg` links.
- Trusting `Content-Type` alone: error and login pages often echo the type the client asked for.
- Writing straight to the final path: a killed process leaves a half file that later looks valid and gets skipped by dedupe.
- Putting `tmp_root` on another filesystem from `media_root`: `os.replace` then fails, and the fallback is a non-atomic copy.
- Hashing after the move: an extra full read, plus a window where unverified content sits in the store.

## Where the capstone stands now

You have a download stage with content-addressed storage, a two-moment dedupe rule, and validation that runs on every transfer, so a video fragment join and a single image both end in the same trustworthy place. The next lesson is the module checkpoint, where you audit this stage end to end against the image and video downloaders you built.

Front: Why name a media file by its content hash instead of its URL?
Back: Same bytes always map to one path, so reposts do not create copies and paths are reproducible on any machine.

Front: Where should hashing happen during a download?
Back: Inside the write loop, so each chunk is hashed and written in a single pass.

Front: Why write to data/tmp and then os.replace into media/?
Back: The rename is atomic, so the store never contains a half-written file.

Front: What does the two-level shard directory use?
Back: The first two and next two hex characters of the content hash.

Front: Which check catches an HTML login page served as image/jpeg?
Back: Magic-byte sniffing of the leading bytes, not the Content-Type header.

Front: Where does the stored file extension come from?
Back: The sniffed format, never the URL path.

Front: What does a bytes-written versus Content-Length mismatch indicate?
Back: A truncated transfer, so the temp file must be discarded.

Front: At which two moments do you check for duplicates?
Back: Before the transfer (canonical URL to hash) and after it (hash already present in the store).

Front: What are the columns of the refs table?
Back: source_url as primary key, sha256, kind, bytes, fetched_at.

Front: What are the PNG magic bytes?
Back: 89 50 4E 47 0D 0A 1A 0A at offset 0.

Front: At what offset does the MP4 ftyp signature sit?
Back: Offset 4, after the 4-byte box size.

Front: What does re-running a crawl over an unchanged feed reveal?
Back: Zero bytes downloaded and no new refs rows, which proves dedupe works.

The pipeline now moves bytes through SHA-256, then _____ rename into a content-addressed path, so a crash never publishes a partial file. The dedupe lookup happens at two moments: before the transfer via the canonical URL, and after it when the _____ already exists. Every stored file name ends with an extension taken from _____, not from the URL. Temp files must live under _____ on the same filesystem as `media/`, otherwise the rename stops being atomic. A file whose re-hash disagrees with its own name is reported by _____.

---

## Contents

Part of [🎞️ Media Crawler Architecture: Queue, Storage and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling.md)

- **🧭 Foundations and Capstone Skeleton**
  - [🔍 How Facebook, Weibo and Bilibili Serve Media](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/media-serving-anatomy-three-platforms.md)
  - [🧱 Capstone Blueprint: Config, Storage Layout, Pipeline Modules](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-blueprint-config-storage-pipeline.md) (sign-in required)
  - [🧩 Capstone Step 1: Normalize Media Records from Feed Responses](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-normalize-media-records.md)
  - [✅ Checkpoint: Platform Anatomy and Project Skeleton](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-platform-anatomy-skeleton.md)
- **📥 Media Download Engineering**
  - [🌊 Streaming Image Downloads: Chunks, Resume, Retries, Timeouts](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/streaming-image-downloads-resume-retries.md)
  - [🎥 Video Downloads: Manifests, Fragments, Resolution Selection](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/video-downloads-manifests-fragments-resolution.md)
  - [🗄️ Capstone Step 2: Storage, Content Hashing and Integrity Checks](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-storage-hashing-integrity.md) — this lesson
  - [✅ Checkpoint: Download Engineering](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-download-engineering.md)
- **🏗️ Architecture, Scheduling and Scaling**
  - [🔁 Queue-Based Workers: Task Model, Backpressure, Rate Budget](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/queue-workers-task-model-backpressure.md)
  - [🚀 Capstone Step 3: Observability, Recovery and Scaling Out](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-observability-recovery-scaling.md) (premium content)
  - [✅ Checkpoint: Architecture and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-architecture-scaling.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/cheatsheet.md) (premium content)