Capstone Step 2: Storage, Content Hashing and Integrity Checks

By Lê Nguyễn Tuân

The previous lesson ended with a video downloader that reads a manifest, picks a resolution and concatenates fragments into a byte stream. Your image downloader from 2.1 does the same, in chunks. Both now hand you bytes.

This step is about what happens to those bytes after the last chunk arrives. A crawler that saves files is a script. A crawler whose files you can still trust six months later, and which does not store the same photo 40 times, is a tool. That is the difference we build here.

The storage contract your download stage must satisfy

Four guarantees, each one a bug you would otherwise meet later:

• Deterministic path: the same bytes land at the same path, on any machine, on any day. The path derives from the content hash, never from the URL, the timestamp, or the post ID.
• One file per unique content: if three Weibo accounts repost the same image, you keep one file and three references.
• No partial files in the store: a killed process or a timeout must never leave behind a file that looks complete.
• Type and size verified after transfer: a 200 response carrying an HTML login page is not a JPEG, even when the URL ends in .jpg.

data/
  tmp/
  media/
    image/
      3f/
        9a/
          3f9a...e1.jpg
  index.sqlite

The two-level shard uses the first two and next two hex characters of the hash. That gives 65,536 leaf directories, so millions of files never pile into one folder and slow down readdir.

Naming: hash first, then name

You cannot name a file by its hash before you have its bytes, so the order is fixed.

1. Write the response to data/tmp/<random>.part, feeding every chunk to SHA-256 as it arrives.
2. Flush and fsync, then run size and magic-byte validation on the temp file.
3. Build the final path media/<kind>/<h[0:2]>/<h[2:4]>/<hash>.<ext>, with ext taken from the sniffed format.
4. If the final path already exists, delete the temp file and record only a reference.
5. Otherwise call os.replace(temp, final), an atomic rename on the same filesystem.

def store_stream(resp, kind, cfg):
    tmp = cfg.tmp_root / f"{uuid4().hex}.part"
    h, size = hashlib.sha256(), 0
    with open(tmp, "wb") as f:
        for chunk in resp.iter_content(1 << 16):
            h.update(chunk)
            f.write(chunk)
            size += len(chunk)
        f.flush()
        os.fsync(f.fileno())
    digest = h.hexdigest()
    ext = validate(tmp, kind, size, cfg)   # raises + deletes tmp on mismatch
    final = cfg.media_root / kind / digest[:2] / digest[2:4] / f"{digest}.{ext}"
    final.parent.mkdir(parents=True, exist_ok=True)
    os.replace(tmp, final)
    return digest, final

Hashing inside the write loop is free: one pass over the bytes. Hashing after the move means reading the whole file a second time, and letting unverified content into the store first.

Signatures worth hardcoding: JPEG FF D8 FF, PNG 89 50 4E 47 0D 0A 1A 0A, GIF 47 49 46 38, WebP 52 49 46 46 at offset 0 with 57 45 42 50 at offset 8, WebM 1A 45 DF A3, and MP4 with 66 74 79 70 (ftyp) at offset 4.

Dedupe at two moments

Before the transfer, a canonical URL to hash lookup can skip the download entirely. After the transfer, a hash that already exists means the content is new to this URL but not to your store, so you keep the existing file and add one row.

CREATE TABLE refs (
  source_url TEXT PRIMARY KEY,
  sha256     TEXT NOT NULL,
  kind       TEXT NOT NULL,
  bytes      INTEGER NOT NULL,
  fetched_at TEXT NOT NULL
);

That table is also the seed for the incremental sync work later: the URL is the natural key, the hash is the identity of the content itself.

Exercise. A response returns 200, Content-Type: image/jpeg, Content-Length: 12480, and the body starts with <!DOCTYPE html. Which check catches it, and what should the code do?

Check the answer

The magic-byte sniff catches it: 3C 21 44 4F is not FF D8 FF. Content-Type alone would have passed it, so treat the header as a hint and the bytes as truth. The stage should delete the temp file, log the URL with a type_mismatch reason, and count it as a failed task rather than retrying forever.

Order of work for this step

1. Extend the config with media_root, tmp_root, min_bytes and max_bytes per kind, and the allowed formats list.
2. Write validate(path, kind, size, cfg), returning the extension or raising a typed error.
3. Write store_stream as above, so images and videos share one path to disk.
4. Create the refs table and insert with conflict handling, so a repeated URL updates rather than explodes.
5. Wire both downloaders from 2.1 and 2.2 through store_stream, deleting their ad hoc open(..., "wb") calls.
6. Add a verify-store command that walks media/, re-hashes each file, and compares the result with the file name.

Exercise. After a second full crawl, refs holds 4,000 rows but only 3,900 distinct hashes. Is that a defect?

Check the answer

No. It means 100 URLs served content you already had, which is exactly the duplication the store is designed to absorb. The real signal to watch is a hash appearing under two incompatible kinds, or a file whose re-hash disagrees with its name, and verify-store is what reports both.

How you know the step is done

• Interrupt a download mid-stream: media/ contains no new file, only a .part that your cleanup sweep removes on the next run.
• Crawl the same feed twice: the second run downloads zero bytes for known URLs and adds no rows.
• Feed the stage a fake JPEG whose body is HTML: rejected, temp deleted, failure logged with URL and reason.
• Two different URLs serving one image: one file on disk, two rows in refs.
• verify-store reports zero mismatches, and rebuilding the index from disk reproduces the same paths.

Traps that leave corrupt or duplicated files

• Naming by URL, post ID or timestamp: every crawl becomes a fresh copy, and storage grows without bound.
• Trusting the URL extension: WebP and AVIF are routinely served from .jpg links.
• Trusting Content-Type alone: error and login pages often echo the type the client asked for.
• Writing straight to the final path: a killed process leaves a half file that later looks valid and gets skipped by dedupe.
• Putting tmp_root on another filesystem from media_root: os.replace then fails, and the fallback is a non-atomic copy.
• Hashing after the move: an extra full read, plus a window where unverified content sits in the store.

Where the capstone stands now

You have a download stage with content-addressed storage, a two-moment dedupe rule, and validation that runs on every transfer, so a video fragment join and a single image both end in the same trustworthy place. The next lesson is the module checkpoint, where you audit this stage end to end against the image and video downloaders you built.

Front: Why name a media file by its content hash instead of its URL? Back: Same bytes always map to one path, so reposts do not create copies and paths are reproducible on any machine.

Front: Where should hashing happen during a download? Back: Inside the write loop, so each chunk is hashed and written in a single pass.

Front: Why write to data/tmp and then os.replace into media/? Back: The rename is atomic, so the store never contains a half-written file.

Front: What does the two-level shard directory use? Back: The first two and next two hex characters of the content hash.

Front: Which check catches an HTML login page served as image/jpeg? Back: Magic-byte sniffing of the leading bytes, not the Content-Type header.

Front: Where does the stored file extension come from? Back: The sniffed format, never the URL path.

Front: What does a bytes-written versus Content-Length mismatch indicate? Back: A truncated transfer, so the temp file must be discarded.

Front: At which two moments do you check for duplicates? Back: Before the transfer (canonical URL to hash) and after it (hash already present in the store).

Front: What are the columns of the refs table? Back: source_url as primary key, sha256, kind, bytes, fetched_at.

Front: What are the PNG magic bytes? Back: 89 50 4E 47 0D 0A 1A 0A at offset 0.

Front: At what offset does the MP4 ftyp signature sit? Back: Offset 4, after the 4-byte box size.

Front: What does re-running a crawl over an unchanged feed reveal? Back: Zero bytes downloaded and no new refs rows, which proves dedupe works.

The pipeline now moves bytes through SHA-256, then _____ rename into a content-addressed path, so a crash never publishes a partial file. The dedupe lookup happens at two moments: before the transfer via the canonical URL, and after it when the _____ already exists. Every stored file name ends with an extension taken from _____, not from the URL. Temp files must live under _____ on the same filesystem as media/, otherwise the rename stops being atomic. A file whose re-hash disagrees with its own name is reported by _____.