# 🧱 Capstone Blueprint: Config, Storage Layout, Pipeline Modules

By Lê Nguyễn Tuân

> Sign in to view the full content of this lesson.

In the previous lesson we traced how Facebook, Weibo and Bilibili hand out media, and we agreed on one habit: identify each item with a stable key such as post id plus media index before you build anything else. This lesson turns that habit into a skeleton you will keep for the rest of the course: one config file that describes every source, one predictable place on disk for every byte, and three modules with hard boundaries between fetching, parsing and downloading. Module 2 and Module 3 both bolt onto this skeleton, so an hour spent here saves a rewrite later.

By the end of this lesson you can:

- Write a source config file where platform differences live in data, not in code.
- Design a storage layout that is stable across re-runs and safe under crashes.
- State the input and output contract of the fetch, parse and download stages.
- Name the fields of a `MediaRecord`, the single object that travels between stages.

## Config as the single source of truth

The rule is short: anything that can differ between sources is data, anything that is identical everywhere is a default. Endpoints, cookie references, rate budgets, parser names and storage subdirectories vary per source, so they belong in config. Timeouts, retries and the storage root do not, so they belong in a `defaults` block that every source inherits.

```yaml
# config/sources.yaml
defaults:
  user_agent: "capstone-crawler/0.1"
  timeout_s: 20
  retries: 3
  storage_root: ./data

sources:
  - id: fb_page_demo          # unique and stable: becomes part of the path
    platform: facebook
    enabled: true
    auth:
      cookie_file: ./secrets/fb.cookies.json   # reference, never a value
    endpoints:
      feed: "https://www.facebook.com/api/graphql"
    parse:
      profile: facebook_feed_v1
    storage:
      subdir: facebook/fb_page_demo
    rate_limit_per_min: 20

  - id: weibo_user_demo
    platform: weibo
    enabled: true
    auth:
      cookie_file: ./secrets/weibo.cookies.json
    endpoints:
      feed: "https://m.weibo.cn/api/container/getIndex"
    parse:
      profile: weibo_container_v1
    storage:
      subdir: weibo/weibo_user_demo
```

Three properties do the heavy lifting. `id` is stable, because it is written into every storage path and every record; renaming it orphans files. `enabled` lets you switch a source off without deleting its history. And `parse.profile` names a parser implementation, which keeps the pipeline platform agnostic: the scheduler never imports a Facebook module, it looks a name up in a registry.

Validate the config at startup: unknown keys, a `subdir` that escapes the storage root, or a missing cookie file should stop the process with a readable message. A crash at boot is cheap; a crawler that silently writes outside its root is not.

**Exercise.** Add a second Weibo source that shares the same cookie file but writes to a different subdirectory, with a stricter rate budget.
**Answer**

Copy the `weibo_user_demo` block, give it a new `id` (for example `weibo_user_second`), change `storage.subdir` to `weibo/weibo_user_second`, and set `rate_limit_per_min: 10`. Reusing `cookie_file` is fine; reusing `id` is not.

## The storage layout on disk

The layout answers one question: given an immutable key, where do the bytes go? Keep five rules.

1. **The path is a function of identifiers.** Same record in, same path out, on every re-run.
2. **Partition for humans.** Platform, then source, then date, then file. Listing one day of one source should be one directory read.
3. **Never write to the final path.** Stream into `tmp/`, then rename. Rename is atomic, so a crash leaves a `.part` file you can ignore instead of a truncated JPEG that looks finished.
4. **Ship a sidecar.** Each media file gets a sibling `.json` with the `MediaRecord`, so any directory is self-describing.
5. **Separate state from media.** Cursors, the seen set and logs live outside the media tree, so you can wipe media without losing dedupe history.

```tree
data/
  facebook/
    fb_page_demo/
      2024/
        05/
          12/
            9f2c1a04_0.jpg
            9f2c1a04_0.jpg.json
  tmp/
    fb_page_demo/
      9f2c1a04_0.jpg.part
  state/
    cursors.json
    seen.db
  logs/
    crawl-2024-05-12.log
```

The date partition comes from `posted_at` when the platform provides it. When it does not, use the timestamp of the first successful fetch and persist it in the sidecar, so later runs compute the same path.

**Exercise.** A record has `source_id: weibo_user_demo`, `post_id: 5012345678`, `media_index: 1`, `posted_at: 2024-03-09`, `ext_hint: mp4`. Write the final path and the temporary path.
**Answer**

Final: `data/weibo/weibo_user_demo/2024/03/09/5012345678_1.mp4`. Temporary: `data/tmp/weibo_user_demo/5012345678_1.mp4.part`, renamed on completion.

## Where fetching ends and parsing begins

Three stages, three contracts. **The parser is the only place that knows what a platform looks like.**

| Stage | Input | Output | Must not |
| --- | --- | --- | --- |
| Fetch | source id, URL, params | `FetchResult`: status, headers, body bytes, timing | parse, guess media, write files |
| Parse | raw bytes, source config | `list[MediaRecord]` | touch the network or the disk |
| Download | one `MediaRecord` | `DownloadResult`: path, byte count, checksum | know the platform name |

```mermaid
flowchart TD
  Sched["🗓️ Scheduler"] --> Fetch["📥 Fetch"]
  Fetch --> Parse["🧩 Parse"]
  Parse --> Record["📄 MediaRecord"]
  Record --> Download["⬇️ Download"]
  Download --> Store["🗄️ Storage"]
```

The object crossing the boundary is a `MediaRecord`: `source_id`, `platform`, `post_id`, `media_index`, `kind` (image or video), `canonical_url`, `ext_hint`, `posted_at`, `raw_ref`. It is deliberately flat and serialisable, because in Module 3 it becomes a queue message.

Why bother with the boundary? When a platform renames a JSON field, you edit one parser. When you add a platform, you write one parser and touch nothing else. When a download fails, you replay the stored `MediaRecord` without hitting the feed again. Next lesson you will parse real feed fixtures, and a pure parser is what makes that testable offline.

**Exercise.** A platform changes its response shape. Which module changes, and which files can stay untouched?
**Answer**

Only the parser registered under that source's `parse.profile` changes. Fetch, download, storage, and the config file stay untouched, unless the new shape also changes the endpoint, which is a config edit, not a code edit.

Anything that differs between platforms belongs in the _____ file, not in code.
Secrets are referenced by _____, never stored inline in the config.
The fetch stage returns a raw _____ containing status, headers and body bytes.
Only the _____ stage is allowed to hold platform specific knowledge.
Downloads are written to a temporary file and then _____ into the final path.
Cursors and the seen set live in a separate _____, outside the media tree.

## Rules of thumb before you write a line of code

- **Config out of code, secrets out of config.** A cookie value pasted into YAML ends up in git.
- **If a module needs the platform name, the boundary is wrong.** The sole exception is the parser registry lookup.
- **Log the decision, not the bytes.** `source_id`, canonical URL and the skip reason tell you more than a hex dump.
- **Common mistake: dedupe state inside the media tree.** Delete a bad media partition and you lose the record of what you already downloaded.
- **Common mistake: writing straight to the final path.** Half-downloaded files then look real to the next run.
- **Cheap heuristic:** if a function performs I/O and also interprets a payload, split it. That split is exactly the fetch/parse line.

Front: Why must every source have a unique, stable id?
Back: It is embedded in storage paths and records, so renaming it orphans existing files.

Front: Which config values belong in the defaults block?
Back: Values identical for every source: timeouts, retries, user agent, storage root.

Front: How are secrets stored in the source config?
Back: By reference only, as a file path or environment variable name.

Front: What does parse.profile select?
Back: The named parser implementation that turns raw payloads into MediaRecords.

Front: What does the fetch stage output?
Back: A raw FetchResult with status, headers, body bytes, timing and request URL.

Front: Which stage may hold platform specific knowledge?
Back: The parser; fetch and download stay platform agnostic.

Front: Why stream to tmp and rename instead of writing the final path?
Back: Rename is atomic, so a crash never leaves a truncated file that looks complete.

Front: Where do cursors and the seen set live?
Back: In a separate top-level state directory, outside the media tree.

Front: List the fields of a MediaRecord.
Back: source_id, platform, post_id, media_index, kind, canonical_url, ext_hint, posted_at, raw_ref.

Front: What is a metadata sidecar?
Back: A small JSON file next to each media file, holding its record so the directory is self-describing.

Front: Which date should the storage path use?
Back: posted_at when available, otherwise the first stored fetch timestamp, persisted so it stays stable.

Front: Why must the parser be a pure function?
Back: Same input gives same output, so it can be unit tested against stored fixtures with no network.

## What the skeleton buys you

You now have three decisions locked in: platform differences live in `config/sources.yaml`, every byte has a deterministic path that is safe under crashes, and fetching, parsing and downloading talk only through `FetchResult` and `MediaRecord`. That is the whole architecture. Module 2 fills in the download stage behind the contract you just defined, and Module 3 turns the scheduler box into real queue-based workers.

Next up is Capstone Step 1: you will write the first parser against real feed fixtures and produce `MediaRecord` objects from actual responses.

---

## Contents

Part of [🎞️ Media Crawler Architecture: Queue, Storage and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling.md)

- **🧭 Foundations and Capstone Skeleton**
  - [🔍 How Facebook, Weibo and Bilibili Serve Media](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/media-serving-anatomy-three-platforms.md)
  - [🧱 Capstone Blueprint: Config, Storage Layout, Pipeline Modules](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-blueprint-config-storage-pipeline.md) (sign-in required) — this lesson
  - [🧩 Capstone Step 1: Normalize Media Records from Feed Responses](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-normalize-media-records.md)
  - [✅ Checkpoint: Platform Anatomy and Project Skeleton](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-platform-anatomy-skeleton.md)
- **📥 Media Download Engineering**
  - [🌊 Streaming Image Downloads: Chunks, Resume, Retries, Timeouts](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/streaming-image-downloads-resume-retries.md)
  - [🎥 Video Downloads: Manifests, Fragments, Resolution Selection](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/video-downloads-manifests-fragments-resolution.md)
  - [🗄️ Capstone Step 2: Storage, Content Hashing and Integrity Checks](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-storage-hashing-integrity.md)
  - [✅ Checkpoint: Download Engineering](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-download-engineering.md)
- **🏗️ Architecture, Scheduling and Scaling**
  - [🔁 Queue-Based Workers: Task Model, Backpressure, Rate Budget](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/queue-workers-task-model-backpressure.md)
  - [🚀 Capstone Step 3: Observability, Recovery and Scaling Out](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-observability-recovery-scaling.md) (premium content)
  - [✅ Checkpoint: Architecture and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-architecture-scaling.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/cheatsheet.md) (premium content)