Capstone Blueprint: Config, Storage Layout, Pipeline Modules

# config/sources.yaml
defaults:
  user_agent: "capstone-crawler/0.1"
  timeout_s: 20
  retries: 3
  storage_root: ./data

sources:
  - id: fb_page_demo          # unique and stable: becomes part of the path
    platform: facebook
    enabled: true
    auth:
      cookie_file: ./secrets/fb.cookies.json   # reference, never a value
    endpoints:
      feed: "https://www.facebook.com/api/graphql"
    parse:
      profile: facebook_feed_v1
    storage:
      subdir: facebook/fb_page_demo
    rate_limit_per_min: 20

  - id: weibo_user_demo
    platform: weibo
    enabled: true
    auth:
      cookie_file: ./secrets/weibo.cookies.json
    endpoints:
      feed: "https://m.weibo.cn/api/container/getIndex"
    parse:
      profile: weibo_container_v1
    storage:
      subdir: weibo/weibo_user_demo

Three properties do the heavy lifting. id is stable, because it is written into every storage path and every record; renaming it orphans files. enabled lets you switch a source off without deleting its history. And parse.profile names a parser implementation, which keeps the pipeline platform agnostic: the scheduler never imports a Facebook module, it looks a name up in a registry.

Validate the config at startup: unknown keys, a subdir that escapes the storage root, or a missing cookie file should stop the process with a readable message. A crash at boot is cheap; a crawler that silently writes outside its root is not.

Exercise. Add a second Weibo source that shares the same cookie file but writes to a different subdirectory, with a stricter rate budget. Answer

Copy the weibo_user_demo block, give it a new id (for example weibo_user_second), change storage.subdir to weibo/weibo_user_second, and set rate_limit_per_min: 10. Reusing cookie_file is fine; reusing id is not.

The storage layout on disk

The layout answers one question: given an immutable key, where do the bytes go? Keep five rules.

1. The path is a function of identifiers. Same record in, same path out, on every re-run.
2. Partition for humans. Platform, then source, then date, then file. Listing one day of one source should be one directory read.
3. Never write to the final path. Stream into tmp/, then rename. Rename is atomic, so a crash leaves a .part file you can ignore instead of a truncated JPEG that looks finished.
4. Ship a sidecar. Each media file gets a sibling .json with the MediaRecord, so any directory is self-describing.
5. Separate state from media. Cursors, the seen set and logs live outside the media tree, so you can wipe media without losing dedupe history.

data/
  facebook/
    fb_page_demo/
      2024/
        05/
          12/
            9f2c1a04_0.jpg
            9f2c1a04_0.jpg.json
  tmp/
    fb_page_demo/
      9f2c1a04_0.jpg.part
  state/
    cursors.json
    seen.db
  logs/
    crawl-2024-05-12.log

The date partition comes from posted_at when the platform provides it. When it does not, use the timestamp of the first successful fetch and persist it in the sidecar, so later runs compute the same path.

Exercise. A record has source_id: weibo_user_demo, post_id: 5012345678, media_index: 1, posted_at: 2024-03-09, ext_hint: mp4. Write the final path and the temporary path. Answer

Final: data/weibo/weibo_user_demo/2024/03/09/5012345678_1.mp4. Temporary: data/tmp/weibo_user_demo/5012345678_1.mp4.part, renamed on completion.

Where fetching ends and parsing begins

Three stages, three contracts. The parser is the only place that knows what a platform looks like.

Stage	Input	Output	Must not
Fetch	source id, URL, params	FetchResult: status, headers, body bytes, timing	parse, guess media, write files
Parse	raw bytes, source config	list[MediaRecord]	touch the network or the disk
Download	one MediaRecord	DownloadResult: path, byte count, checksum	know the platform name

flowchart TD
  Sched["🗓️ Scheduler"] --> Fetch["📥 Fetch"]
  Fetch --> Parse["🧩 Parse"]
  Parse --> Record["📄 MediaRecord"]
  Record --> Download["⬇️ Download"]
  Download --> Store["🗄️ Storage"]

The object crossing the boundary is a MediaRecord: source_id, platform, post_id, media_index, kind (image or video), canonical_url, ext_hint, posted_at, raw_ref. It is deliberately flat and serialisable, because in Module 3 it becomes a queue message.

Why bother with the boundary? When a platform renames a JSON field, you edit one parser. When you add a platform, you write one parser and touch nothing else. When a download fails, you replay the stored MediaRecord without hitting the feed again. Next lesson you will parse real feed fixtures, and a pure parser is what makes that testable offline.

Exercise. A platform changes its response shape. Which module changes, and which files can stay untouched? Answer

Only the parser registered under that source's parse.profile changes. Fetch, download, storage, and the config file stay untouched, unless the new shape also changes the endpoint, which is a config edit, not a code edit.

Anything that differs between platforms belongs in the _____ file, not in code. Secrets are referenced by _____, never stored inline in the config. The fetch stage returns a raw _____ containing status, headers and body bytes. Only the _____ stage is allowed to hold platform specific knowledge. Downloads are written to a temporary file and then _____ into the final path. Cursors and the seen set live in a separate _____, outside the media tree.

Rules of thumb before you write a line of code

• Config out of code, secrets out of config. A cookie value pasted into YAML ends up in git.
• If a module needs the platform name, the boundary is wrong. The sole exception is the parser registry lookup.
• Log the decision, not the bytes. source_id, canonical URL and the skip reason tell you more than a hex dump.
• Common mistake: dedupe state inside the media tree. Delete a bad media partition and you lose the record of what you already downloaded.
• Common mistake: writing straight to the final path. Half-downloaded files then look real to the next run.
• Cheap heuristic: if a function performs I/O and also interprets a payload, split it. That split is exactly the fetch/parse line.

Front: Why must every source have a unique, stable id? Back: It is embedded in storage paths and records, so renaming it orphans existing files.

Front: Which config values belong in the defaults block? Back: Values identical for every source: timeouts, retries, user agent, storage root.

Front: How are secrets stored in the source config? Back: By reference only, as a file path or environment variable name.

Front: What does parse.profile select? Back: The named parser implementation that turns raw payloads into MediaRecords.

Front: What does the fetch stage output? Back: A raw FetchResult with status, headers, body bytes, timing and request URL.

Front: Which stage may hold platform specific knowledge? Back: The parser; fetch and download stay platform agnostic.

Front: Why stream to tmp and rename instead of writing the final path? Back: Rename is atomic, so a crash never leaves a truncated file that looks complete.

Front: Where do cursors and the seen set live? Back: In a separate top-level state directory, outside the media tree.

Front: List the fields of a MediaRecord. Back: source_id, platform, post_id, media_index, kind, canonical_url, ext_hint, posted_at, raw_ref.

Front: What is a metadata sidecar? Back: A small JSON file next to each media file, holding its record so the directory is self-describing.

Front: Which date should the storage path use? Back: posted_at when available, otherwise the first stored fetch timestamp, persisted so it stays stable.

Front: Why must the parser be a pure function? Back: Same input gives same output, so it can be unit tested against stored fixtures with no network.

What the skeleton buys you

You now have three decisions locked in: platform differences live in config/sources.yaml, every byte has a deterministic path that is safe under crashes, and fetching, parsing and downloading talk only through FetchResult and MediaRecord. That is the whole architecture. Module 2 fills in the download stage behind the contract you just defined, and Module 3 turns the scheduler box into real queue-based workers.

Next up is Capstone Step 1: you will write the first parser against real feed fixtures and produce MediaRecord objects from actual responses.