Cheatsheet: Media Crawler Architecture: Queue, Storage and Scaling

By Lê Nguyễn Tuân

Quick reference for the whole course. Use it while building and debugging the capstone.

Module 1: Foundations and Capstone Skeleton

1.1 How the three platforms serve media

Platform	Media delivery pattern	Key gotchas
Facebook	Feed JSON (GraphQL / internal endpoints) returns media URLs; images and video on CDN hosts	URLs are time-signed and expire; pagination via cursors; heavy JS rendering
Weibo	Timeline JSON (`/statuses/...` style endpoints); images on `sinaimg` CDN with size suffixes	Size suffix in URL (e.g. `thumbnail` vs `large`); referer checks; rate limits
Bilibili	Video via DASH manifest (JSON) with separate video + audio streams; images on CDN	Manifest lists fragments; audio and video are separate; needs referer header

Core mental model: every platform is feed response -> media records -> downloadable asset URLs. Normalize early, so downstream code never branches on platform.

flowchart LR
    Feed["📥<br/>Feed response"] --> Norm["🧩<br/>Normalize"] --> Store["🗄️<br/>Store record"] --> DL["⬇️<br/>Download"]

1.2 Capstone blueprint

Layer	Responsibility	Rule of thumb
Config	Endpoints, headers, rate budgets, output dirs, retry policy	One config object, loaded once, injected everywhere
Storage	Raw records, media files, metadata DB, logs	Predictable directory layout, never overwrite silently
Pipeline	Fetch -> normalize -> enqueue -> download -> verify -> persist	Each stage is a pure function on a record

Suggested storage layout

data/
  raw/            # untouched feed JSON
  media/
    <platform>/<account>/<yyyy>/<mm>/
  meta/           # sqlite or jsonl index
  logs/

Config skeleton (Python)

This cheatsheet is premium content. Sign in and purchase the course to view the full content. Sign in