# 🎞️ Cheatsheet: Media Crawler Architecture: Queue, Storage and Scaling

By Lê Nguyễn Tuân

> This cheatsheet is premium content. Sign in and purchase the course to view the full content.

Quick reference for the whole course. Use it while building and debugging the capstone.

---

## Module 1: Foundations and Capstone Skeleton

### 1.1 How the three platforms serve media

| Platform | Media delivery pattern | Key gotchas |
| --- | --- | --- |
| Facebook | Feed JSON (GraphQL / internal endpoints) returns media URLs; images and video on CDN hosts | URLs are time-signed and expire; pagination via cursors; heavy JS rendering |
| Weibo | Timeline JSON (`/statuses/...` style endpoints); images on `sinaimg` CDN with size suffixes | Size suffix in URL (e.g. `thumbnail` vs `large`); referer checks; rate limits |
| Bilibili | Video via DASH manifest (JSON) with separate video + audio streams; images on CDN | Manifest lists fragments; audio and video are separate; needs referer header |

**Core mental model:** every platform is *feed response -> media records -> downloadable asset URLs*. Normalize early, so downstream code never branches on platform.

```mermaid
flowchart LR
    Feed["📥<br/>Feed response"] --> Norm["🧩<br/>Normalize"] --> Store["🗄️<br/>Store record"] --> DL["⬇️<br/>Download"]
```

### 1.2 Capstone blueprint

| Layer | Responsibility | Rule of thumb |
| --- | --- | --- |
| Config | Endpoints, headers, rate budgets, output dirs, retry policy | One config object, loaded once, injected everywhere |
| Storage | Raw records, media files, metadata DB, logs | Predictable directory layout, never overwrite silently |
| Pipeline | Fetch -> normalize -> enqueue -> download -> verify -> persist | Each stage is a pure function on a record |

**Suggested storage layout**

```tree
data/
  raw/            # untouched feed JSON
  media/
    <platform>/<account>/<yyyy>/<mm>/
  meta/           # sqlite or jsonl index
  logs/
```

**Config skeleton (Python)**