🚀 Capstone Step 3: Observability, Recovery and Scaling Out

By Lê Nguyễn Tuân

Project Brief: Make the Crawler Operable

By the end of the previous lesson, the capstone could already claim a task, resolve a canonical URL, and skip anything it had seen before. That gives you a pipeline that is correct when everything goes right. This lesson makes it survivable when things go wrong: a worker killed mid-download, a platform returning 500s for an hour, a deliberate restart during a deploy, and the moment you decide one worker is no longer enough.

Three outcomes define the finished build:

  1. You can read the logs and answer "what is failing, where, and since when" in under a minute.
  2. Any task can be interrupted at any point and the system converges to the same result after restart.
  3. Going from one worker to four is a config change, not a rewrite.

Important: Recovery is a design property, not an error-handling afterthought. If the only way your crawler survives a crash is that you re-run it from scratch, the pipeline is not finished.

Spec: What the Final Build Must Include

Structured logs. One JSON object per line, written to stdout, with at least these fields: ts, level, worker, task_id, platform, stage (fetch, download, verify, commit), canonical_url, attempt, duration_ms, outcome, and error_class on failures.

{"ts":"2025-03-14T09:12:44.318Z","level":"info","worker":"w-3","task_id":"t_8812","platform":"bilibili","stage":"download","canonical_url":"https://www.bilibili.com/video/BV1xx?q=720","attempt":1,"duration_ms":4310,"outcome":"ok","bytes":18432011}

Error classification and retry. The retry policy is only as good as the classification feeding it.

In this lesson

  • Build Order for the Last Mile
  • Definition of Done
  • Where Operable Crawlers Usually Break

About this course

Part of Media Crawler Architecture: Queue, Storage and Scaling — A fast, project-driven course where you build one reusable media crawler for Facebook, Weibo and Bilibili. You start from the request/response anatomy of each platform, then assemble a capstone tool with config-driven sources, streaming media downloads, deduplicated storage, and a queue-based architecture that scales and survives failures. Anti-bot evasion is explicitly out of scope.

Level
Intermediate
Goal
Build a project
Language
English
Focus
media download architecture scaling
Practice
Capstone project
Time
~4.2h
Voice
Một giảng viên trình bày có hệ thống, đi từ lý thuyết đến thực hành
Not covered
anti_bot_bypass

This lesson is premium content. Sign in and purchase the course to view the full content. Sign in