# 🚀 Capstone Step 3: Observability, Recovery and Scaling Out

By Lê Nguyễn Tuân

> This lesson is premium content. Sign in and purchase the course to view the full content.

## Project Brief: Make the Crawler Operable

By the end of the previous lesson, the capstone could already claim a task, resolve a canonical URL, and skip anything it had seen before. That gives you a pipeline that is *correct* when everything goes right. This lesson makes it *survivable* when things go wrong: a worker killed mid-download, a platform returning 500s for an hour, a deliberate restart during a deploy, and the moment you decide one worker is no longer enough.

Three outcomes define the finished build:

1. You can read the logs and answer "what is failing, where, and since when" in under a minute.
2. Any task can be interrupted at any point and the system converges to the same result after restart.
3. Going from one worker to four is a config change, not a rewrite.

> **Important:** 
> Recovery is a design property, not an error-handling afterthought. If the only way your crawler survives a crash is that you re-run it from scratch, the pipeline is not finished.

## Spec: What the Final Build Must Include

**Structured logs.** One JSON object per line, written to stdout, with at least these fields: `ts`, `level`, `worker`, `task_id`, `platform`, `stage` (`fetch`, `download`, `verify`, `commit`), `canonical_url`, `attempt`, `duration_ms`, `outcome`, and `error_class` on failures.

```json
{"ts":"2025-03-14T09:12:44.318Z","level":"info","worker":"w-3","task_id":"t_8812","platform":"bilibili","stage":"download","canonical_url":"https://www.bilibili.com/video/BV1xx?q=720","attempt":1,"duration_ms":4310,"outcome":"ok","bytes":18432011}
```

**Error classification and retry.** The retry policy is only as good as the classification feeding it.

## In this lesson

- Build Order for the Last Mile
- Definition of Done
- Where Operable Crawlers Usually Break


---

## Contents

Part of [🎞️ Media Crawler Architecture: Queue, Storage and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling.md)

- **🧭 Foundations and Capstone Skeleton**
  - [🔍 How Facebook, Weibo and Bilibili Serve Media](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/media-serving-anatomy-three-platforms.md)
  - [🧱 Capstone Blueprint: Config, Storage Layout, Pipeline Modules](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-blueprint-config-storage-pipeline.md) (sign-in required)
  - [🧩 Capstone Step 1: Normalize Media Records from Feed Responses](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-normalize-media-records.md)
  - [✅ Checkpoint: Platform Anatomy and Project Skeleton](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-platform-anatomy-skeleton.md)
- **📥 Media Download Engineering**
  - [🌊 Streaming Image Downloads: Chunks, Resume, Retries, Timeouts](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/streaming-image-downloads-resume-retries.md)
  - [🎥 Video Downloads: Manifests, Fragments, Resolution Selection](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/video-downloads-manifests-fragments-resolution.md)
  - [🗄️ Capstone Step 2: Storage, Content Hashing and Integrity Checks](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-storage-hashing-integrity.md)
  - [✅ Checkpoint: Download Engineering](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-download-engineering.md)
- **🏗️ Architecture, Scheduling and Scaling**
  - [🔁 Queue-Based Workers: Task Model, Backpressure, Rate Budget](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/queue-workers-task-model-backpressure.md)
  - [🚀 Capstone Step 3: Observability, Recovery and Scaling Out](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-observability-recovery-scaling.md) (premium content) — this lesson
  - [✅ Checkpoint: Architecture and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-architecture-scaling.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/cheatsheet.md) (premium content)