# 🌊 Streaming Image Downloads: Chunks, Resume, Retries, Timeouts

By Lê Nguyễn Tuân

Module 1 left you with a normalized media record per post and a storage layout to write into. That record now points at a URL that may be 200 KB or 20 MB, served from a CDN that can stall, rate-limit or drop the connection halfway. This lesson builds the download primitive the rest of the module will reuse, so that every file entering your store is complete.

By the end of this lesson you can:

- Read a response body in fixed-size chunks instead of buffering whole images in memory
- Publish files atomically, so the store never contains a half-written image
- Resume an interrupted transfer with a `Range` request and validate the server's answer
- Separate retryable failures from permanent ones, and back off with a cap
- Set connect, read and total deadlines that a stalled socket cannot escape

## From Feed Record to File on Disk

Every image download moves through the same lifecycle, and each stage is a place where a long-running tool loses data if you skip it.

```mermaid
flowchart TD
    A["📋 Media record from Module 1"] --> B{"Partial .part file exists?"}
    B -->|Yes| C["📡 GET with Range header"]
    B -->|No| D["📡 GET from byte 0"]
    C --> E["📥 Append chunks to .part"]
    D --> E
    E --> F{"Byte count matches Content-Length?"}
    F -->|Yes| G["✅ os.replace to final path"]
    F -->|No| H["⏱️ Backoff, then retry"]
    H --> B
```

## Chunks: Read the Body as a Stream

A plain `requests.get(url)` reads the entire body into memory before returning. With a queue of workers each pulling 20 MB videos, that is how a crawler dies. Streaming keeps memory flat per item and lets you react while bytes are still arriving.

```python
import requests

CHUNK = 128 * 1024  # 128 KB

with requests.get(url, stream=True, timeout=(5, 20)) as r:
    r.raise_for_status()
    with open(part_path, "wb") as f:
        for chunk in r.iter_content(CHUNK):
            if chunk:           # keep-alive packets arrive as b""
                f.write(chunk)
```

`stream=True` defers the body download; `iter_content(n)` hands you at most `n` bytes at a time. A chunk size of 64 to 256 KB is the practical band: below that you pay syscall overhead per image, above it you gain nothing because the socket buffer is the real limit. Check `Content-Length` before you start when it exists; if the header is missing (chunked transfer encoding is common on CDNs), you cannot pre-check the size, which is why the integrity check in the next capstone step hashes the finished file.

## Atomic Writes: No Half Files in the Store

Write to `<dest>.part`, close it, then publish with `os.replace(part, dest)`. On the same filesystem this rename is atomic: any reader either sees the old state or the complete file, never an 800 KB fragment wearing the final name. This matters because later stages (content hashing, dedupe by canonical URL, the database insert) all assume a file at its final path is complete. If you skip the temp file, a crash leaves a truncated image that hashes cleanly and silently poisons your dedupe index.

Add `f.flush()` and `os.fsync(f.fileno())` before the rename if the machine can lose power mid-run; for a crawl tool that runs for days, it is cheap insurance.

**Practice 2.1.** A worker crashes after writing 900 KB of a 1 MB image. On restart, what should the downloader inspect first, and what must it never do?

**Answer**

Inspect `<dest>.part` and its size. If it exists, resume from that offset with a range request guarded by the stored `ETag`. Never rename a partial file to the final path, and never append bytes to a `.part` file that was created from a different URL or CDN variant, because the two halves would not belong to the same image.

## Resume: Range Requests and Their Traps

Resuming means asking for the tail of the file:

```
GET /media/abc.jpg HTTP/1.1
Range: bytes=921600-
If-Range: "6f3a1c9e-8b2"
```

A correct server answers `206 Partial Content` with a `Content-Range: bytes 921600-1048575/1048576` header. Open the `.part` in append mode and continue. Three traps catch most implementations:

1. **The server ignores the range.** You get `200` with the whole body. Truncate the `.part` and write from byte 0, otherwise you splice two copies together.
2. **The file changed since the partial write.** Without `If-Range`, a rotated CDN asset produces a corrupt hybrid. Store the `ETag` (or `Last-Modified`) next to the `.part` and send it back as `If-Range`.
3. **The partial file is already complete.** If `os.path.getsize(part)` equals the total from `Content-Range`, skip the request entirely and rename.

## Retries: Bounded, Classified, Backed Off

Retrying everything is as wrong as retrying nothing. Classify first:

| Failure | Retry? | Why |
| --- | --- | --- |
| Read or connect timeout | Yes | network blip |
| `ConnectionError`, reset | Yes | transient |
| `429 Too Many Requests` | Yes, after `Retry-After` | rate budget, not an error |
| `500`, `502`, `503`, `504` | Yes | server-side, transient |
| `403`, `401` | No | policy or auth signal, backoff will not help |
| `404`, `410` | No | the record is stale, mark it and move on |

```python
def with_retries(fn, attempts=4, base=1.0):
    for i in range(attempts):
        try:
            return fn()
        except TRANSIENT as e:
            if i == attempts - 1:
                raise
            time.sleep(base * 2 ** i + random.random() * 0.5)  # jitter
```

Exponential backoff with jitter spreads a thundering herd of workers across time instead of synchronizing them into a burst that triggers the next `429`. Cap attempts at 3 to 5 and also cap the per-item total time, so one pathological image cannot occupy a worker for an hour.

## Timeouts: Connect, Read and a Total Deadline

`timeout=(5, 20)` means 5 seconds to establish the TCP connection and 20 seconds of socket inactivity while reading. That read timeout resets on every byte received, so it does not protect you from a server that trickles one byte every 19 seconds. For that case you need a wall-clock deadline:

```python
deadline = time.monotonic() + 120
for chunk in r.iter_content(CHUNK):
    if time.monotonic() > deadline:
        raise DownloadTooSlow(url)
    f.write(chunk)
```

Always have a default timeout on the session object. A worker with no read timeout will eventually hang on a dead peer and stay hung, and with a bounded worker pool, a handful of hung workers stops the whole pipeline.

**Practice 2.2.** Your tool downloads at roughly 40 items per minute, then drops to zero for 10 minutes on the same 20 URLs, and the log shows `403`. Which setting is misconfigured, and which one is not the problem?

**Answer**

The retry classifier is misconfigured: `403` is a permanent, policy-level answer and should be marked and dropped, not retried with backoff. Attempt caps and timeouts are not the problem here; adding more retries only spends your rate budget on requests that will never succeed.

## Quick Recall Check

A resumable download sends the _____ request header and expects status code _____ back.
In-progress bytes live in a temporary .part file and are published with _____, which is atomic on the same filesystem.
The timeout pair (5, 20) means _____.
Status _____ means rate limited, so honor Retry-After instead of retrying immediately.
Transient failures are retried with _____ and a capped attempt count.

## Hard-Won Rules for Image Downloads

- Keep one `.part` file per destination path and never share it between URLs or CDN variants.
- Compare the final byte count against `Content-Length` whenever that header exists; a short read is a truncated file, not a success.
- Set `timeout` once on the session, not per call, so nobody forgets it in a new code path.
- Budget time per item as well as per request. Three retries of a 20 second read timeout already costs a minute.
- Log `url`, `attempt`, `status`, `bytes_written` and `elapsed` on every attempt. When a run degrades at 3 a.m., that log is your only witness.
- Treat `403` as information, not noise. Retrying it burns your rate budget and makes the next real request more likely to fail.

## Before You Move On to Video

Chunked reads keep memory flat, `.part` plus `os.replace` guarantees the store only ever sees complete files, `Range` with `If-Range` resumes safely, classified retries with jitter and a cap keep failures local, and timeouts with a wall-clock deadline keep a worker from disappearing.

Next, the same four ideas meet a harder target: video, where the URL often points at a manifest, the real bytes live in dozens of fragments, and you must also choose which resolution to keep.

Front: Why stream a response body instead of loading it with a plain GET?
Back: Memory stays flat per item and you can abort as soon as the transfer goes wrong.

Front: Which two requests features make chunked reading possible?
Back: stream=True on the request and iter_content(n) on the response.

Front: What is atomic publish in a downloader?
Back: Write to a .part file, then os.replace it to the final path on the same filesystem.

Front: Why must finished images never be written directly to their final path?
Back: A crash would leave a truncated file that later stages treat as complete and valid.

Front: Which header resumes a partial download?
Back: Range: bytes=<offset>-

Front: Which status code confirms the server honored the range?
Back: 206 Partial Content.

Front: What does the Content-Range response header tell you?
Back: The byte offset, the end offset, and the total size of the representation.

Front: What should you do if a range request returns 200?
Back: Truncate the .part file and write from byte 0, because the server sent the whole body.

Front: Why send If-Range with the stored ETag?
Back: It guarantees the partial bytes still belong to the same version of the file.

Front: Which failures are safe to retry?
Back: Timeouts, connection errors, 5xx responses, and 429 after honoring Retry-After.

Front: Which failures should never be retried?
Back: 400, 401, 403, 404 and 410, which signal a permanent condition.

Front: What does timeout=(5, 20) mean?
Back: 5 seconds to connect and 20 seconds of read inactivity before the request fails.

Front: Why is a read timeout not enough on its own?
Back: It resets on every received byte, so a slow trickle can keep a worker busy forever.

Front: What does a final byte count below Content-Length indicate?
Back: A truncated body, so the file must be resumed or discarded, never published.

---

## Contents

Part of [🎞️ Media Crawler Architecture: Queue, Storage and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling.md)

- **🧭 Foundations and Capstone Skeleton**
  - [🔍 How Facebook, Weibo and Bilibili Serve Media](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/media-serving-anatomy-three-platforms.md)
  - [🧱 Capstone Blueprint: Config, Storage Layout, Pipeline Modules](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-blueprint-config-storage-pipeline.md) (sign-in required)
  - [🧩 Capstone Step 1: Normalize Media Records from Feed Responses](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-normalize-media-records.md)
  - [✅ Checkpoint: Platform Anatomy and Project Skeleton](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-platform-anatomy-skeleton.md)
- **📥 Media Download Engineering**
  - [🌊 Streaming Image Downloads: Chunks, Resume, Retries, Timeouts](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/streaming-image-downloads-resume-retries.md) — this lesson
  - [🎥 Video Downloads: Manifests, Fragments, Resolution Selection](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/video-downloads-manifests-fragments-resolution.md)
  - [🗄️ Capstone Step 2: Storage, Content Hashing and Integrity Checks](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-storage-hashing-integrity.md)
  - [✅ Checkpoint: Download Engineering](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-download-engineering.md)
- **🏗️ Architecture, Scheduling and Scaling**
  - [🔁 Queue-Based Workers: Task Model, Backpressure, Rate Budget](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/queue-workers-task-model-backpressure.md)
  - [🚀 Capstone Step 3: Observability, Recovery and Scaling Out](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/capstone-observability-recovery-scaling.md) (premium content)
  - [✅ Checkpoint: Architecture and Scaling](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/checkpoint-architecture-scaling.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/media-crawler-architecture-queue-storage-and-scaling/cheatsheet.md) (premium content)