Streaming Image Downloads: Chunks, Resume, Retries, Timeouts

By Lê Nguyễn Tuân

Module 1 left you with a normalized media record per post and a storage layout to write into. That record now points at a URL that may be 200 KB or 20 MB, served from a CDN that can stall, rate-limit or drop the connection halfway. This lesson builds the download primitive the rest of the module will reuse, so that every file entering your store is complete.

By the end of this lesson you can:

• Read a response body in fixed-size chunks instead of buffering whole images in memory
• Publish files atomically, so the store never contains a half-written image
• Resume an interrupted transfer with a Range request and validate the server's answer
• Separate retryable failures from permanent ones, and back off with a cap
• Set connect, read and total deadlines that a stalled socket cannot escape

From Feed Record to File on Disk

Every image download moves through the same lifecycle, and each stage is a place where a long-running tool loses data if you skip it.

flowchart TD
    A["📋 Media record from Module 1"] --> B{"Partial .part file exists?"}
    B -->|Yes| C["📡 GET with Range header"]
    B -->|No| D["📡 GET from byte 0"]
    C --> E["📥 Append chunks to .part"]
    D --> E
    E --> F{"Byte count matches Content-Length?"}
    F -->|Yes| G["✅ os.replace to final path"]
    F -->|No| H["⏱️ Backoff, then retry"]
    H --> B

Chunks: Read the Body as a Stream

A plain requests.get(url) reads the entire body into memory before returning. With a queue of workers each pulling 20 MB videos, that is how a crawler dies. Streaming keeps memory flat per item and lets you react while bytes are still arriving.

import requests

CHUNK = 128 * 1024  # 128 KB

with requests.get(url, stream=True, timeout=(5, 20)) as r:
    r.raise_for_status()
    with open(part_path, "wb") as f:
        for chunk in r.iter_content(CHUNK):
            if chunk:           # keep-alive packets arrive as b""
                f.write(chunk)

stream=True defers the body download; iter_content(n) hands you at most n bytes at a time. A chunk size of 64 to 256 KB is the practical band: below that you pay syscall overhead per image, above it you gain nothing because the socket buffer is the real limit. Check Content-Length before you start when it exists; if the header is missing (chunked transfer encoding is common on CDNs), you cannot pre-check the size, which is why the integrity check in the next capstone step hashes the finished file.

Atomic Writes: No Half Files in the Store

Write to <dest>.part, close it, then publish with os.replace(part, dest). On the same filesystem this rename is atomic: any reader either sees the old state or the complete file, never an 800 KB fragment wearing the final name. This matters because later stages (content hashing, dedupe by canonical URL, the database insert) all assume a file at its final path is complete. If you skip the temp file, a crash leaves a truncated image that hashes cleanly and silently poisons your dedupe index.

Add f.flush() and os.fsync(f.fileno()) before the rename if the machine can lose power mid-run; for a crawl tool that runs for days, it is cheap insurance.

Practice 2.1. A worker crashes after writing 900 KB of a 1 MB image. On restart, what should the downloader inspect first, and what must it never do?

Answer

Inspect <dest>.part and its size. If it exists, resume from that offset with a range request guarded by the stored ETag. Never rename a partial file to the final path, and never append bytes to a .part file that was created from a different URL or CDN variant, because the two halves would not belong to the same image.

Resume: Range Requests and Their Traps

Resuming means asking for the tail of the file:

GET /media/abc.jpg HTTP/1.1
Range: bytes=921600-
If-Range: "6f3a1c9e-8b2"

A correct server answers 206 Partial Content with a Content-Range: bytes 921600-1048575/1048576 header. Open the .part in append mode and continue. Three traps catch most implementations:

1. The server ignores the range. You get 200 with the whole body. Truncate the .part and write from byte 0, otherwise you splice two copies together.
2. The file changed since the partial write. Without If-Range, a rotated CDN asset produces a corrupt hybrid. Store the ETag (or Last-Modified) next to the .part and send it back as If-Range.
3. The partial file is already complete. If os.path.getsize(part) equals the total from Content-Range, skip the request entirely and rename.

Retries: Bounded, Classified, Backed Off

Retrying everything is as wrong as retrying nothing. Classify first:

def with_retries(fn, attempts=4, base=1.0):
    for i in range(attempts):
        try:
            return fn()
        except TRANSIENT as e:
            if i == attempts - 1:
                raise
            time.sleep(base * 2 ** i + random.random() * 0.5)  # jitter

Exponential backoff with jitter spreads a thundering herd of workers across time instead of synchronizing them into a burst that triggers the next 429. Cap attempts at 3 to 5 and also cap the per-item total time, so one pathological image cannot occupy a worker for an hour.

Timeouts: Connect, Read and a Total Deadline

timeout=(5, 20) means 5 seconds to establish the TCP connection and 20 seconds of socket inactivity while reading. That read timeout resets on every byte received, so it does not protect you from a server that trickles one byte every 19 seconds. For that case you need a wall-clock deadline:

deadline = time.monotonic() + 120
for chunk in r.iter_content(CHUNK):
    if time.monotonic() > deadline:
        raise DownloadTooSlow(url)
    f.write(chunk)

Always have a default timeout on the session object. A worker with no read timeout will eventually hang on a dead peer and stay hung, and with a bounded worker pool, a handful of hung workers stops the whole pipeline.

Practice 2.2. Your tool downloads at roughly 40 items per minute, then drops to zero for 10 minutes on the same 20 URLs, and the log shows 403. Which setting is misconfigured, and which one is not the problem?

Answer

The retry classifier is misconfigured: 403 is a permanent, policy-level answer and should be marked and dropped, not retried with backoff. Attempt caps and timeouts are not the problem here; adding more retries only spends your rate budget on requests that will never succeed.

Quick Recall Check

A resumable download sends the _____ request header and expects status code _____ back. In-progress bytes live in a temporary .part file and are published with _____, which is atomic on the same filesystem. The timeout pair (5, 20) means _____. Status _____ means rate limited, so honor Retry-After instead of retrying immediately. Transient failures are retried with _____ and a capped attempt count.

Hard-Won Rules for Image Downloads

• Keep one .part file per destination path and never share it between URLs or CDN variants.
• Compare the final byte count against Content-Length whenever that header exists; a short read is a truncated file, not a success.
• Set timeout once on the session, not per call, so nobody forgets it in a new code path.
• Budget time per item as well as per request. Three retries of a 20 second read timeout already costs a minute.
• Log url, attempt, status, bytes_written and elapsed on every attempt. When a run degrades at 3 a.m., that log is your only witness.
• Treat 403 as information, not noise. Retrying it burns your rate budget and makes the next real request more likely to fail.

Before You Move On to Video

Chunked reads keep memory flat, .part plus os.replace guarantees the store only ever sees complete files, Range with If-Range resumes safely, classified retries with jitter and a cap keep failures local, and timeouts with a wall-clock deadline keep a worker from disappearing.

Next, the same four ideas meet a harder target: video, where the URL often points at a manifest, the real bytes live in dozens of fragments, and you must also choose which resolution to keep.

Front: Why stream a response body instead of loading it with a plain GET? Back: Memory stays flat per item and you can abort as soon as the transfer goes wrong.

Front: Which two requests features make chunked reading possible? Back: stream=True on the request and iter_content(n) on the response.

Front: What is atomic publish in a downloader? Back: Write to a .part file, then os.replace it to the final path on the same filesystem.

Front: Why must finished images never be written directly to their final path? Back: A crash would leave a truncated file that later stages treat as complete and valid.

Front: Which header resumes a partial download? Back: Range: bytes=-

Front: Which status code confirms the server honored the range? Back: 206 Partial Content.

Front: What does the Content-Range response header tell you? Back: The byte offset, the end offset, and the total size of the representation.

Front: What should you do if a range request returns 200? Back: Truncate the .part file and write from byte 0, because the server sent the whole body.

Front: Why send If-Range with the stored ETag? Back: It guarantees the partial bytes still belong to the same version of the file.

Front: Which failures are safe to retry? Back: Timeouts, connection errors, 5xx responses, and 429 after honoring Retry-After.

Front: Which failures should never be retried? Back: 400, 401, 403, 404 and 410, which signal a permanent condition.

Front: What does timeout=(5, 20) mean? Back: 5 seconds to connect and 20 seconds of read inactivity before the request fails.

Front: Why is a read timeout not enough on its own? Back: It resets on every received byte, so a slow trickle can keep a worker busy forever.

Front: What does a final byte count below Content-Length indicate? Back: A truncated body, so the file must be resumed or discarded, never published.