Video Downloads: Manifests, Fragments, Resolution Selection

By Lê Nguyễn Tuân

In the previous lesson you streamed images: fixed-size chunks, resume from the last good byte, retries with backoff, hard timeouts. Video keeps those same rules but adds one twist: the file you want usually does not exist as a single URL. The platform publishes a manifest, and your job is to pick one variant, pull a numbered sequence of fragments, and stitch them back together. That is exactly the machinery your capstone needs for the video half of the pipeline.

By the end of this lesson you can:

• Read an HLS (.m3u8) and a DASH (.mpd) manifest and identify variants, init segments and fragments.
• Choose a resolution deliberately instead of grabbing the biggest one.
• Fetch fragments in the correct order, with concurrency and resume.
• Merge segments into one playable file and verify it before storing it.

Why Platforms Split Video at All

A 200 MB MP4 is a bad fit for mobile networks and CDN caches. So platforms use adaptive streaming: the same video is re-encoded at several resolutions, then each rendition is cut into short segments, typically 2 to 10 seconds. The player downloads segment after segment and switches rendition when bandwidth drops.

Two container families dominate what you will meet on Facebook, Weibo and Bilibili:

flowchart TD
  M["📜<br/>Manifest"] --> P["🔍<br/>Parse variants"]
  P --> S["🎯<br/>Select variant"]
  S --> F["🧩<br/>Fetch fragments"]
  F --> J["🔗<br/>Merge"]
  J --> V["✅<br/>Verify with ffprobe"]

An HLS master playlist lists the renditions:

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=2200000,RESOLUTION=1280x720
v720/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
v360/index.m3u8

The media playlist lists the actual pieces:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:4
#EXT-X-MAP:URI="init.mp4"
#EXTINF:4.000,
seg_00001.m4s
#EXTINF:4.000,
seg_00002.m4s
#EXT-X-ENDLIST

DASH carries the same information in XML:

<MPD mediaPresentationDuration="PT10M">
  <Period>
    <AdaptationSet mimeType="video/mp4">
      <Representation id="v-720" bandwidth="2200000" width="1280" height="720">
        <SegmentTemplate media="v720/seg_$Number%05d$.m4s"
                         initialization="v720/init.mp4"
                         duration="4" startNumber="1"/>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Read the manifest as data, not as a document to eyeball. Extract a normalized list of (variant_id, height, bandwidth, init_url, [segment_urls]) and feed that structure to the downloader. Your record from Module 1 then gains fields: duration, chosen_height, segment_count.

Note: Manifests are usually signed and time-limited. Parse a fresh manifest per job and never persist the segment URLs as a permanent cache key. Treat them as valid for one run only.

Practice 1. Point your parser at the HLS master above. Which URL do you fetch first if your target is 720p, and what field tells you the playlist is complete? Answer

Fetch v720/index.m3u8 (the variant playlist), then its #EXT-X-MAP init segment, then the media segments. #EXT-X-ENDLIST marks a finished VOD playlist; without it the playlist is live and the segment count can grow.

Choosing a Resolution on Purpose

Newcomers grab the highest rendition. In a long-running crawler that is the wrong default: it burns bandwidth, fills disk, and lengthens the window in which a job can fail.

Select by policy, not by greed:

def choose_variant(variants, max_height=720):
    eligible = [v for v in variants if v.height <= max_height]
    pool = eligible or variants
    return max(pool, key=lambda v: (v.height, v.bandwidth))

Rules that hold up in production:

• Set a max_height in config, per platform. 720p is a sane default for archival.
• Among variants at the same height, take the one with the highest bitrate (usually the best encode).
• If the preferred height is absent, fall back to the nearest lower height, not to the absolute minimum.
• Some renditions are video-only. Keep the best audio representation separately and mux it during merge.

Practice 2. A DASH manifest offers 1080p, 720p, 480p (video) and one audio-only Representation. Your config says max_height: 720. What do you download, and what would break if you skipped the audio representation? Answer

Take the 720p video representation plus the audio representation. Skipping audio yields a silent file: DASH and many fMP4 HLS streams keep video and audio in separate adaptation sets, so the merge step must combine both.

Fetching Fragment Sequences in Order

Fragments are numbered, and the number is the contract. Segment 7 must land at offset 7 in the output, whether you download them serially or in parallel.

The safe pattern:

1. Write the init segment first, always byte 0 of the output.
2. Download fragments with a bounded worker pool (4 to 8 is plenty), then reassemble by index.
3. Persist each fragment to a temp file keyed by job_id/segment_index. A crash resumes by skipping existing, size-checked fragments.
4. Reuse last lesson's machinery per fragment: timeout, retry with backoff, byte-range resume.
5. After the last fragment, assert len(fragments) == segment_count from the manifest. A gap means a corrupt timeline.

Byte-range playlists (#EXT-X-BYTERANGE) are the exception: all fragments live in one file, so you issue ranged requests and concatenate the byte slices.

Practice 3. Your worker pool downloads 120 fragments and one 502 kills fragment 63 permanently. What does your job do, and why is silently skipping it worse than failing? Answer

Retry with backoff, then fail the job and keep the downloaded fragments for the next attempt (resume). Skipping fragment 63 produces a file that plays but jumps ahead, corrupting duration and audio sync, and the corruption is invisible until someone reviews the archive.

Merging Segments into a Playable File

The merge depends on the segment format:

• MPEG-TS segments: raw concatenation works, cat seg_*.ts > out.ts. No init segment involved. Remux to MP4 with ffmpeg -i out.ts -c copy out.mp4.
• fMP4 segments: write init.mp4 first, then append seg_*.m4s in order. Use the concat demuxer for reliability:

ffmpeg -f concat -safe 0 -i list.txt -c copy -movflags +faststart out.mp4

Then verify, because a merged file that exists is not the same as a merged file that is correct:

ffprobe -v error -show_entries format=duration:stream=codec_name,width,height -of json out.mp4

Compare duration against the manifest (allow a few hundred ms of slack for segment rounding) and confirm resolution matches the variant you chose. Only after this check does the file move from tmp/ into your storage layout. That handoff is where the next lesson picks up.

Practice 4. ffprobe reports 9m12s for a video whose manifest sums to 10m00s. Name two plausible causes and the check that distinguishes them. Answer

Either missing fragments (segment count mismatch) or a playlist that was live and still growing when parsed (no #EXT-X-ENDLIST). Compare len(fragments) with the manifest's expected count first: a short count means dropped segments, a full count with short duration means the manifest itself was incomplete.

Field Notes from Real Video Crawls

• Prefer one variant per job. Rendition switching mid-download is a player feature, not a crawler feature.
• Assert the segment count before merging. Duration checks catch what file-size checks miss.
• Write to tmp/ and rename into storage. A half-merged MP4 in the archive is worse than no file.
• Log variant_id, segment_count and final duration on every job. When a platform changes its ladder, that log is how you notice.
• If the manifest has no #EXT-X-ENDLIST or a live MPD type, refuse the job rather than downloading an open-ended stream.
• Keep a cap on total fragments per job. A runaway count is usually a parsing bug, not a long video.

Taking This Into the Capstone

Video is a manifest plus a policy plus an ordered fetch plus a verified merge. Choose the variant from config, fetch fragments with the resume and retry logic you already built, merge with a format-aware step, then prove the result with ffprobe before it enters storage.

Key facts to carry forward:

• HLS uses .m3u8, DASH uses .mpd XML.
• fMP4 output needs the init segment written before any media fragment.
• Resolution is a config decision, not a per-url guess.
• Always verify duration and dimensions after merging.

Next, in Capstone Step 2, we turn these merged files and your downloaded images into a durable storage layer with content hashing, so the same video is never stored twice.

An HLS master playlist is a plain-text file with the _____ extension. A DASH manifest is an XML document whose root element is _____. For fMP4 streams, the initialization segment declared by _____ must be written before any media fragment. Choosing a rendition means trading _____ against bitrate and download time. After merging, validate duration and dimensions with _____.

Front: What file extension does an HLS manifest use? Back: A .m3u8 playlist, either a master playlist or a media playlist.

Front: What is the root format of a DASH manifest? Back: An XML document rooted at MPD (Media Presentation Description).

Front: What does an HLS master playlist list? Back: The available variant streams, each with BANDWIDTH and RESOLUTION, pointing to a media playlist.

Front: What does #EXT-X-MAP declare? Back: The initialization segment (init.mp4) that must be written before any fMP4 media segment.

Front: What does #EXTINF record? Back: The duration in seconds of the segment that follows it.

Front: What does #EXT-X-ENDLIST signal? Back: The playlist is complete VOD, so the segment count is final.

Front: How do you merge MPEG-TS segments? Back: Raw concatenation works, then remux to MP4 with ffmpeg -c copy.

Front: How do you merge fMP4 segments? Back: Write init.mp4 first, then append segments in index order via the concat demuxer.

Front: How should a crawler pick a resolution? Back: By config policy (max_height), taking the best bitrate at that height and falling back to the nearest lower height.

Front: Why keep audio as a separate representation? Back: DASH and fMP4 HLS often split video and audio, so the merge step must mux both or the output is silent.

Front: What does #EXT-X-BYTERANGE imply? Back: All segments live in one file, so you issue ranged requests and concatenate the byte slices.

Front: Which two checks validate a merged video? Back: Segment count matches the manifest, and ffprobe duration and dimensions match the chosen variant.

Front: Why re-parse the manifest for every job? Back: Manifest and segment URLs are usually signed and time-limited, so cached URLs expire.