Capstone Step 1: Normalize Media Records from Feed Responses

By Lê Nguyễn Tuân

In the blueprint lesson you fixed the storage layout and the pipeline modules: fetch, parse, normalize, download, store. This lesson builds the middle one. Everything downstream (downloads, dedupe, queues) reads the same struct, so getting it right here saves rework later.

Answer No answer needed here, this is the framing paragraph. Skip ahead to the spec.

One Response In, Many Records Out

A single feed response is not one media item. One post may carry four images, another a video and a cover image, a third nothing at all. The job of this slice is to flatten that nesting into a flat list of records, one per media asset.

The rule that keeps the pipeline sane: the normalizer is the only code that knows what a platform response looks like. From the next module onward, the downloader sees only records, never raw JSON.

flowchart TD
  Raw["📥<br/>Saved response"] --> Adapter["🔌<br/>Platform adapter"]
  Adapter --> Norm["🧩<br/>Normalizer"]
  Norm --> Recs["🗂️<br/>MediaRecord list"]
  Recs --> Store["💾<br/>Record store"]

An adapter is a small per-platform module exposing three functions: post_id(raw), assets(raw), and canonical_url(asset). The normalizer loops over assets() and builds records. Adding a platform later means adding one adapter file, not touching the pipeline.

Important: Work from response fixtures you captured yourself, from accounts you own or have explicit permission to analyze. Respect each platform's terms of service and access rules, keep request volume low, and never share raw fixtures or cookies in a repository.

The Record Spec

Six required fields, all serializable to JSON:

Deliberately absent: author names, handles, profile URLs, raw captions, cookies. If you later need to group records per author, store a salted hash of the author id as author_key and nothing more. Metadata you never collect cannot leak.

@dataclass(frozen=True)
class MediaRecord:
    source: str
    post_id: str
    media_index: int
    media_type: str
    url: str
    capture_time: str  # ISO 8601, UTC

    def key(self) -> str:
        return f"{self.source}:{self.post_id}:{self.media_index}"

key() is the identity of the asset. You will lean on it in Module 3 when dedupe arrives; introduce it now so callers never invent their own string format.

Field names differ per platform. The adapter maps them, not the record:

def normalize(source, raw, capture_time):
    a = ADAPTERS[source]
    return [
        MediaRecord(
            source=source,
            post_id=a.post_id(raw),
            media_index=i,
            media_type=a.media_type(asset),
            url=a.canonical_url(asset),
            capture_time=capture_time,
        )
        for i, asset in enumerate(a.assets(raw))
    ]

The normalizer should return [] for a post with no media, never None. Callers get one contract.

The record's fixed shape is _____. Timestamps are always stored in _____ and serialized as ISO 8601. A post with four images produces _____. Instead of keeping raw captions or author handles, the record stores _____.

Exercise 1

A Bilibili response contains a post with a video and a cover image. How many records, and what distinguishes them?

Answer

Two records. Same source, post_id and capture_time; media_index 0 and 1; media_type differs (video and image). Canonical URLs must not collide, so key() stays unique.

Order of Work

1. Freeze three fixtures, one per platform, in a scratch folder outside the repo. Note the capture time for each.
2. Write the MediaRecord dataclass plus key(), and a test that round-trips it through JSON.
3. Write the Facebook adapter first. It is usually the flattest of the three.
4. Add the Weibo and Bilibili adapters, comparing field names against your fixtures rather than guessing.
5. Wire normalize() behind a small CLI: python -m crawler.normalize fixtures/fb_001.json prints the record list.
6. Add a fixture with zero media and one with a malformed asset, and assert both behave as specified.

Definition of Done

• Running the CLI on each fixture prints valid JSON, one object per asset, sorted by media_index.
• Every URL is absolute and resolves without the crawler adding query parameters later.
• Timestamps are UTC and end in Z or +00:00; no naive local times.
• A post with no media yields [] and exit code 0.
• A malformed asset is skipped with a warning to stderr, not a crash.
• grep -ri over the output finds no author names, handles, captions, or tokens.

Exercise 2

Your fixture parses, but two records share the same key(). What is the likely bug?

Answer

media_index is not derived from the asset's position, or two assets were merged during adapter mapping. Check that enumerate runs over the deduplicated asset list, and that the adapter does not return the same asset twice (for example, a video plus its poster frame pointing at one element).

Hints and Pitfalls

Warning: The most common mistake is normalizing too early: pulling a URL inside the adapter and losing the asset object. Keep the adapter returning structured assets, and let the normalizer call canonical_url().

• Relative URLs. Some responses give a path fragment. Resolve against the platform's base before building the record, not at download time.
• Capture time drift. Use the time the response was saved, not datetime.now() at parse time. Reparsing an old fixture must not change the record.
• media_type guessing. Do not infer type from the file extension alone. Trust the response's asset kind, and fall back to extension only when the kind is absent.
• Silent drops. Count parsed assets versus emitted records and log the difference. A missing record is invisible until a download never happens.
• Fixtures in git. Keep raw responses and any tokens out of the repository; commit only the small hand-written test cases you constructed.

What You Have Now

The capstone now has a working normalization slice: a frozen six-field record, one adapter per platform, key() for stable identity, and a CLI that turns a saved response into JSON. Next, the checkpoint lesson audits the whole module: you will trace one media item from platform serving through the skeleton to a normalized record, and confirm the module objective before download engineering begins.

Front: What six fields make up a normalized MediaRecord? Back: source, post_id, media_index, media_type, url, capture_time.

Front: Why is the record flat instead of nested like the response? Back: Downstream modules read one record per asset and never parse platform JSON.

Front: What three functions does a platform adapter expose? Back: post_id(raw), assets(raw), canonical_url(asset).

Front: What does key() return? Back: source:post_id:media_index, the stable identity of one asset.

Front: What time goes into capture_time? Back: When the response was captured, in UTC, as ISO 8601.

Front: Why must capture_time not use datetime.now() at parse time? Back: Reparsing an old fixture would change the record and break reproducibility.

Front: What should normalize() return for a post with no media? Back: An empty list, never None.

Front: How many records does a post with one video and one cover image produce? Back: Two, distinguished by media_index and media_type.

Front: Where should relative media URLs be resolved? Back: In the adapter, before the record is built, so records always hold absolute URLs.

Front: What replaces raw author handles and captions in the record? Back: Nothing, or a salted author hash if grouping is required.

Front: Why keep raw fixtures out of the repository? Back: They contain personal content and possibly tokens; commit only hand-built test cases.

Front: What is the symptom of a duplicated media_index? Back: Two records share the same key(), so identity and dedupe break.

Front: How should a malformed asset be handled? Back: Skipped with a warning, while the rest of the post still normalizes.