🔍 How Facebook, Weibo and Bilibili Serve Media

By Lê Nguyễn Tuân

From a Post to Its Pixels

Every media crawler is really three questions asked in sequence: where is the descriptor, what does the URL look like, and how long does it live? This lesson answers those questions for Facebook, Weibo and Bilibili, because the capstone you build across this course will only be as stable as its understanding of how each platform hands out media links.

By the end of this lesson you can:

  • Describe the shared request flow from feed response to CDN bytes.
  • Name the media fields each platform exposes and why they differ.
  • Recognize signed, time-limited URLs and plan around their expiry.
  • Identify the fields your crawler must preserve for later modules.
  • State the access boundaries a long-lived tool must respect.
flowchart TD
  A["📄<br/>Feed response"] --> B["🔎<br/>Parse media descriptors"]
  B --> C["🔗<br/>Canonical media record"]
  C --> D["⬇️<br/>CDN fetch"]
  D --> E["🗄️<br/>Store with content hash"]

The Shape All Three Platforms Share

None of these platforms serve images or video from the HTML page itself. A feed or detail response returns a JSON payload containing media descriptors: a URL, an identifier, dimensions, and often a container or codec hint. The bytes then come from a CDN host on a different domain, frequently one of many mirrors.

Three properties follow from that shape, and they drive the whole capstone:

  1. The payload is the source of truth, not the rendered page.
  2. The URL is a lease, not a permanent address, and some leases expire.
  3. A record can carry several equivalent URLs, and picking a good one is a design decision.

Facebook: GraphQL Nodes and Signed Links

Public Facebook pages and posts deliver media inside GraphQL responses. Images appear as nodes with a uri plus width and height, and the uri points at *.fbcdn.net. Those URLs carry signature parameters and an expiry value, commonly a hex timestamp in the oe parameter. Once that timestamp passes, the same URL returns a 403 even though nothing about the file changed.

Videos are described in delivery fields that expose progressive MP4 variants and adaptive manifests. The adaptive path separates video and audio representations, so a complete video file may require two downloads and a merge step, which is exactly what Module 2.2 covers.

Practical consequence: parse the payload, capture every variant, and download promptly. Do not cache Facebook CDN URLs in your database as if they were durable identifiers.

Exercise: which field is your stable key?

A colleague stores the full fbcdn.net URL as the primary key for each image. Two days later, half the records fail to download. What is wrong and what should the key be?

The URL is a signed, expiring lease, not an identifier. Use a stable media identifier from the payload (post id plus photo or video id) as the key, and keep the CDN URL as a replaceable attribute with an expiry timestamp beside it.

Weibo: Predictable Image Servers and Stream Lists

Weibo is the opposite case. Image URLs are largely deterministic: a pid from the status, combined with a path segment that selects the size (thumbnail, bmiddle, large), and a host on a sinaimg.cn mirror. Because the size is encoded in the path rather than a signature, you can often construct the variant you want directly from the pid, which makes deduplication and re-fetching far simpler than on Facebook.

Videos live in page_info, which exposes media info with stream URLs, sometimes a high-definition variant, and a map of quality keys to MP4 URLs. The lesson for your crawler: normalize these into an ordered list of candidates rather than picking the first entry you see.

Exercise: dedupe across sizes

You crawl the same Weibo post twice and collect large, bmiddle and thumbnail URLs for one picture. How do you avoid storing three copies?

Derive the identity from the pid (plus the post id), keep the size variants as attributes of one record, and fetch only the highest size your policy allows. Content hashing in Module 2.3 gives you a second safety net when a platform re-hosts the same file under a new path.

Bilibili: Identifiers, Playurl, and Split Tracks

Bilibili separates identity from delivery more cleanly than the other two. A video item is addressed by bvid (or the numeric aid), and each part inside a multi-part post has its own cid. Image assets sit on hdslb.com hosts and accept an @ suffix that requests a processed variant, for example a resized WebP.

Playback data comes from the player endpoint keyed by bvid and cid. It returns a dash object with a video array and an audio array. Each entry carries a base URL, a list of backup URLs, a quality id, a codec id (commonly 7 for AVC, 12 for HEVC, 13 for AV1), and a bandwidth value. URLs include a deadline parameter and are time-limited.

So for Bilibili your crawler must: resolve cid first, choose a quality and codec policy, download both tracks, then merge. Choose the codec your remux pipeline can actually handle rather than the smallest file.

Exercise: choose a download policy

You want maximum compatibility for an archive that will be played on older devices. Which codecid do you pick, and what do you do with backupUrl?

Prefer AVC (codecid 7) for broad playback support, and keep backups in the record so a failed host can be retried immediately instead of re-resolving the whole playurl response.

Fields Your Crawler Must Keep

Across all three platforms your internal record needs the same core set, and this is the contract the next two lessons build on:

  • A stable media id derived from platform ids, never from the URL.
  • The primary URL plus every backup or alternate variant.
  • Fixup metadata: expiry timestamp, required referrer or headers, host family.
  • Container facts: MIME type, codec, width, height, bitrate or bandwidth.
  • Provenance: post id, author, capture time, and the endpoint that produced it.

Access Boundaries a Long-Lived Tool Respects

A crawler built to run for years survives on restraint, not on tricks this course will not teach. Fetch only content that is publicly reachable, honor robots directives and platform terms, stay inside published rate limits, and never attempt to defeat authentication, paywalls or DRM. Store only media you have the right to keep, and record the license context alongside it. The architecture modules assume these boundaries, so build the habit now.

Field Notes from the Fetch Path

A few rules of thumb earned the hard way:

  • Treat every media URL as expiring within hours. Resolve, download, store, then forget the URL.
  • Log the endpoint and the payload timestamp with each descriptor. When something breaks in six months, this is what tells you why.
  • Never retry a 403 from a signed CDN URL more than once. Re-resolve from the payload instead; the retry will fail identically.
  • Prefer explicit identifiers over regexing filenames out of URLs.
  • Respect rate limits per host family. CDN mirrors will also throttle you, not just the API.

What to Carry Forward

Media delivery on all three platforms reduces to the same three steps: find the descriptor in the payload, treat the URL as a short-lived lease with backups, and keep a stable identifier that has nothing to do with the link. Bilibili adds identifier and track splitting, Weibo adds predictable paths, Facebook adds hard expiry, and your canonical record absorbs all three differences.

Next, in Capstone Blueprint: Config, Storage Layout, Pipeline Modules, you turn these observations into the concrete project skeleton: a config file, a storage layout on disk, and the module boundaries that every later lesson will fill in.

Facebook image links on CDN hosts carry a signature and an expiry, so a URL that worked yesterday may return _____ today. On Weibo, the same picture can be requested at several sizes because the size segment (thumbnail, bmiddle, large) sits in the _____ rather than in a query parameter. Bilibili's playurl response splits a video into separate _____ video and audio arrays that must be merged after download. The Bilibili field that identifies one specific stream inside a multi-part post is the _____. A crawler should store _____ URLs next to the primary media URL, because CDN hosts fail independently.

Front: What three artifacts make up the media delivery path on all three platforms? Back: A response payload, a media descriptor inside it, and the bytes served by a CDN host.

Front: Why do Facebook CDN image URLs stop working after a while? Back: They are signed and carry an expiry timestamp, so they return 403 once the lease ends.

Front: What does the Weibo pid identify? Back: The image file identity, from which size-specific URLs can be constructed.

Front: Which Weibo field holds video stream variants? Back: page_info, which exposes media info with stream URLs and quality keys.

Front: What does a Bilibili cid identify? Back: One specific stream or part inside a video item, needed to request playback data.

Front: What structure does Bilibili playurl return? Back: A dash object with a video array and an audio array, each entry carrying base and backup URLs.

Front: On Bilibili, what does codecid 7 mean? Back: AVC (H.264), the most broadly compatible choice; 12 is HEVC and 13 is AV1.

Front: Why does Bilibili require a merge step after download? Back: Video and audio are delivered as separate DASH tracks, not as one combined file.

Front: Which stable key should replace a media URL in your database? Back: A platform-derived media id, never the CDN URL.

Front: What should you do when a signed media URL returns 403? Back: Re-resolve it from the payload instead of retrying the dead URL.

Front: Why keep backup URLs in a media record? Back: Mirrors fail independently, so a fallback avoids a full re-resolve round trip.

Front: What access boundaries should a long-lived crawler respect? Back: Public content only, robots and platform terms honored, rate limits followed, no DRM or auth bypass, and only media you may keep.