Coordinated Origin Range Requests in Pingora: Cutting Latency in Half with Automatic Partial Fetches

Introduction

Serving large static assets—videos, datasets, firmware bundles—has always been a battle between origin bandwidth consumption and edge latency. The naïve approach, popularized by early CDN designs, caches whole objects on the edge and falls back to byte‑range requests when a client asks for a slice. In practice, that model leaks massive amounts of upstream traffic because every new range request that misses the cache forces the origin to re‑stream the same bytes over and over.

In a recent Cloudflare engineering post, the team disclosed a radical redesign of their cache engine, Pingora, that coordinates partial fetches across concurrent requests. By turning independent range requests into a single coordinated fetch, they slashed the 99th‑percentile latency (R2 p99) by ~50 % and saved gigabytes of origin egress.

This editorial unpacks the under‑the‑hood mechanics, critiques common pitfalls, provides a Rust blueprint, and projects where this technology will head in the next 18‑36 months.

Deep Architectural Breakdown

1. Traditional Byte‑Range Caching Pitfalls

The core issue is state isolation: each range request is treated as a distinct object. The cache never knows that the two requests belong to the same logical file, so it cannot reuse already‑fetched bytes. The result is linear scaling of origin traffic with the number of distinct ranges, which is catastrophic for large files accessed in many tiny chunks (e.g., video streaming, resumable uploads).

2. Pingora’s Coordinated Partial Fetch Engine

Pingora, Cloudflare’s Rust‑based reverse‑proxy/cache, introduced a Partial‑Fetch Coordinator (PFC) that sits between the HTTP request dispatcher and the origin fetcher. Its responsibilities are:

1. Range Normalization – Convert arbitrary client ranges into a canonical interval list per object.
2. Fetch Coalescing – Merge overlapping or adjacent missing intervals across concurrent requests.
3. Shared Buffer Pool – Allocate a single in‑memory buffer per missing interval, backed by an arena allocator to avoid per‑request heap churn.
4. Completion Notification – Broadcast fetched bytes to all waiting request contexts via a lock‑free MPSC (multi‑producer, single‑consumer) channel.

Memory Model and Buffer Management

• Arena‑Based Allocation: Pingora pre‑allocates a 64 MiB arena per worker thread. Each fetch interval reserves a slice from the arena; when the interval completes, the slice is returned to the arena without invoking the system allocator. This design yields < 5 µs allocation latency and eliminates fragmentation.
• Reference‑Counted Views: Waiting requests hold a Arc<Bytes> pointing into the arena slice. The reference count drops to zero only after every dependent request has streamed the data downstream, guaranteeing zero‑copy transmission to the client.
• Back‑Pressure Integration: The PFC respects the downstream write‑ready signal. If a client is throttled, the buffer stays resident but is not copied further, preventing head‑of‑line blocking.

Request Coalescing Algorithm (Pseudo‑code)

// Simplified Rust‑like pseudocode
fn handle_range_req(req: &Request) -> Response {
    let key = req.uri.clone();
    let ranges = parse_range_header(req.headers["Range"]);
    let mut missing = Vec::new();
    // 1️⃣ Look up cached intervals
    for r in &ranges {
        if !cache.contains_interval(&key, r) {
            missing.push(r.clone());
        }
    }
    // 2️⃣ Register pending fetches with the coordinator
    let fetch_fut = coordinator.fetch_or_subscribe(key.clone(), missing);
    // 3️⃣ Await completion and stream
    async move {
        let fetched = fetch_fut.await; // resolves when all missing intervals are ready
        assemble_response(&cached_parts, &fetched)
    }
}

The coordinator maintains a hashmap HashMap<(Key, Interval), FetchState>. When a new request arrives:

• If FetchState::InProgress exists for the interval, the request subscribes to the existing future.
• If not, the coordinator spawns a single origin fetch for the union of all missing intervals and records the future.
• Upon completion, the future resolves, the buffer is stored in the cache, and all subscribers are notified.

Latency Path & Compute Dynamics

By collapsing N overlapping range requests into a single origin fetch, Pingora eliminates the N‑1 extra network round‑trips and the associated TCP slow‑start penalties. The measured p99 latency reduction (≈ 50 %) aligns with the theoretical latency saved: p99 ≈ (N‑1) * RTT + origin‑fetch‑overhead.

3. Integration with Cloudflare R2 and Origin Servers

R2, Cloudflare’s S3‑compatible object store, emits byte‑range support natively. Pingora’s PFC simply forwards the union of missing intervals as a single Range: bytes=0‑524287 request. The origin responds with a 206 Partial Content and a Content-Range header. Pingora then slices the response stream into the original sub‑ranges before delivering to each waiting client.

Key integration points:

• ETag / If‑Range Validation – The coordinator caches the ETag of the underlying object. If any subscriber presents a stale If‑Range, the fetch is aborted and a full‑object validation is triggered.
• Cache‑Control Propagation – Pingora respects Cache‑Control: max‑age on partial responses, storing each interval with its own TTL while preserving the overall object freshness semantics.
• Error Handling – A partial fetch failure (e.g., 416 Range Not Satisfiable) triggers a fallback full‑object fetch for the missing intervals, ensuring robustness.

FutureStack Perspective (Mubashir Arham)

*When I first saw the “range‑request tuning” blog, my immediate reaction was: “Why are we still treating a range request as a first‑class cache key?” In most legacy edge platforms, the cache key is a concatenation of URI + Range Header. That design decision seems innocent but it fundamentally breaks the principle of idempotent storage: the same logical object should have a single canonical representation in the cache.

At FutureStack, we have built several multi‑tenant edge services where a single video file is streamed to millions of users, each with a different start offset. Our initial implementation suffered exactly the same symptom—gigabytes of redundant origin traffic per hour. The naive fix was to increase the cache size, but that merely postponed the bandwidth bill.

Pingora’s Partial‑Fetch Coordinator is a textbook example of co‑design between network protocol semantics and storage layout. By elevating the interval‑coalescing logic to the cache layer, they turned a stateless HTTP primitive into a stateful, bandwidth‑aware optimizer.

Where teams stumble:

1. Treating Range as a Cache Dimension – Most CDN SDKs expose range as part of the key. The result is a combinatorial explosion of keys.
2. Neglecting Back‑Pressure – Without tying the coordinator to downstream flow control, you can flood the origin with a burst of large range unions, causing head‑of‑line latency spikes.
3. Static Buffer Pools – Allocating a fresh buffer per request leads to GC pressure (in Go/Java) or heap fragmentation (in C++). Pingora’s arena model is a clean solution that scales across thousands of concurrent fetches.

Our take: When building an edge cache, start with a single canonical object abstraction and layer interval metadata on top. Use lock‑free data structures for the subscription map, and always expose a back‑pressure hook to the origin fetcher. The payoff is not just lower latency—it’s a measurable reduction in egress cost, which directly translates to a better unit economics for any SaaS that relies on large‑file distribution.

In the next iteration of FutureStack’s EdgeStore, we plan to adopt a similar coordinator but with predictive pre‑fetch based on client telemetry (e.g., seeking patterns). That will push the latency envelope even further, edging toward sub‑10 ms p99 for popular media assets.

Code / Implementation Blueprint (Rust)

Below is a minimal, self‑contained Rust example that demonstrates the core ideas of Pingora’s partial‑fetch coordination. It uses tokio for async I/O, bytes::Bytes for zero‑copy buffers, and a lock‑free DashMap for the subscription registry.

use bytes::Bytes;
use dashmap::DashMap;
use std::sync::Arc;
use tokio::sync::{Notify, RwLock};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use hyper::{Request, Response, Body, StatusCode};

// ---------- Types ----------
type ObjectKey = String; // e.g., "bucket/video.mp4"
#[derive(Clone, PartialEq, Eq, Hash, Debug)]
struct ByteRange { start: u64, end: u64 }

// Holds a future that resolves when the interval is fetched.
struct FetchState {
    notify: Arc<Notify>,
    // Filled once the origin response arrives.
    data: RwLock<Option<Bytes>>, 
}

// Global coordinator (in a real service this would be per‑worker).
static COORDINATOR: once_cell::sync::Lazy<DashMap<(ObjectKey, ByteRange), Arc<FetchState>>> =
    once_cell::sync::Lazy::new(DashMap::new);

// ---------- Helper Functions ----------
fn parse_range(header: &str) -> Vec<ByteRange> {
    // Very naive parser: "bytes=0-1023,2048-3071"
    header.trim_start_matches("bytes=")
        .split(',')
        .filter_map(|part| {
            let mut split = part.split('-');
            let s = split.next()?.parse().ok()?;
            let e = split.next()?.parse().ok()?;
            Some(ByteRange { start: s, end: e })
        })
        .collect()
}

async fn fetch_from_origin(key: &ObjectKey, range: &ByteRange) -> Bytes {
    // Placeholder: in production this would be an async HTTP client.
    // Here we just allocate a buffer filled with dummy data.
    let size = (range.end - range.start + 1) as usize;
    let mut buf = vec![0u8; size];
    for i in 0..size { buf[i] = ((range.start + i as u64) % 256) as u8; }
    Bytes::from(buf)
}

// ---------- Core Coordinator Logic ----------
async fn coordinate_fetch(key: ObjectKey, missing: Vec<ByteRange>) -> Vec<(ByteRange, Bytes)> {
    let mut handles = Vec::new();
    for range in missing {
        let entry_key = (key.clone(), range.clone());
        let state = COORDINATOR.entry(entry_key.clone()).or_insert_with(|| {
            Arc::new(FetchState { notify: Arc::new(Notify::new()), data: RwLock::new(None) })
        }).clone();

        // If this is the first requester, spawn the origin fetch.
        if state.data.read().await.is_none() {
            let fetch_state = state.clone();
            let key_clone = key.clone();
            let range_clone = range.clone();
            tokio::spawn(async move {
                let payload = fetch_from_origin(&key_clone, &range_clone).await;
                *fetch_state.data.write().await = Some(payload);
                fetch_state.notify.notify_waiters();
            });
        }
        handles.push((range, state));
    }

    // Await all fetches.
    let mut results = Vec::new();
    for (range, state) in handles {
        // Wait until the payload is ready.
        loop {
            if let Some(buf) = state.data.read().await.clone() {
                results.push((range.clone(), buf));
                break;
            }
            state.notify.notified().await;
        }
    }
    results
}

// ---------- Request Handler ----------
pub async fn handle(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
    let uri = req.uri().path().to_string(); // Simplified object key.
    let range_header = req.headers().get("Range").and_then(|v| v.to_str().ok()).unwrap_or("bytes=0-0");
    let ranges = parse_range(range_header);

    // In a real cache we would check stored intervals first.
    // For brevity we assume everything is missing.
    let fetched = coordinate_fetch(uri.clone(), ranges.clone()).await;

    // Assemble a multipart/byteranges response (simplified to single range).
    if fetched.len() == 1 {
        let (_, data) = &fetched[0];
        let mut resp = Response::new(Body::from(data.clone()));
        *resp.status_mut() = StatusCode::PARTIAL_CONTENT;
        resp.headers_mut().insert("Content-Range", format!("bytes {}-{}/*", ranges[0].start, ranges[0].end).parse().unwrap());
        Ok(resp)
    } else {
        // For multiple ranges we would build a multipart body.
        Ok(Response::builder()
            .status(StatusCode::OK)
            .body(Body::from("multiple ranges not implemented"))?)
    }
}

Explanation of the Blueprint

• DashMap provides a lock‑free concurrent hashmap for the coordinator registry.
• Each FetchState contains a Notify primitive; waiting requests are awakened as soon as the origin payload arrives.
• The coordinate_fetch function guarantees at most one origin fetch per (key, interval) pair, regardless of how many concurrent callers arrive.
• The example uses an in‑process dummy origin fetch; swapping in a real hyper::Client with proper TLS is a drop‑in change.

Empirical Trade‑Off Table

Future Industry Trajectory (18‑36 months)

1. Predictive Interval Pre‑Fetching – Machine‑learning models will infer the next likely byte range (e.g., video seek patterns) and proactively issue the union fetch before the client asks. This will push p99 latency toward the sub‑10 ms regime for hot media.
2. Standardized Cache‑Control for Ranges – HTTP/3 and QUIC extensions are already discussing Cache‑Range directives that allow origins to hint at range granularity and prefetch windows. Edge platforms will expose these hints to the coordinator, reducing unnecessary fetches.
3. Edge‑Native Storage Engines – Instead of treating the cache as a simple key/value store, future CDNs will embed log‑structured merge trees (LSM) that natively index byte intervals, enabling O(log n) lookups for partially cached objects.
4. Cross‑Region Coordinated Fetch – As multi‑region edge fabrics mature, a request in Europe could subscribe to a fetch that originated in North America, leveraging inter‑edge replication for large files that are rarely fully cached anywhere.
5. Zero‑Copy End‑to‑End Encryption – With TLS‑13 0‑RTT and QUIC, the buffer that the coordinator hands to the client can stay encrypted end‑to‑end, eliminating the need for decryption/re‑encryption at the edge while still benefiting from partial fetch coordination.

Takeaway for Engineers

• Adopt interval‑aware caching as a first‑class design principle, not an after‑thought optimization.
• Invest in lock‑free coordination primitives early; retrofitting them later is costly.
• Keep an eye on emerging HTTP extensions that expose range‑specific cache directives; they will become the lingua franca for bandwidth‑aware edge services.

By internalizing these lessons, engineering teams can avoid the classic bandwidth‑leak trap and deliver the ultra‑low latency experiences users now expect from streaming, data‑science pipelines, and large‑file SaaS workloads.

Prepared by Mubashir Arham, Founder & Chief Architect, BS Artificial Intelligence, FutureStack