PDF.js Source Walkthrough: chunked_stream

Introduction

This is the first article in the "Bytes and Streams" layer.

The previous articles kept moving upward: how PDFManager sets up a document, how XRef locates objects, and how PartialEvaluator interprets a content stream into an operatorList. But all of that work rests on a plain assumption — the bytes are already there. This article goes back to the very bottom and answers one question: how does a PDF file on the network become a byte stream that the parser can seek through at will?

PDF.js has two byte-supply paths. For a local file, or an in-memory ArrayBuffer, it constructs a Stream directly and keeps the whole file in memory. For a URL with HTTP Range, it constructs a ChunkedStream: the file is divided into fixed-size chunks, and only the chunks that are actually read get downloaded.

Why not simply "download everything, then parse"? Because PDF reading is inherently random access. startxref sits at the end of the file; the xref section points at 12 0 obj entries scattered across the file; page content streams, fonts and images each live at arbitrary offsets. Right after reading the header, the next line may need to jump tens of megabytes away.

PDF.js separates "the logical file" from "the bytes that have physically arrived". ChunkedStream allocates a Uint8Array of the full file length at construction, so every offset is addressable, but reading a chunk that has not arrived throws MissingDataException. ChunkedStreamManager translates "which chunk is missing" into Range requests, writes the responses back into the buffer, and wakes the parsing logic that is waiting.

In one sentence: ChunkedStream is a byte array with a page-fault mechanism, and ChunkedStreamManager is its page-fault handler.

Version note: This article follows the version of chunked_stream.js in the current repository. The ChunkedStream state fields are publicly named (progressiveDataLength, _loadedChunks), while the manager mixes private fields (#aborted, #loadedStreamCapability) with underscore-prefixed fields. The code uses Promise.withResolvers and Map.prototype.getOrInsertComputed, so do not map it onto the _loadedChunks implementation or hand-written deferred objects of older versions.

Module Overview

chunked_stream.js is 542 lines long and splits into seven parts:

Where it sits in the chain:

The whole chain compresses into one sentence: the network provides slices, the manager schedules those slices into chunks, ChunkedStream disguises the chunks as a complete file, and the parser never knows it is reading a file downloaded on demand.

Two Kinds of Byte Supply: Stream and ChunkedStream

Stream is only about seventy lines long, and it assumes bytes is already complete. ChunkedStream extends it, keeps the same read API, but inserts a "is this chunk loaded" check before every read.

Preallocating the full-file buffer looks extravagant: a 50 MB PDF immediately occupies 50 MB. But it lets the parser read and write at any offset, makes zero-copy subarray views depend on the same contiguous memory, and keeps chunk-state queries at O(1). The cost is that the memory ceiling equals the file size, which for a browser reader is the simpler and more controllable side of the trade.

Design note: ChunkedStream splits "length" and "availability" into two orthogonal concepts. length comes from the HTTP response headers and determines the buffer size and total chunk count; _loadedChunks determines what can be read right now. The parser depends only on the former, the scheduler maintains only the latter. Any implementation that mixes the two degrades into a full-file download in the "long file, but only the tail has been read" scenario.

ChunkedStream: Preallocated Buffer and Missing-Data Checks

Construction and State

class ChunkedStream extends Stream {
  progressiveDataLength = 0;
 
  _lastSuccessfulEnsureByteChunk = -1; // Single-entry cache
 
  _loadedChunks = new Set();
 
  constructor(length, chunkSize, manager) {
    super(
      /* arrayBuffer = */ new Uint8Array(length),
      /* start = */ 0,
      /* length = */ length,
      /* dict = */ null
    );
 
    this.chunkSize = chunkSize;
    this.numChunks = Math.ceil(length / chunkSize);
    this.manager = manager;
  }

The constructor does only four things: allocate the buffer for the file length, set the start to 0, compute the chunk count from rangeChunkSize, and keep a reference to the manager. It sends no request at all — object creation is synchronous and cheap, and the real data only starts flowing after the first read triggers a page fault.

progressiveDataLength exists for the mode where the entire response streams down sequentially instead of arriving as Range responses. In that mode the network layer keeps calling onReceiveProgressiveData, and data is appended piece by piece starting at 0. As long as the read position falls inside that contiguous prefix, the data cannot be missing, and not even the Set needs to be consulted.

_lastSuccessfulEnsureByteChunk is a tiny optimization: the parser often reads dozens of consecutive bytes inside the same chunk, and querying the Set every time is wasteful. Remembering the last successful chunk number makes a hit return immediately. Since a chunk is never unloaded once loaded, this cache never goes stale.

onReceiveData: Filling the Buffer from a Range Response

  onReceiveData(begin, chunk) {
    const chunkSize = this.chunkSize;
    if (begin % chunkSize !== 0) {
      throw new Error(`Bad begin offset: ${begin}`);
    }
 
    // Using `this.length` is inaccurate here since `this.start` can be moved
    // (see the `moveStart` method).
    const end = begin + chunk.byteLength;
    if (end % chunkSize !== 0 && end !== this.bytes.length) {
      throw new Error(`Bad end offset: ${end}`);
    }
 
    if (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) {
      assert(
        chunk instanceof ArrayBuffer,
        "onReceiveData - expected an ArrayBuffer."
      );
    }
    this.bytes.set(new Uint8Array(chunk), begin);
    const beginChunk = Math.floor(begin / chunkSize);
    const endChunk = Math.floor((end - 1) / chunkSize) + 1;
 
    for (let curChunk = beginChunk; curChunk < endChunk; ++curChunk) {
      // Since a value can only occur *once* in a `Set`, there's no need to
      // manually check `Set.prototype.has()` before adding the value here.
      this._loadedChunks.add(curChunk);
    }
  }

This is where a Range response lands. Four details are worth expanding:

1. The start must align to a chunk boundary. The scheduler issues requests chunk by chunk, so begin is always a multiple of chunkSize; a misalignment means the layer above computed something wrong.
2. The end is either aligned or exactly the end of the file. The final chunk may be short; apart from that, a response must cover whole chunks, and a server that arbitrarily returns half a chunk is rejected.
3. The end check uses bytes.length, not this.length. this.start can be moved by moveStart, which makes the length getter unreliable for detecting the end of the file; the real capacity of the buffer is the trustworthy upper bound.
4. Chunk registration deduplicates through a Set. The server may merge Range requests and return more data than asked for, and Set.add is naturally idempotent, saving a has lookup.

Chunk-number conversion is just Math.floor(begin / chunkSize) and Math.floor((end - 1) / chunkSize) + 1, and the half-open interval never counts one chunk too many.

onReceiveProgressiveData: Appending a Sequential Stream

  onReceiveProgressiveData(data) {
    let position = this.progressiveDataLength;
    const beginChunk = Math.floor(position / this.chunkSize);
 
    if (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) {
      assert(
        data instanceof ArrayBuffer,
        "onReceiveProgressiveData - expected an ArrayBuffer."
      );
    }
    this.bytes.set(new Uint8Array(data), position);
    position += data.byteLength;
    this.progressiveDataLength = position;
    const endChunk =
      position >= this.end
        ? this.numChunks
        : Math.floor(position / this.chunkSize);
 
    for (let curChunk = beginChunk; curChunk < endChunk; ++curChunk) {
      // Since a value can only occur *once* in a `Set`, there's no need to
      // manually check `Set.prototype.has()` before adding the value here.
      this._loadedChunks.add(curChunk);
    }
  }

Sequential mode has no begin: data always follows progressiveDataLength. Chunk registration here is more conservative than in Range mode:

• A chunk is only counted once Math.floor(position / chunkSize) shows that the data has filled it.
• If this write happens to finish the file (position reaches this.end), every chunk up to numChunks is marked as loaded at once — the last chunk has genuinely arrived, even if it is short.

So in sequential mode, progressiveDataLength usually runs ahead of "the coverage of the registered chunks". Reads check progressiveDataLength first, exploiting exactly that: the prefix is contiguous, so it is fine for the chunk markers to lag a little.

Design note: _loadedChunks is not the only source of truth for availability. Range mode consults the chunk set, sequential mode consults the contiguous prefix, and the two paths meet in ensureByte and ensureRange. This dual-track design avoids the complexity of registering every half chunk in sequential mode, while keeping progress reporting chunk-based.

ensureByte and ensureRange

These two methods are the page-fault entry points for the whole streaming parser. They do not return data; they answer one question: can this position be read right now.

  ensureByte(pos) {
    if (pos < this.progressiveDataLength) {
      return;
    }
 
    const chunk = Math.floor(pos / this.chunkSize);
    if (chunk > this.numChunks) {
      return;
    }
    if (chunk === this._lastSuccessfulEnsureByteChunk) {
      return;
    }
 
    if (!this._loadedChunks.has(chunk)) {
      throw new MissingDataException(pos, pos + 1);
    }
    this._lastSuccessfulEnsureByteChunk = chunk;
  }

ensureByte has four fast paths, and their order is itself a design decision:

Path 2 uses the loose upper bound chunk > this.numChunks. Valid chunk numbers are already less than numChunks, so this check merely lets obviously out-of-range reads through; the real end of file is handled by pos >= this.end inside getByte, so callers never read garbage.

  ensureRange(begin, end) {
    if (begin >= end) {
      return;
    }
    if (end <= this.progressiveDataLength) {
      return;
    }
 
    const beginChunk = Math.floor(begin / this.chunkSize);
    if (beginChunk > this.numChunks) {
      return;
    }
    const endChunk = Math.min(
      Math.floor((end - 1) / this.chunkSize) + 1,
      this.numChunks
    );
    for (let chunk = beginChunk; chunk < endChunk; ++chunk) {
      if (!this._loadedChunks.has(chunk)) {
        throw new MissingDataException(begin, end);
      }
    }
  }

ensureRange is the interval version of ensureByte, used by getBytes and makeSubStream, which need many bytes at once. It has three notable properties:

1. Empty intervals pass straight through. begin >= end means there are no bytes to read, so any buffer state satisfies it.
2. A contiguous prefix passes straight through. When end <= progressiveDataLength, the entire interval lies inside sequential data and no per-chunk check is needed.
3. The exception carries the caller's full requested interval, not the first missing chunk. This is deliberate: the caller needs a whole span, and the scheduler can merge and deduplicate it as needed. If only the first gap were reported, filling it would produce another exception for the next gap, multiplying the round trips.

The interval in the exception may include chunks that are already loaded, and that wastes nothing. _requestChunks first filters out chunks where hasChunk is true, and groupChunks then merges the rest into as few contiguous ranges as possible.

Design note: The granularity of the missing-data check is the chunk, not the byte. ensureRange only cares whether the chunks covering the interval have arrived; it does not guarantee that every byte in the interval came from the same response. That lowers the check cost and lets the scheduler freely merge adjacent chunks into large requests. The price is possibly downloading a bit beyond the chunk boundary, but chunk size is fixed, so the waste is bounded.

getByte, getBytes, getByteRange

The read interface is almost identical to Stream, plus the missing-data check:

  getByte() {
    const pos = this.pos;
    if (pos >= this.end) {
      return -1;
    }
    if (pos >= this.progressiveDataLength) {
      this.ensureByte(pos);
    }
    return this.bytes[this.pos++];
  }
 
  getBytes(length) {
    const pos = this.pos;
    const endPos = !length ? this.end : Math.min(pos + length, this.end);
 
    if (endPos > this.progressiveDataLength) {
      this.ensureRange(pos, endPos);
    }
    this.pos = endPos;
    return this.bytes.subarray(pos, endPos);
  }
 
  getByteRange(begin, end) {
    if (begin < 0) {
      begin = 0;
    }
    if (end > this.end) {
      end = this.end;
    }
    if (end > this.progressiveDataLength) {
      this.ensureRange(begin, end);
    }
    return this.bytes.subarray(begin, end);
  }

Three observations:

• getByte only calls ensureByte when the position is past the contiguous prefix. If the position is inside progressiveDataLength, that span is guaranteed to have arrived, so not even the chunk set needs to be inspected. This is the hottest path in sequential mode.
• getBytes clamps endPos to this.end before checking, so a request beyond the end of the file does not falsely report missing data; it simply returns a shorter span. subarray is a view, not a copy, so the caller gets the same memory inside the buffer.
• getByteRange is an "absolute-offset read" that does not move pos. It clamps the interval into the legal range, runs the same check, and returns a view. XRef reading xref sections and Parser reading object headers both rely on it.

Compared with Stream, the only read differences are the "past the contiguous prefix" test and one possible exception. The rest of the parser needs no changes at all, which is exactly the value of extending Stream.

Chunk State Queries

  getMissingChunks() {
    const chunks = [];
    for (let chunk = 0, n = this.numChunks; chunk < n; ++chunk) {
      if (!this._loadedChunks.has(chunk)) {
        chunks.push(chunk);
      }
    }
    return chunks;
  }
 
  get numChunksLoaded() {
    return this._loadedChunks.size;
  }
 
  get isDataLoaded() {
    return this.numChunksLoaded === this.numChunks;
  }

getMissingChunks is a linear scan and is only called for "prefetch the whole file" or for debugging; everyday reads go through the O(1) hasChunk. numChunksLoaded returns the Set size directly, and isDataLoaded compares it with the total chunk count.

nextEmptyChunk scans circularly starting from a given chunk: it returns the first unloaded chunk, returns null once everything is loaded, and wraps around to the beginning after reaching the end to catch leftovers. hasChunk is just a set lookup. Together they form the manager's query interface.

A chunk has only two states: present in the set means loaded and never unloaded; absent means not yet arrived. The "in flight" state is tracked separately by the manager's _requestsByChunk.

makeSubStream: A Substream Without Copying Memory

makeSubStream is the cleverest piece in the whole file. PDF stream objects (content streams, font files, image data) are almost always "a span of bytes in the file with a Length", and the parser needs to open another stream on top of that span. Stream creates a new Stream sharing the ArrayBuffer; ChunkedStream does not even want to create another object, so it makes the substream inherit from the parent object directly.

  makeSubStream(start, length, dict = null) {
    if (length) {
      if (start + length > this.progressiveDataLength) {
        this.ensureRange(start, start + length);
      }
    } else if (start >= this.progressiveDataLength) {
      // When the `length` is undefined you do *not*, under any circumstances,
      // want to fallback on calling `this.ensureRange(start, this.end)` since
      // that would force the *entire* PDF file to be loaded, thus completely
      // breaking the whole purpose of using streaming and/or range requests.
      //
      // However, not doing any checking here could very easily lead to wasted
      // time/resources during e.g. parsing, since `MissingDataException`s will
      // require data to be re-parsed, which we attempt to minimize by at least
      // checking that the *beginning* of the data is available here.
      this.ensureByte(start);
    }
 
    function ChunkedStreamSubstream() {}
    ChunkedStreamSubstream.prototype = Object.create(this);
    ChunkedStreamSubstream.prototype.getMissingChunks = function () {
      const chunkSize = this.chunkSize;
      const beginChunk = Math.floor(this.start / chunkSize);
      const endChunk = Math.floor((this.end - 1) / chunkSize) + 1;
      const missingChunks = [];
      for (let chunk = beginChunk; chunk < endChunk; ++chunk) {
        if (!this._loadedChunks.has(chunk)) {
          missingChunks.push(chunk);
        }
      }
      return missingChunks;
    };
    Object.defineProperty(ChunkedStreamSubstream.prototype, "isDataLoaded", {
      get() {
        return (
          this.numChunksLoaded === this.numChunks ||
          this.getMissingChunks().length === 0
        );
      },
      configurable: true,
    });
 
    const subStream = new ChunkedStreamSubstream();
    subStream.pos = subStream.start = start;
    subStream.end = start + length || this.end;
    subStream.dict = dict;
    return subStream;
  }

Start with the two branches in the first half:

• With a length: the substream needs all bytes in [start, start + length), so when that goes past the sequential prefix, ensureRange is called and any missing chunk throws. Ordinary stream objects have a known /Length, so this is the common case.
• Without a length: only ensureByte(start) is called, never ensureRange(start, this.end). The source comment is emphatic about why: that would force the entire PDF file to load and completely defeat the purpose of streaming and Range requests. Doing no check at all is also wrong, because parsing would then fail on a later byte after all the earlier work was wasted.

The prototype trick in the second half has three parts:

1. ChunkedStreamSubstream.prototype = Object.create(this) makes the parent instance itself the prototype of the substream. The substream has no bytes, chunkSize or _loadedChunks of its own; everything resolves up the prototype chain to the parent object.
2. getMissingChunks is overridden to scan only the substream's own chunk window [start, end). Keeping the parent implementation would return the chunks missing from the whole file, and the manager would download the entire file.
3. isDataLoaded is redefined so it reports either "the parent stream is fully loaded" or "nothing is missing inside the substream window". The BaseStream version is cached through shadow, so reusing it would give the wrong answer; configurable: true keeps the ability to override it again. After instantiation, only four own fields are set: pos, start, end and dict. The NaN fallback for end handles the case where length is undefined.

The benefit is zero memory copying, and chunks that arrive at the parent later are automatically visible to the substream, because _loadedChunks is the same Set. The cost is that the substream strongly references the parent, but in PDF.js a substream is always a temporary view of its parent.

getBaseStreams returns itself, which is used to trace the original stream of data such as images; substreams inherit the method through the prototype chain, so they naturally return themselves.

Design note: A substream is not "a copy of the parent's state" but "the parent instance acting as its own prototype". This requires every base-class read method to access state only through this, never through own fields of the instance. getByte, getBytes and ensureRange all meet that requirement, so the prototype chain can be reused directly. Conversely, any method that needs "substream window" semantics (getMissingChunks, isDataLoaded) must be overridden explicitly rather than inherited.

ChunkedStreamManager: Request Scheduling

ChunkedStream only detects missing chunks; ChunkedStreamManager is what actually brings them back. Its core work is three things: merge requests, backfill data, and prefetch automatically.

Construction and Dependencies

class ChunkedStreamManager {
  #aborted = false;
 
  currRequestId = 0;
 
  _chunksNeededByRequest = new Map();
 
  #loadedStreamCapability = Promise.withResolvers();
 
  _promisesByRequest = new Map();
 
  _requestsByChunk = new Map();
 
  constructor(pdfStream, args) {
    this.length = args.length;
    this.chunkSize = args.rangeChunkSize;
    this.stream = new ChunkedStream(this.length, this.chunkSize, this);
    this.pdfStream = pdfStream;
    this.disableAutoFetch = args.disableAutoFetch;
    this.msgHandler = args.msgHandler;
  }

The three maps answer three questions:

length usually comes from HTTP Content-Length, and chunkSize comes from a caller parameter, defaulting to 2 to the 16th power (65536 bytes; see the rangeChunkSize value in api.js). pdfStream is the data source implementing getRangeReader, disableAutoFetch controls whether prefetching continues while idle, and msgHandler sends DocProgress to the main thread.

The constructor sends no request and reads no bytes. The manager and the stream it manages are created together, then PDFDocument starts parsing, and only then do page-fault requests begin to appear.

sendRequest: From the Range Reader to the Buffer

  async sendRequest(begin, end) {
    const rangeReader = this.pdfStream.getRangeReader(begin, end);
    let chunks = [];
 
    while (true) {
      const { value, done } = await rangeReader.read();
 
      if (this.#aborted) {
        chunks = null;
        return; // Ignoring any data after abort.
      }
      if (done) {
        break;
      }
      if (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) {
        assert(
          value instanceof ArrayBuffer,
          "sendRequest - expected an ArrayBuffer."
        );
      }
      chunks.push(value);
    }
 
    if (chunks.length === 0 && this.disableAutoFetch) {
      // The range request wasn't dispatched, see the "GetRangeReader" handler
      // in the `src/display/api.js` file.
      return;
    }
    const data = arrayBuffersToBytes(chunks);
    chunks = null;
    this.onReceiveData({ chunk: data.buffer, begin });
  }

sendRequest is an asynchronous read loop: getRangeReader(begin, end) returns a streaming reader, each read() yields one ArrayBuffer slice, #aborted is checked after every slice and any data arriving after an abort is discarded, all slices are merged by arrayBuffersToBytes into one contiguous buffer with as few copies as possible, and finally onReceiveData writes the data into the ChunkedStream at offset begin.

chunks.length === 0 && this.disableAutoFetch covers the case where the request was never actually dispatched: the main thread's GetRangeReader handler sees that the data has already been read through the whole-file stream but has not yet reached the worker, so it closes the sink without issuing a Range request. An empty response is then not an error but a signal that "this will arrive later through another path", and returning is correct.

Design note: sendRequest is not responsible for knowing which logical request a network request belongs to. It only moves a byte range back into the buffer; who is waiting for those chunks is looked up by onReceiveData through _requestsByChunk. This lets several logical requests share one network request without any special-casing in the transport layer.

requestRange and requestRanges

  requestRange(begin, end) {
    end = Math.min(end, this.length);
 
    const beginChunk = this.getBeginChunk(begin);
    const endChunk = this.getEndChunk(end);
 
    const chunks = [];
    for (let chunk = beginChunk; chunk < endChunk; ++chunk) {
      chunks.push(chunk);
    }
    return this._requestChunks(chunks);
  }
 
  requestRanges(ranges = []) {
    const chunksToRequest = [];
    for (const range of ranges) {
      const beginChunk = this.getBeginChunk(range.begin);
      const endChunk = this.getEndChunk(range.end);
      for (let chunk = beginChunk; chunk < endChunk; ++chunk) {
        if (!chunksToRequest.includes(chunk)) {
          chunksToRequest.push(chunk);
        }
      }
    }
 
    chunksToRequest.sort((a, b) => a - b);
    return this._requestChunks(chunksToRequest);
  }

Byte ranges are converted to chunk ranges by two small helpers: getBeginChunk returns Math.floor(begin / chunkSize), and getEndChunk returns Math.floor((end - 1) / chunkSize) + 1. The end - 1 is the key detail: the interval is half-open, so the end position itself is not part of the request. Writing Math.floor(end / chunkSize) instead would request one extra chunk whenever end falls exactly on a chunk boundary.

requestRanges fills several non-contiguous ranges at once (for example the xref section and one page's content together). It first collects every chunk, deduplicates, sorts, and then hands the result to _requestChunks. Sorting is the precondition for groupChunks to work.

requestAllChunks is more direct: it passes getMissingChunks to _requestChunks and returns #loadedStreamCapability.promise. It requests every currently missing chunk at once and returns a promise for "the whole stream is loaded"; when noFetch is true it only waits without requesting, for the case where "data is already arriving through another path".

_requestChunks: One Request per Chunk

  _requestChunks(chunks) {
    const requestId = this.currRequestId++;
 
    const chunksNeeded = new Set();
    this._chunksNeededByRequest.set(requestId, chunksNeeded);
    for (const chunk of chunks) {
      if (!this.stream.hasChunk(chunk)) {
        chunksNeeded.add(chunk);
      }
    }
 
    if (chunksNeeded.size === 0) {
      return Promise.resolve();
    }
 
    const capability = Promise.withResolvers();
    this._promisesByRequest.set(requestId, capability);
 
    const chunksToRequest = [];
    for (const chunk of chunksNeeded) {
      const requestIds = this._requestsByChunk.getOrInsertComputed(
        chunk,
        () => {
          chunksToRequest.push(chunk);
          return [];
        }
      );
      requestIds.push(requestId);
    }
 
    if (chunksToRequest.length > 0) {
      const groupedChunksToRequest = this.groupChunks(chunksToRequest);
      for (const groupedChunk of groupedChunksToRequest) {
        const begin = groupedChunk.beginChunk * this.chunkSize;
        const end = Math.min(
          groupedChunk.endChunk * this.chunkSize,
          this.length
        );
        this.sendRequest(begin, end).catch(capability.reject);
      }
    }
 
    return capability.promise.catch(reason => {
      if (this.#aborted) {
        return; // Ignoring any pending requests after abort.
      }
      throw reason;
    });
  }

This is the heart of request merging, and it deserves a step-by-step breakdown:

1. The request id increments. Every logical request gets a monotonically increasing id, used to associate requests and chunks many-to-many.
2. Loaded chunks are filtered out first. The interval passed by the caller may include chunks that have already arrived (for example, a MissingDataException carrying a whole interval), so they are removed here to avoid downloading them again.
3. If everything is loaded, complete immediately. Return an already-resolved promise without occupying the maps.
4. Register the demand. _promisesByRequest stores the deferred, and _chunksNeededByRequest stores which chunks this request still needs.
5. Register in reverse, per chunk. getOrInsertComputed means "get the value for the key, or compute and insert it". When a chunk is requested for the first time, the callback pushes it into chunksToRequest and creates an empty request-id array; either way, the current request id is appended afterwards.
6. Only new chunks are requested. chunksToRequest contains only chunks seen for the first time; chunks already in flight are not requested again, and later requests merely attach to the same chunk's waiter array.
7. Send by contiguous range. groupChunks merges chunks into segments, each segment's endpoints are converted to byte offsets, and sendRequest is called. end is clamped to the file length so the last chunk cannot overflow.
8. Failures propagate. A failing sendRequest rejects this request's deferred; the returned promise swallows the error when #aborted is true, so the layer above does not see cancellation exceptions.

The difference before and after merging:

Design note: Merging happens at the chunk level, not the byte-interval level. Chunks have a fixed size, so merging only needs to know whether chunk numbers are consecutive, without computing overlaps between arbitrary intervals. That lets groupChunks finish in a single linear pass and keeps _requestsByChunk keys as simple integers.

groupChunks: Merging Scattered Chunks into Ranges

  groupChunks(chunks) {
    const groupedChunks = [];
    let beginChunk = -1;
    let prevChunk = -1;
 
    for (let i = 0, ii = chunks.length; i < ii; ++i) {
      const chunk = chunks[i];
      if (beginChunk < 0) {
        beginChunk = chunk;
      }
 
      if (prevChunk >= 0 && prevChunk + 1 !== chunk) {
        groupedChunks.push({ beginChunk, endChunk: prevChunk + 1 });
        beginChunk = chunk;
      }
      if (i + 1 === chunks.length) {
        groupedChunks.push({ beginChunk, endChunk: chunk + 1 });
      }
 
      prevChunk = chunk;
    }
    return groupedChunks;
  }

The algorithm is a single pass that maintains the first chunk of the current contiguous run, beginChunk, and the previous chunk, prevChunk. On the first chunk it records the run's start; when the current chunk is not consecutive with the previous one, it emits a [beginChunk, prevChunk + 1) range and starts a new run at the current chunk; on the last chunk it closes the final run.

The source comment explicitly requires the input to be sorted. That is why requestRanges collects, deduplicates and then sorts; requestRange builds the array in increasing chunk order; and requestAllChunks gets its chunk numbers from a sequential scan. Inside _requestChunks, chunksToRequest comes from the chunksNeeded Set, whose insertion order is the order the caller passed chunks in. As long as the upstream is ordered, groupChunks produces as few ranges as possible. An unsorted array does not make the ranges overflow, but it shatters mergeable chunks into many fragments and inflates the request count, which is why sorting must happen before merging.

Ranges express their upper bound with endChunk as an open interval, so both prevChunk + 1 and chunk + 1 include the last chunk. For example, chunk numbers [3, 4, 5, 9] merge into two ranges: [3, 6) and [9, 10). Four chunks that would have taken four requests now take two.

onReceiveData: Settling Requests and Auto-Prefetch

  onReceiveData(args) {
    const { chunkSize, length, stream } = this;
 
    const chunk = args.chunk;
    const isProgressive = args.begin === undefined;
    const begin = isProgressive ? stream.progressiveDataLength : args.begin;
    const end = begin + chunk.byteLength;
 
    const beginChunk = Math.floor(begin / chunkSize);
    const endChunk =
      end < length ? Math.floor(end / chunkSize) : Math.ceil(end / chunkSize);
 
    if (isProgressive) {
      stream.onReceiveProgressiveData(chunk);
    } else {
      stream.onReceiveData(begin, chunk);
    }
 
    if (stream.isDataLoaded) {
      this.#loadedStreamCapability.resolve(stream);
    }
 
    const loadedRequests = [];
    for (let curChunk = beginChunk; curChunk < endChunk; ++curChunk) {
      // The server might return more chunks than requested.
      const requestIds = this._requestsByChunk.get(curChunk);
      if (!requestIds) {
        continue;
      }
      this._requestsByChunk.delete(curChunk);
 
      for (const requestId of requestIds) {
        const chunksNeeded = this._chunksNeededByRequest.get(requestId);
        if (chunksNeeded.has(curChunk)) {
          chunksNeeded.delete(curChunk);
        }
 
        if (chunksNeeded.size > 0) {
          continue;
        }
        loadedRequests.push(requestId);
      }
    }
 
    // If there are no pending requests, automatically fetch the next
    // unfetched chunk of the PDF file.
    if (!this.disableAutoFetch && this._requestsByChunk.size === 0) {
      let nextEmptyChunk;
      if (stream.numChunksLoaded === 1) {
        // This is a special optimization so that after fetching the first
        // chunk, rather than fetching the second chunk, we fetch the last
        // chunk.
        const lastChunk = stream.numChunks - 1;
        if (!stream.hasChunk(lastChunk)) {
          nextEmptyChunk = lastChunk;
        }
      } else {
        nextEmptyChunk = stream.nextEmptyChunk(endChunk);
      }
      if (Number.isInteger(nextEmptyChunk)) {
        this._requestChunks([nextEmptyChunk]);
      }
    }
 
    for (const requestId of loadedRequests) {
      const capability = this._promisesByRequest.get(requestId);
      this._promisesByRequest.delete(requestId);
      capability.resolve();
    }
 
    this.msgHandler.send("DocProgress", {
      loaded: MathClamp(
        stream.numChunksLoaded * chunkSize,
        stream.progressiveDataLength,
        length
      ),
      total: length,
    });
  }

This method combines "receive data" and "send notifications" into five sequential steps:

Step one: detect the mode and write. args.begin being undefined means a sequential stream, with the start at the current progressiveDataLength; otherwise it is a Range response. The write is delegated to one of the two ChunkedStream methods.

Step two: compute the chunk window. When end is not at the end of the file, use Math.floor(end / chunkSize); at the end of the file, use Math.ceil(end / chunkSize), so a short final chunk is still counted.

Step three: fulfil the completion promise. #loadedStreamCapability resolves only once, when every chunk has arrived.

Step four: settle the logical requests. Iterate over every chunk covered by the response: take the request ids waiting on it from _requestsByChunk, and skip if there are none; delete the chunk mapping immediately so no later request is sent for it; for each waiter, remove the chunk from chunksNeeded, and once the set is empty, add the request id to loadedRequests. The server may return more chunks than requested, so the loop covers every chunk the response actually contains, and the extras are registered too.

Step five: auto-prefetch. Prefetch only happens when _requestsByChunk is empty, so it never competes with an explicit user request for the connection. The policy is in the table below.

"Fetch the first chunk, then jump straight to the last" is the most notable optimization in the whole file. A PDF's cross-reference table lives at the end of the file; reading sequentially from the second chunk would require the entire file before object lookup could even start. Jumping to the tail chunk makes startxref and the trailer parseable immediately, and reading the few chunks that the xref points at is then enough to start rendering the first page. The many chunks in between are filled in gradually by later sequential prefetches and do not affect first paint.

Completed requests resolve only after step five, and the progress message is sent after that. The order matters: fulfilling a request means the caller may immediately continue parsing and produce new page faults. Resolving earlier would let those new requests interrupt the prefetch decision; doing it afterwards makes the most of the single window where "nothing is in flight".

The loaded value in the progress message is clamped three ways by MathClamp: the lower bound is progressiveDataLength, because in sequential mode the chunk markers may lag behind the actual bytes and progress must never go backwards; the upper bound is length, because a short final chunk makes chunks times chunk size exceed the file length and progress must never report more than 100%; the middle value is numChunksLoaded * chunkSize. The main thread updates the progress bar when it receives DocProgress; the worker only has to report the state accurately.

Design note: Prefetching is opportunistic: it grabs one extra chunk only when nothing is in flight, and it prefers the tail chunk, which carries the highest information density. It keeps no separate prefetch queue and sets no concurrency limit, because the cases that genuinely need concurrency are already covered by _requestChunks merging. Replacing a "prefetch scheduler" with "prefetch when idle" is the key to keeping this implementation simple.

abort: Waking Every Waiter

  abort(reason) {
    this.#aborted = true;
    this.pdfStream?.cancelAllRequests(reason);
 
    for (const capability of this._promisesByRequest.values()) {
      capability.reject(reason);
    }
    this.#loadedStreamCapability.reject(reason);
  }

abort does four things: set the termination flag, cancel every underlying network request, reject every per-request deferred, and reject the whole-stream load promise.

The flag has two effects: the read loop in sendRequest sees it and discards later slices, and the promise returned by _requestChunks returns silently on rejection once termination has happened. The first stops cancelled requests from writing into the buffer; the second keeps callers from seeing a "failed to fetch data" error after termination.

abort itself does not clear the maps, because termination means the whole manager is no longer used; keeping the references only delays garbage collection a little and causes no logical error.

Full Timeline of a Page Load

Stringing all the pieces together, opening the first page of a PDF over the network looks roughly like this:

1. NetworkPdfManager constructs a ChunkedStreamManager, which creates the ChunkedStream and the PDFDocument together. The buffer is already allocated, and no chunk is loaded yet.
2. The parser starts reading at the file header, lands in chunk 0, ensureByte throws, and NetworkPdfManager.ensure catches it and calls requestRange.
3. _requestChunks registers chunk 0 as a request, groupChunks produces a single-chunk range, and sendRequest issues the first Range request.
4. The response arrives; onReceiveData writes chunk 0, settles the request, then notices that only one chunk is loaded and triggers the "fetch the tail after the first chunk" optimization, immediately requesting the last chunk.
5. The waiters on chunk 0 wake up and parsing continues through the header and the first object, while the tail request is in flight.
6. The tail chunk arrives, PDFDocument finds startxref from the end of the file, XRef.parse starts reading the xref section, produces new page faults, and re-enters _requestChunks.
7. Those page faults may cover several adjacent chunks and get merged into one or two requests; non-contiguous ranges each become their own segment.
8. Every response also prefetches one empty chunk, until all objects needed by the page are ready and PartialEvaluator produces the operatorList.
9. If the user scrolls to later pages, page faults keep happening on demand; if disableAutoFetch is false, chunks are gradually filled in while idle, and progress is reported through DocProgress.

This timeline explains a common observation: in the network panel, PDF requests are not strictly sequential. The last chunk often follows chunk 0 immediately, and the middle only comes afterwards. That is not random; it is the direct result of the tail-first policy.

Working with MissingDataException

ChunkedStream does not wait or retry by itself; it only throws. The retry logic lives in NetworkPdfManager.ensure:

  async ensure(obj, prop, args) {
    try {
      const value = obj[prop];
      if (typeof value === "function") {
        return await value.apply(obj, args);
      }
      return value;
    } catch (ex) {
      if (!(ex instanceof MissingDataException)) {
        throw ex;
      }
      await this.requestRange(ex.begin, ex.end);
      return this.ensure(obj, prop, args);
    }
  }

This is "retry at the property-access level": first read the property or call the method normally; only MissingDataException is caught, and every other exception is rethrown unchanged; request the interval carried by the exception and wait for it to complete; finally perform the same property access again from the beginning.

Retrying bluntly from the start is safe because all parsing reads are idempotent: calling the getter again re-runs Lexer and Parser, and the part that was not finished before produced no side effects anyway. Anything that genuinely needs committing (objects, pages, operatorLists) is written to cache only once the data is complete.

This also explains why ensureRange throws the full interval rather than the first missing chunk: only by filling every byte the caller needs does the retry make sense. Filling just one chunk would very likely produce another exception at the next chunk, inflating both recursion depth and network round trips.

Design note: Missing data is not an error; it is control flow. ChunkedStream propagates "cannot read yet" upward as an exception, and NetworkPdfManager translates it into "wait a moment, then read again". The parser itself has no idea the network exists; it only sees the single action "read bytes". Streaming can therefore be added transparently underneath any parsing code.

Conclusion

chunked_stream.js is PDF.js's byte-supply layer: upward it disguises an on-demand file as a complete Uint8Array, and downward it translates the parser's random access into as few HTTP Range requests as possible.

Where It Sits in the Overall Chain

In one sentence: the network supplies slices, the manager turns slices into chunks, ChunkedStream turns chunks into an addressable file, and the parser completes the on-demand read of the whole document between page faults and retries.

Five Design Principles

1. Logical length and physical availability are separated. The buffer is allocated once for the file length at construction, and chunk state is recorded in two independent dimensions: _loadedChunks and progressiveDataLength. The parser can access any offset without caring in what order data arrives.
2. Missing data is expressed as an exception, not as a blocking wait. ensureByte and ensureRange only check; when a chunk is missing they throw MissingDataException. Retries and network waits are concentrated in NetworkPdfManager.ensure. Parsing code stays fully synchronous.
3. Requests are merged per chunk and deduplicated on demand. _requestChunks uses _requestsByChunk to record the waiters for each chunk, so the same chunk is requested only once; groupChunks merges consecutive chunks into as few ranges as possible. The number of requests is decided by "chunk distribution", not by "number of calls".
4. Prefetching is restrained, tail first and sequential after. One chunk is prefetched only when nothing is in flight; after the first data arrives, the last chunk is preferred so the trailer and startxref become available early. First paint never waits for the whole file, and the background fills in the rest sequentially.
5. Substreams share the parent state without copying memory. makeSubStream uses Object.create(this) to make the parent instance the prototype, overriding only the window-related getMissingChunks and isDataLoaded. That saves the buffer copy and makes chunks arriving at the parent later automatically visible to the substream.

The next article moves to the decoding side: decode_stream.js. The network layer hands over only raw bytes; Flate, LZW, ASCII85, predictors and image decoders all have to turn those bytes into truly readable buffers on top of this layer, and DecodeStream is their common foundation.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.