PDF.js Source Walkthrough: worker_stream

Introduction

This is the article in the PDF.js source walkthrough series about "byte transport". It covers PDFWorkerStream, PDFWorkerStreamReader and PDFWorkerStreamRangeReader in src/core/worker_stream.js (89 lines).

Earlier articles followed the parsing chain downward: Parser cuts bytes into tokens, XRef maps object numbers to file offsets, and ChunkedStream disguises network chunks as one complete file. All of those modules share one premise, though: the Worker already holds a readable BasePDFStream. This article supplies that premise. When a PDF comes from a URL, who exactly delivers the bytes into the Worker, and how.

PDF.js keeps network access on the main thread, the display layer, and puts parsing in the Worker. The Worker never calls fetch itself and never holds the main thread's network stream; the two sides move bytes only through the streaming message protocol of MessageHandler. worker_stream.js is the adapter on the Worker side: it implements the BasePDFStream contract, translates read() into cross-thread messages, and turns the Uint8Array received in those messages back into an ArrayBuffer.

In one sentence: PDFWorkerStream is the Worker's virtual byte source, with the display-layer network stream upstream and ChunkedStreamManager plus the parser downstream.

Version note: This article follows exactly the version of worker_stream.js in the current repository. It has no network request, cache or retry logic of its own and relies only on src/shared/base_pdf_stream.js and the sendWithStream and sendWithPromise methods of MessageHandler. The PDFNetworkStream naming common in older versions belongs to the display layer; do not confuse it with PDFWorkerStream here.

Module Overview

The file has only 89 lines and splits into four roles:

It defines no new data structures and stores no bytes. Its work boils down to four steps:

1. Ask the display layer for a full stream with the GetReader message;
2. Obtain contentLength, isStreamingSupported and isRangeSupported with the ReaderHeadersReady message;
3. Unwrap every Uint8Array received into an ArrayBuffer and hand it to the layer above;
4. Request additional bytes for an interval with the GetRangeReader message.

Where it sits in the whole chain:

Note the direction of this table: PDFWorkerStream lives in the Worker, but it is not the producer of the data. It is only a consumer that asks the main thread for bytes. The object that really issues the HTTP request is #networkStream in the display layer.

Why This Adapter Layer Exists

getPdfManager on the Worker has two paths. When the caller passes in a whole ArrayBuffer, it takes LocalPdfManager, and the bytes are already in memory. When the caller only supplies a URL, the Worker has no network capability and can only construct a PDFWorkerStream:

const pdfStream = new PDFWorkerStream({ msgHandler: handler }),
  fullReader = pdfStream.getFullReader();

The source here is neither a URL nor a byte array, but an object that carries only msgHandler. Everything the adapter needs comes from that message handler.

The Worker then waits for headersReady and only afterwards decides which manager to use:

fullReader.headersReady
  .then(() => {
    if (!fullReader.isRangeSupported) {
      return;
    }
    pdfManagerArgs.source = pdfStream;
    pdfManagerArgs.length = fullReader.contentLength;
    // We don't need auto-fetch when streaming is enabled.
    pdfManagerArgs.disableAutoFetch ||= fullReader.isStreamingSupported;
 
    newPdfManager = new NetworkPdfManager(pdfManagerArgs);
    // There may be a chance that `newPdfManager` is not initialized for
    // the first few iterations of the `readData` code. Be sure to send
    // all cached chunks, if any, to chunked_stream via pdf_manager.
    for (const chunk of cachedChunks) {
      newPdfManager.sendProgressiveData(chunk);
    }
    cachedChunks = null;
 
    resolve(newPdfManager);
    cancelXHRs = null;
  })
  .catch(reason => {
    reject(reason);
    cancelXHRs = null;
  });

This code explains the purpose of three fields:

If the server does not support ranges, the Worker keeps reading fullReader sequentially, caches the slices in cachedChunks, concatenates them into the complete bytes after the read finishes, and falls back to LocalPdfManager. The full reader therefore does not only serve a full download; it is also the probe that determines the transport capabilities.

Design note: The adapter splits "transport capability negotiation" and "reading bytes" into two messages. ReaderHeadersReady returns metadata only and carries no data chunks; GetReader is responsible for the data stream only and never judges capabilities. That lets the Worker decide between the streaming and whole-file path before any byte arrives, instead of guessing first and patching later.

The BasePDFStream Contract

PDFWorkerStream itself has only three effective lines:

class PDFWorkerStream extends BasePDFStream {
  constructor(source) {
    super(source, PDFWorkerStreamReader, PDFWorkerStreamRangeReader);
  }
}

The parent class saves source at construction time and records the two reader classes in private fields:

  constructor(source, PDFStreamReader, PDFStreamRangeReader) {
    if (
      (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) &&
      this.constructor === BasePDFStream
    ) {
      unreachable("Cannot initialize BasePDFStream.");
    }
    this._source = source;
 
    this.#PDFStreamReader = PDFStreamReader;
    this.#PDFStreamRangeReader = PDFStreamRangeReader;
  }

This inheritance relationship is why PDFWorkerStream does not need its own getFullReader() or getRangeReader(). The parent class already provides them:

  getFullReader() {
    assert(
      !this._fullReader,
      "BasePDFStream.getFullReader can only be called once."
    );
    return (this._fullReader = new this.#PDFStreamReader(this));
  }
 
  /**
   * Gets a reader for the range of the PDF data.
   *
   * NOTE: Currently this method is only expected to be invoked *after*
   * the `BasePDFStreamReader.prototype.headersReady` promise has resolved.
   *
   * @param {number} begin - the start offset of the data.
   * @param {number} end - the end offset of the data.
   * @returns {BasePDFStreamRangeReader}
   */
  getRangeReader(begin, end) {
    if (end <= this._progressiveDataLength) {
      return null;
    }
    const reader = new this.#PDFStreamRangeReader(this, begin, end);
    this._rangeReaders.add(reader);
    return reader;
  }

There are two contracts that are easy to miss:

• The full reader can be created only once; _fullReader is cached, and a repeated call trips an assertion.
• A range reader is created only when the requested interval extends beyond the data already received sequentially; otherwise the method returns null directly, because those bytes have already arrived through the full stream.

_progressiveDataLength comes from the parent: it reads _fullReader._loaded and returns 0 when there is no full reader. For the Worker this value means "how many bytes the sequential stream has delivered so far", and range requests use it to avoid duplicates.

  cancelAllRequests(reason) {
    this._fullReader?.cancel(reason);
 
    // Always create a copy of the rangeReaders.
    for (const reader of new Set(this._rangeReaders)) {
      reader.cancel(reason);
    }
  }

Cancellation notifies the full reader and every range reader at the same time. The Set is copied before iterating because cancel can modify the collection in turn, and iterating directly risks skipping entries.

Design note: BasePDFStream prescribes only "how to create a reader" and "how to cancel everything uniformly"; it does not prescribe where bytes come from. URLs, ranges, Node files and message channels can all fit behind the same reader interface, so PDFManager needs no branch for each transport.

PDFWorkerStreamReader

Constructor

The full reader's constructor does two things: it establishes the data stream, then waits for the headers separately.

class PDFWorkerStreamReader extends BasePDFStreamReader {
  _reader = null;
 
  constructor(stream) {
    super(stream);
    const { msgHandler } = stream._source;
 
    const readableStream = msgHandler.sendWithStream("GetReader");
    this._reader = readableStream.getReader();
 
    msgHandler.sendWithPromise("ReaderHeadersReady").then(data => {
      this._contentLength = data.contentLength;
      this._isStreamingSupported = data.isStreamingSupported;
      this._isRangeSupported = data.isRangeSupported;
 
      this._headersCapability.resolve();
    }, this._headersCapability.reject);
  }

sendWithStream("GetReader") returns a local ReadableStream. This object is not a network response; it is a stream that MessageHandler simulates through the message protocol: its start sends a request to the display layer, pull asks for the next chunk, and cancel sends a cancellation signal. getReader() merely turns that stream into a handle that can be read chunk by chunk.

The sendWithPromise("ReaderHeadersReady") right after it is a separate, independent request. The display layer waits for headersReady of the main-thread network stream and then returns three fields:

After writing the fields, the constructor calls this._headersCapability.resolve(). This capability is defined on the parent, BasePDFStreamReader, and is exposed as headersReady. So the headersReady the Worker sees is not produced by the Worker itself; it is the projection of the main thread's network response headers after one message round trip.

The failure branch passes this._headersCapability.reject directly as the second argument of then. There is no extra try or catch: an error from the message layer enters the capability unchanged, and the awaiting side receives the same reason.

Design note: Splitting the data stream and the headers into two messages means bytes can arrive before the headers or after them. read() only cares about the data stream and headersReady only cares about metadata, so neither blocks the other. For a streaming response this avoids the unnecessary serialization of "wait for the full response headers first, then start reading bytes that have already arrived".

read

  async read() {
    const { value, done } = await this._reader.read();
    if (done) {
      return { value: undefined, done: true };
    }
    // `value` is wrapped into Uint8Array, we need to
    // unwrap it to ArrayBuffer for further processing.
    return { value: value.buffer, done: false };
  }

read() follows the BasePDFStreamReader contract strictly: the end is done: true with value of undefined, and a normal chunk is done: false with value of ArrayBuffer.

Why the extra value.buffer? The display layer writes this when enqueuing:

sink.enqueue(new Uint8Array(value), 1, [value]);

The main-thread reader produces an ArrayBuffer, but the payload of a ReadableStream must be a serializable view, so it is wrapped in a Uint8Array here and the original ArrayBuffer is placed in the transfer list. On the Worker side the message layer hands back a Uint8Array, and the adapter takes out the ArrayBuffer behind it. The layer above, ChunkedStream, only understands ArrayBuffer, so this type restoration is necessary.

Note that value.buffer takes the whole underlying buffer. The display layer always builds the view over a complete ArrayBuffer, so there is no case where a view covers only part of the buffer. If this were ever changed to reuse a large buffer and slice it, the code here would have to check byteOffset and byteLength as well and must not blindly take .buffer.

cancel

  cancel(reason) {
    this._reader.cancel(reason);
  }

Cancellation just hands the reason to the local ReadableStream's cancel. The message layer turns it into a cancellation message to the display layer, which calls the main-thread reader's cancel and finally aborts the network request. The whole cancellation chain keeps no extra state on the Worker side and does not retry; the adapter only forwards.

reason must be an Error, an assertion required by the streaming protocol of MessageHandler. PDFWorkerStream.cancelAllRequests usually passes an AbortException, so it passes without issue.

PDFWorkerStreamRangeReader

Constructor

The range reader has a structure similar to the full reader, except for the message name and parameters, and it has no header phase.

class PDFWorkerStreamRangeReader extends BasePDFStreamRangeReader {
  _reader = null;
 
  constructor(stream, begin, end) {
    super(stream, begin, end);
    const { msgHandler } = stream._source;
 
    const readableStream = msgHandler.sendWithStream("GetRangeReader", {
      begin,
      end,
    });
    this._reader = readableStream.getReader();
  }

begin and end are sent to the display layer as message data. The handler there calls the main thread's #networkStream.getRangeReader(begin, end) and returns the ReadableStream for that interval. The Worker side knows nothing about HTTP range headers, chunk sizes or cache state; it only says "I want this slice".

The range reader does not wait for ReaderHeadersReady. It should be created only after headersReady has resolved, which is the precondition written explicitly in the comment on BasePDFStream.getRangeReader. ChunkedStreamManager issues chunk requests only after it has the file length and the range capability, so the ordering is guaranteed by the layer above.

read and cancel

  async read() {
    const { value, done } = await this._reader.read();
    if (done) {
      return { value: undefined, done: true };
    }
    return { value: value.buffer, done: false };
  }
 
  cancel(reason) {
    this._reader.cancel(reason);
  }

These two methods are identical to those of the full reader, down to the omitted comment. The difference is only in construction: the full reader uses GetReader and includes capability negotiation, while the range reader uses GetRangeReader and carries only an interval.

The display layer has one special branch: if the requested interval is already covered by the full stream, getRangeReader returns null and the handler calls sink.close() directly. The range reader on the Worker side then reads done: true and receives zero chunks. ChunkedStreamManager.sendRequest has a dedicated check for the "zero chunks with auto-fetch disabled" case, so a request that was never actually issued is not mistaken for a response.

Design note: The full reader and the range reader share the same message-stream mechanism, but their responsibilities are strictly separated. The full reader owns "the sequential view of the whole file" and "transport capability probing"; the range reader owns "filling the gaps of random access". Merging the two into one interface with an offset parameter would entangle headersReady with the timing of range requests and make correctness harder to guarantee.

The Other End of the Message Channel

The GetReader on the Worker side is only a request; the thing that actually answers it is WorkerTransport.setupMessageHandler in the display layer:

    messageHandler.on("GetReader", (data, sink) => {
      assert(
        this.#networkStream,
        "GetReader - no `BasePDFStream` instance available."
      );
      this.#fullReader = this.#networkStream.getFullReader();
      // If stream or range turn out to be disabled, once `headersReady` is
      // resolved, this is our only way to report loading progress.
      this.#fullReader.onProgress = evt => this.#onProgress(evt);
 
      sink.onPull = () => {
        this.#fullReader
          .read()
          .then(function ({ value, done }) {
            if (done) {
              sink.close();
              return;
            }
            assert(
              value instanceof ArrayBuffer,
              "GetReader - expected an ArrayBuffer."
            );
            // Enqueue data chunk into sink, and transfer it
            // to other side as `Transferable` object.
            sink.enqueue(new Uint8Array(value), 1, [value]);
          })
          .catch(reason => {
            sink.error(reason);
          });
      };

#networkStream is a PDFDataTransportStream or a NetworkStream, the object on the main thread that really opens the URL, processes response headers and issues range requests. Every read() on the Worker side eventually triggers sink.onPull here, and the main-thread reader then reads one network chunk.

The headers handler waits on the main thread for the network stream's headersReady:

messageHandler.on('ReaderHeadersReady', async data => {
  await this.#fullReader.headersReady;
 
  const { isStreamingSupported, isRangeSupported, contentLength } = this.#fullReader;
 
  if (isStreamingSupported && isRangeSupported) {
    this.#fullReader.onProgress = null; // See comment in "GetReader" above.
  }
  return { isStreamingSupported, isRangeSupported, contentLength };
});

The object it returns matches exactly the three fields written on the Worker side. contentLength may be unknown at first in a streaming response, but headersReady guarantees that it has been determined; if the main thread cannot obtain it, the promise rejects and the Worker's headersReady rejects with it.

The range handler is shorter:

    messageHandler.on("GetRangeReader", (data, sink) => {
      assert(
        this.#networkStream,
        "GetRangeReader - no `BasePDFStream` instance available."
      );
      const rangeReader = this.#networkStream.getRangeReader(
        data.begin,
        data.end
      );

Only with both pieces together is the full shape of the "channel" visible:

Design note: sendWithStream does not postMessage the whole response body at once; it maps the pull and cancel of a ReadableStream onto cross-thread messages. Data chunks move only when requested, and changes in desiredSize produce backpressure. If the Worker reads slowly, the main thread does not enqueue without bound, and memory use stays inside a predictable window.

Failure, Cancellation and Lifetime

The adapter has no retry logic of its own; all error handling relies on two mechanisms:

1. Promise and stream errors: a failing ReaderHeadersReady rejects headersReady; a failing read() puts the stream into an error state, and the Worker's read() call receives a rejection.
2. Cancellation propagation: the Worker's cancel enters the message layer, the display's sink.onCancel calls the main-thread reader's cancel, and the network request is aborted.

PDFWorkerStream's lifetime follows the document. WorkerTransport.destroy calls #networkStream.cancelAllRequests, so the main thread cancels all network readers first; once the Worker is terminated, the message port closes and unfinished streams end in an error. The adapter needs no separate destroy method because it holds no resources of its own.

There is also an easily missed edge case: when disableAutoFetch is true and a range request was never actually issued, ChunkedStreamManager.sendRequest receives zero chunks. That logic is not in worker_stream.js, but it is directly related to the "empty result" of the range reader. Reading only read() in this file makes an empty stream look strange; connecting the three pieces — getRangeReader returning null, the display closing sink, and sendRequest returning early — is what explains it.

Conclusion

PDFWorkerStream is PDF.js's cross-thread byte adapter layer: it translates the Worker's reads of BasePDFStream into network reads in the display layer, then wraps the ArrayBuffer produced by the network layer in the reader interface the Worker's parser expects.

Where It Sits in the Chain

In one sentence: the display layer turns network responses into bytes, PDFWorkerStream turns bytes into a stream the Worker can read, and ChunkedStream disguises that stream as a complete file.

Four Design Principles

1. Transport and parsing are layered. Network requests, range negotiation and response-header parsing stay in the display layer; the Worker only implements the BasePDFStream contract and is unaware of HTTP.
2. Metadata and data are separated. ReaderHeadersReady returns the length and capabilities on its own, while GetReader carries bytes on its own; neither waits for the other.
3. Pull on demand instead of a push flood. sendWithStream uses pull and desiredSize to form backpressure: the main thread delivers only what the Worker has read, so the message queue cannot grow without bound.
4. Cancellation travels down the chain unchanged. The Worker's cancel goes through the message layer, sink.onCancel and the main-thread reader all the way down, and the network request is finally aborted; the adapter neither swallows the reason nor fakes success.

worker_stream.js is short, but it joins two completely different worlds, browser networking and Worker parsing. Once this adapter is understood, the missing-page retry in ChunkedStream no longer looks like the Worker issuing requests itself; it is only asking the main thread for the next chunk.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.