PDF.js Source Walkthrough: object_loader

Introduction

src/core/object_loader.js is only 153 lines long, yet it is one of the keys to whether PDF.js works smoothly in a streaming scenario. It neither parses nor renders; it does one thing: before a resource is actually needed, walk the object graph and ask for every possibly missing byte range in one go.

The root of the problem is the worker's synchronous parsing model. XRef.fetch is synchronous and Dict.get is synchronous; as soon as a chunk that has not been downloaded is read, ChunkedStream throws MissingDataException, carrying the missing byte range. If every call site caught, requested and retried on its own, the parsing code would drown in retry logic; and because the fonts, images and color spaces of one page usually fall into adjacent ranges, per-object requests would produce a large number of small network round trips.

ObjectLoader centralizes that: starting from a handful of keys in the page resource dictionary, it walks the reachable object graph depth first, leaves successfully fetched objects in the XRef cache, and collects the missing ranges into one array that it hands to ChunkedStreamManager in a single call. When the data arrives, the walk resumes from the places that did not get through.

Module Overview

The file has only two module functions, one class and three methods:

The class has one class field, refSet, and a three-argument constructor, and no state that survives between calls. Each ObjectLoader.load(obj, keys, xref) creates a fresh instance that is thrown away afterwards.

Timeline of a Single Page Load

Taking Page.getOperatorList as an example, resource prefetching does not happen in isolation; it runs in parallel with fetching the content stream:

const contentStreamPromise = this.getContentStream();
const resourcesPromise = this.loadResources(RESOURCES_KEYS_OPERATOR_LIST);

Inside loadResources the resource dictionary the page inherits is resolved first, then handed to ObjectLoader:

await (this.#resourcesPromise ??= this.pdfManager.ensure(this, 'resources'));
 
await ObjectLoader.load(this.resources, keys, this.xref);

The full sequence has seven steps:

1. getOperatorList starts two promises at once: fetch the content stream, prefetch resources.
2. loadResources first ensures resources; this step may have the manager fill in missing bytes.
3. ObjectLoader reads the raw values of the seven resource keys into the initial stack.
4. #walk expands references, dictionaries, streams and arrays depth first.
5. The missing ranges are submitted to ChunkedStreamManager in one batch, which waits for the chunks.
6. When the data arrives, the positions that failed in the previous round are walked again until the stack is empty.
7. Both promises resolve, and PartialEvaluator starts interpreting the content stream synchronously.

The boundary between step 6 and step 7 matters: once the interpreter starts running, it no longer tolerates missing bytes. Prefetching puts all network waiting before that line, which is why ObjectLoader deserves its own file.

Why Prefetching Is Needed

Parsing on the worker side is a synchronous chain: Dict.get calls XRef.fetch when it resolves a reference, and XRef.fetch reads the stream directly. The chunked stream throws MissingDataException when a read crosses a boundary, and the exception object carries two byte offsets, begin and end.

ensure retries per property access, which suits the scattered reads of document initialization. Rendering a page is different: one getOperatorList touches dozens of fonts, images and color spaces, and if each of them triggered an ensure retry there would be dozens of serial round trips. The value of ObjectLoader is replacing per-object retries with one walk and one batched request.

Its walk only covers the keys the caller names; it never climbs the page tree up to the whole document. The class comment warns about exactly that: do not start loading from the Catalog or from other pages, or the walk would traverse the whole PDF object graph and streaming would lose its meaning.

Design note: The boundary of prefetching is defined by the caller through keys. ObjectLoader is only responsible for finishing this subgraph, not for deciding which resources deserve to be loaded early. That division lets one traversal serve rendering, text extraction and XFA alike, without a separate loading strategy for each use.

Deciding What Can Be Traversed

In the PDF object model only four things can lead further down: references, dictionaries, streams and arrays. Everything else is a leaf.

mayHaveChildren

function mayHaveChildren(value) {
  return (
    value instanceof Ref ||
    value instanceof Dict ||
    value instanceof BaseStream ||
    Array.isArray(value)
  );
}

The test is deliberately conservative: any of the four types returns true, regardless of whether the node currently has children. Ref is included because the type it points to is unknown until it has been fetched.

addChildren

function addChildren(node, nodesToVisit) {
  if (node instanceof Dict) {
    node = node.getRawValues();
  } else if (node instanceof BaseStream) {
    node = node.dict.getRawValues();
  } else if (!Array.isArray(node)) {
    return;
  }
  for (const rawValue of node) {
    if (mayHaveChildren(rawValue)) {
      nodesToVisit.push(rawValue);
    }
  }
}

Three choices here are deliberate:

1. Use getRawValues, not get. Dict.get resolves Refs into objects, and resolution can throw MissingDataException; once an exception escapes mid-walk, the stack state is hard to recover. Reading raw values touches only the dictionary's internal value table and triggers no resolution.
2. Streams expand only their dictionary. Stream contents can be hundreds of KB or several MB, and the walk does not need them; whether the data is complete is checked later by getBaseStreams.
3. Only potentially expandable children are pushed. Strings, numbers and names are filtered out, so the stack holds only nodes that really need visiting and pointless pop calls are avoided.

getRawValues returns an iterator over the dictionary's internal value table in insertion order, which has no effect on the result. Arrays are traversed by index, and nested arrays become independent nodes that are expanded in turn.

A dictionary cannot directly contain itself in PDF syntax: cycles are only possible through indirect references. The walk therefore only needs to guard against Ref cycles, and does not need object-identity deduplication for Dict. A shared dictionary may be visited several times, but the number of visits is bounded, so the walk cannot loop forever.

The ObjectLoader Entry Points

load

  async load() {
    const { keys, dict } = this;
    // Setup the initial nodes to visit.
    const nodesToVisit = [];
    for (const key of keys) {
      const rawValue = dict.getRaw(key);
      // Skip nodes that are guaranteed to be empty.
      if (rawValue !== undefined) {
        nodesToVisit.push(rawValue);
      }
    }
    await this.#walk(nodesToVisit);
 
    this.refSet = null; // Everything is loaded, clear the cache.
  }

The starting point is not the whole dictionary but the key list the caller supplies. For rendering a page that is ColorSpace, ExtGState, Font, Pattern, Properties, Shading and XObject; text extraction needs only four of them.

getRaw does not resolve references, so this step can never throw for missing bytes. Keys whose value is undefined are skipped, which covers the common case of a resource category that simply does not exist.

After the walk refSet is set to null, and the comment says it plainly: everything is loaded, so the cache can be dropped. RefSet may hold tens of thousands of entries in a large document, while the results of this walk have already settled into the XRef object cache, so the visited set does not need to be kept. Note that this only runs on the success path; if #walk throws anything other than a missing-data exception, refSet is collected along with the instance.

The static load method

  static async load(obj, keys, xref) {
    // Don't walk the graph if all the data is already loaded.
    if (xref.stream.isDataLoaded) {
      return;
    }
    // eslint-disable-next-line no-restricted-syntax
    const objLoader = new ObjectLoader(obj, keys, xref);
    await objLoader.load();
  }

The static entry point does two jobs: factory and short circuit.

The short-circuit condition xref.stream.isDataLoaded decides when the whole mechanism is completely unnecessary. BaseStream.isDataLoaded returns true by default, an in-memory Stream is always true, and only ChunkedStream returns false while chunks are still missing. Opening a local file therefore makes every ObjectLoader.load call free, and a network document skips it too once the whole file has downloaded.

The check is on xref.stream, the raw stream, not on some decoded stream. Decoded streams (FlateStream, DecodeStream and friends) inherit the default true, even though the chunked stream underneath them may not be loaded. That difference is handled by getBaseStreams inside #walk, not by the short circuit.

Iterative Depth-First Walk

#walk

  async #walk(nodesToVisit) {
    const nodesToRevisit = [];
    const pendingRequests = [];
    // DFS walk of the object graph.
    while (nodesToVisit.length) {
      let currentNode = nodesToVisit.pop();
 
      // Only references or chunked streams can cause missing data exceptions.
      if (currentNode instanceof Ref) {
        // Skip nodes that have already been visited.
        if (this.refSet.has(currentNode)) {
          continue;
        }
        try {
          this.refSet.put(currentNode);
          currentNode = this.xref.fetch(currentNode);
        } catch (ex) {
          if (!(ex instanceof MissingDataException)) {
            warn(`ObjectLoader.#walk - requesting all data: "${ex}".`);
 
            await this.xref.stream.manager.requestAllChunks();
            return;
          }
          nodesToRevisit.push(currentNode);
          pendingRequests.push({ begin: ex.begin, end: ex.end });
        }
      }
      if (currentNode instanceof BaseStream) {
        const baseStreams = currentNode.getBaseStreams();
        if (baseStreams) {
          let foundMissingData = false;
          for (const stream of baseStreams) {
            if (stream.isDataLoaded) {
              continue;
            }
            foundMissingData = true;
            pendingRequests.push({ begin: stream.start, end: stream.end });
          }
          if (foundMissingData) {
            nodesToRevisit.push(currentNode);
          }
        }
      }
 
      addChildren(currentNode, nodesToVisit);
    }
 
    if (pendingRequests.length) {
      await this.xref.stream.manager.requestRanges(pendingRequests);
 
      for (const node of nodesToRevisit) {
        // Remove any reference nodes from the current `RefSet` so they
        // aren't skipped when we revist them.
        if (node instanceof Ref) {
          this.refSet.remove(node);
        }
      }
      await this.#walk(nodesToRevisit);
    }
  }

The whole method breaks into four actions: pop the stack, handle references, check streams, push children. After the loop, if the round collected missing ranges, it requests them in one batch and re-enters.

Pop the stack. nodesToVisit.pop() makes the walk depth first: the last node pushed is visited first. There is no recursive function call here; the stack is an explicit array. PDF object graphs can be deep, with page trees, nested arrays and resource dictionaries layered on top of each other, and a corrupt file can build an extremely deep structure. An explicit stack keeps depth on the heap, so the whole worker does not crash from a call stack overflow.

Handle references. References are the only nodes that actively resolve. Before visiting one, the walk checks refSet and skips it if already visited; this is the core of cycle detection. refSet.put happens before fetch, not after, for two reasons: it prevents a self reference or a mutual reference from re-entering the same node while fetch recurses, and it makes a reference that appears twice in one round resolve only once.

The RefSet implementation is light: internally a Set whose elements are the result of Ref.toString(), that is, a compact string like 12R (12R1 when the generation is non-zero). has, put and remove each perform one string conversion. For a single reference the cost is negligible; for a whole page of resources it buys O(1) cycle detection, far cheaper than object-identity comparison on dictionaries or arrays.

A typical cycle forms like this: some XObject dictionary in the page resources references the page itself, and the page points back to that XObject through Resources. The first visit to the page reference puts it into refSet; following the resources back to the same reference, has returns true and the walk skips it. Without that step the walk would bounce between the two objects forever.

When fetch throws MissingDataException, the current node is still the Ref; it goes into nodesToRevisit and the missing range goes into pendingRequests. Because the reference was never replaced by an object, the stream check and addChildren do not expand it, and the round ends there until the data arrives.

When fetch throws anything else, the handling is completely different: log a warning, call requestAllChunks() to pull every missing chunk, and return immediately. This is a backstop. A non-missing-data exception usually means the file itself is broken, or that missing bytes were wrapped in another error; fine-grained range requests are pointless then, and pulling the whole file lets later synchronous parsing decide how to recover. Note that the node is not added to the revisit list: this prefetch round is over, and the failing object is retried when it is actually accessed.

Check streams. Stream nodes need no fetch, but their data may not have arrived. getBaseStreams() reaches through decoded streams to the real chunked streams underneath: ChunkedStream returns itself, DecodeStream delegates to its inner stream, StreamsSequenceStream concatenates the results of all child streams, and a plain Stream returns null. If any base stream is not loaded, its full range goes into the request list and the current stream node goes into the revisit list.

The request covers the whole range from stream.start to stream.end, not the part that is missing. The chunk manager rounds to fixed chunk sizes, so requesting the full range and requesting only the gap usually land on the same chunks, and the full-range form is simpler and avoids new boundary errors when computing the gap.

Push children. Whatever the current node is, a freshly fetched dictionary or a checked stream, addChildren is called at the end. A stream contributes its dictionary, a dictionary contributes its raw values, and an array contributes its elements.

Batch the request and re-enter. After the loop, only a non-empty range list enters the asynchronous phase. requestRanges belongs to ChunkedStreamManager: it converts each range into chunk numbers, deduplicates and sorts them, merges contiguous chunks into groups and issues them concurrently. Once the data arrives, the Refs in the revisit list are removed from refSet; otherwise the next round would skip them as already visited. Stream nodes need no removal because they are not in refSet.

The recursion calls #walk itself, but its depth equals the number of rounds needed to fill data, not the depth of the object graph. Each round requests exactly the ranges that were really missing in the previous one, and normally the next round fetches the objects; only when an object references further unloaded ranges does another round happen.

Batched Requests and Retries

pendingRequests is a local array that is submitted and dropped at the end of every round. Its elements have only a start and an end field:

A page load may produce dozens of ranges, but only a few requests actually go out: adjacent resources usually sit in the same stretch of bytes and are merged into contiguous chunk ranges after sorting. That is the direct payoff of batching.

The default chunk size is 64 KiB (2 ** 16), so even a 100-byte gap makes the manager pull a whole chunk. Requesting the full stream range and requesting only the gap therefore usually land on the same chunks, and the former is simpler and avoids boundary errors when computing the gap. For a small file a single requestRanges may pull the whole document, after which isDataLoaded turns true and prefetching short-circuits for every later page.

Both ObjectLoader and NetworkPdfManager.ensure deal with missing bytes, but at different granularity:

The two are not alternatives. ensure handles getting the dictionary the first time; ObjectLoader handles preparing everything behind that dictionary in one shot. loadResources first ensures resources and then hands it to ObjectLoader, which is exactly that division.

Failed range requests are not caught inside #walk. If the promise returned by requestRanges rejects, for example because the worker was terminated, the error travels up to the caller of ObjectLoader.load, and the layer above decides whether to abort the page or ignore it. ObjectLoader neither swallows exceptions nor counts retries; it assumes the data can be obtained, or the layer above will terminate.

Version Note: Where the Old Interface Went

If you have read the ObjectLoader of an early PDF.js, you may remember has(objId), get(objId, callback), the instance field pendingRequests and the onProgress callback. None of those four exist in this version; the whole file is refSet, load and #walk.

The old interface was designed around waiting for a single object by object number: given an id, the caller could ask whether it was ready and get a callback once it was. This version changes the unit of loading to one key walk and the unit of waiting to the whole promise, and keeps no object-number to callback map internally. The interface is smaller, but callers must state up front which resource keys they need.

The meaning of lazy loading has not changed, though. ObjectLoader prefetches only the subtree the caller names; the page tree, the resources of other pages and catalog objects are out of scope. Objects are still fetched by Dict.get on access, and prefetching merely removes the missing-byte obstacle in advance so that later synchronous access hits the XRef cache.

The getImageData Constraint

There is an easily overlooked comment on BaseStream:

  /**
   * NOTE: This method can only be used to get image-data that is guaranteed
   *       to be fully loaded, since otherwise intermittent errors may occur;
   *       note the `ObjectLoader` class.
   */
  async getImageData(length, decoderOptions) {
    return this.getBytes(length, decoderOptions);
  }

This constraint is directly related to ObjectLoader. Image decoding is not a byte-by-byte fault-tolerant read: the decoder usually takes the whole image payload at once and hands it to the JPEG, JPX or JBIG2 decoders. If only part of the data is loaded, some paths throw MissingDataException and others keep decoding truncated bytes, producing wrong pixels or intermittent rendering.

ObjectLoader provides the guarantee through the stream check: the page resource keys include XObject, so image streams are visited as stream nodes, their base stream ranges are requested in full, and by the time load finishes those bytes are in memory. When PartialEvaluator later parses the image dictionary synchronously and decodes the data asynchronously, it no longer sees half-loaded data.

The comment says "guaranteed to be fully loaded", not that bytes can never be missing. It is a contract: callers must ensure the data is complete, either by prefetching or by downloading the whole file, before calling it. One purpose of ObjectLoader is to keep that contract satisfiable in a streaming scenario.

Design note: ObjectLoader neither parses nor decodes; it only lifts the cross-cutting concern of data availability out of transactional code. Once lifted, the parsing layer can stay synchronous and simple, while the data loading layer can batch, deduplicate and retry on its own. The only contract between them is this: after prefetching completes, the named subtree no longer has missing bytes.

Conclusion

ObjectLoader is PDF.js's object-graph prefetch layer: upward it serves resource loading for Page, Annotation and XFA, and downward it drives XRef.fetch and ChunkedStreamManager.requestRanges.

Where It Sits in the Overall Chain

In one sentence: ObjectLoader works out in advance where bytes will be missing, merges how many bytes a round needs, and lets the synchronous parsing that follows pretend the network does not exist.

Five Design Principles

1. Prefetching beats retrying everywhere. Missing-byte handling is centralized in one place, so business code does not need a catch and retry after every get.
2. Iteration beats recursion. The explicit array stack frees traversal depth from the call stack, which is steadier on corrupt files and very deep object graphs.
3. The walk reads raw values only. getRawValues triggers no reference resolution, so the walk is never interrupted by missing bytes and the stack state stays under control.
4. Only references get cycle protection. RefSet records visited references; dictionaries and arrays cannot form real cycles, so no object-identity comparison is done and the per-visit hash cost is saved.
5. Batch the requests, advance in rounds. Missing ranges are collected before submission and deduplicated and sorted by the chunk manager; each round requests only the real gaps, and the recursion depth is set by the number of data rounds.

The next article follows this data supply chain further down, to see how pdf_manager.js chooses between local bytes and network chunks, and how PDFDocument and XRef get the bytes they need through the manager.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.