PDF.js Deep Dive 10: XRef

Introduction

This is the tenth article in the PDF.js series. It covers the XRef class in the worker-side src/core/xref.js.

The previous articles already explained how a document comes up:

• Article 6 explains PDFManager.initDocument: find startxref at the end of the file, create the XRef, and parse the trailer and the page tree.
• Article 7 explains PartialEvaluator: turn a page content stream into an operatorList.
• Article 8 explains PDFDocument: the page model, Resources, annotations and rendering intent.

In all those articles the same action keeps appearing — xref.fetch(ref). Font dictionaries, image streams, content streams, page-tree nodes, all come back through it. But the step "how does a 12 0 R become an object in memory" was never expanded. This article fills that gap: how objects are located, and how they are retrieved.

In one sentence, XRef is the whole PDF's index plus its librarian. The index records "which object number lives at which byte"; the librarian reads the object out by number; and if an object is stuffed into an object stream (ObjStm), the librarian has to decompress and unpack it first.

Module Overview

xref.js is roughly 1063 lines and splits into six parts:

Where it sits in the chain:

PDF bytes
  → PDFDocument.startXRef        (read startxref from the end of the file)
  → XRef.setStartXRef / parse
  → readXRef                     (classic table / xref stream / Prev chain)
  → #entries                     (object number → sparse array of entries)
  → fetch(ref)                   (read the offset, or decompress the ObjStm)
  → Dict / Array / BaseStream
  → Catalog / PDFDocument / PartialEvaluator

Version note: This article follows exactly the version of xref.js in the current repository. Its constructor is constructor(stream, pdfManager) and takes no catalog, dispatcher, filesize or docId parameters; it also has no cleanup(), statistics or getPage() methods. The entries / _entries common in older versions is the private field #entries here, and fetchAsyncIfRef is called fetchIfRefAsync in this version. The Catalog and Page responsibilities live in PDFDocument and Catalog respectively; XRef only keeps getCatalogObj() returning the root dictionary. Do not apply APIs from other versions you remember when reading the source.

Data Structures: Entries and Caching

PDF indirect references are written 12 0 R: object number 12, generation 0. XRef has only one question to answer — where is object 12. There are two answers:

1. At some byte offset in the file, parsed directly;
2. Inside some object stream (ObjStm): decompress the object stream first, then take it out by index.

There is no enum called XRefEntryType in the source; the three types are encoded by two boolean fields on the entry:

The third row is the easiest to misread: a compressed entry's gen is not a generation but the object's index inside the ObjStm. fetchCompressed ultimately does entries[xrefEntry.gen] to get the result.

All of XRef's private fields are initialized at the top of the class:

class XRef {
  #cacheMap = new Map();
 
  #entries = [];
 
  #newPersistentRefNum = null;
 
  #newTemporaryRefNum = null;
 
  #parsedWithRecovery = false;
 
  #pendingRefs = new RefSet();
 
  #persistentRefsCache = null;
 
  #xrefSectionOffsets = new Set();
 
  #xrefSectionsComplete = true;
 
  #xrefStms = new Set();

#entries is a sparse array rather than a Map because an object number is itself an integer starting at 0, so the array index can address it directly; huge object numbers that appear in corrupt files only leave holes and do not inflate the useful data. When iterating, the source deliberately uses for (const num in this.#entries) instead of for..of; the comment is blunt: do not iterate empty slots.

Looking up an entry from the outside goes through one small gate:

  getEntry(i) {
    const entry = this.#entries[i];
    return entry && !entry.free && entry.offset ? entry : null;
  }

It folds three cases into null: no record, explicitly free, zero offset. Callers therefore only need to ask "is there an entry", and do not care whether the xref wrote f or wrote nothing at all.

Design note: #entries and #cacheMap are two entirely different things. The former is the disk layout, answering "where is the object"; the latter is the in-memory object, answering "has the object already been parsed". Once the two are decoupled, an xref section can be rebuilt without touching the object cache (indexObjects clears both at the same time), and the object cache can keep hitting repeatedly while the xref stays unchanged. Any implementation that mixes the two will break in incremental updates or recovery mode.

constructor and Reference Number Allocation

The constructor takes only two parameters:

  constructor(stream, pdfManager) {
    this.stream = stream;
    this.pdfManager = pdfManager;
 
    if (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) {
      // For testing purposes.
      Object.defineProperty(this, "xrefSectionOffsetsAdd", {
        value: offset => this.#xrefSectionOffsets.add(offset),
      });
    }
  }

The dependencies are kept to a minimum:

PDFDocument passes both in when it is constructed, and only afterwards calls parse:

this.pdfManager = pdfManager;
this.stream = stream;
this.xref = new XRef(stream, pdfManager);

The order matters: when XRef is created the file may be only partially loaded, and parse may throw MissingDataException because bytes are missing, after which PDFManager completes the data and retries. XRef itself holds no "already done" assumptions.

Besides locating objects, XRef also allocates new object numbers. When saving or printing, the editor needs to create new objects, but must not collide with existing object numbers and should reuse where possible. getNewPersistentRef increments from #entries.length || 1 and writes the object straight into the cache; temporary references must "save numbers", so persistent references are temporarily moved out of #cacheMap, avoiding a collision between numbers newly allocated during the save and existing numbers; resetNewTemporaryRef puts them back once saving finishes. This is number-space management for the editor writing the PDF back and has little to do with the read path, but it explains why objects without a matching xref entry can appear in #cacheMap.

startXRef: Finding the First Door from the End of the File

The end of a PDF file usually looks like this:

startxref
123456
%%EOF

The decimal number after startxref is the byte offset of the latest xref section (relative to the %PDF- header). The XRef side only exposes a very small entry point:

  setStartXRef(startXRef) {
    // Store the starting positions of xref tables as we process them
    // so we can recover from missing data errors
    this.startXRefQueue = [startXRef];
  }

Note that it stores a queue, not a single value. Because parsing one xref section also discovers /Prev pointing at an earlier section, and hybrid files also have /XRefStm pointing at an xref stream in the same revision. All those offsets are appended to this queue.

The code that actually scans for startxref is not in xref.js but in the PDFDocument getter:

// Find `startxref` by checking backwards from the end of the file.
const step = 1024;
const startXRefLength = STARTXREF_SIGNATURE.length;
let found = false,
  pos = stream.end;
 
while (!found && pos > 0) {
  pos -= step - startXRefLength;
  if (pos < 0) {
    pos = 0;
  }
  stream.pos = pos;
  found = find(stream, STARTXREF_SIGNATURE, step, true);
}
 
if (found) {
  stream.skip(9);
  let ch;
  do {
    ch = stream.getByte();
  } while (isWhiteSpace(ch));
  let str = '';
  while (ch >= /* Space = */ 0x20 && ch <= /* '9' = */ 0x39) {
    str += String.fromCharCode(ch);
    ch = stream.getByte();
  }
  startXRef = parseInt(str, 10);
  if (isNaN(startXRef)) {
    startXRef = 0;
  }
}

Three details:

1. Backwards, in 1024-byte steps. startxref is always near the end of the file, but may be pushed further away by garbage bytes, comments or repeated incremental updates. A reverse scan is far faster than scanning from the beginning, and the last parameter of find(..., true) means "search backwards".
2. Read only digit characters. After skipping startxref it skips whitespace, then accepts only characters from 0x20 to 0x39. Reading the % of %%EOF stops it.
3. On parse failure, set 0. NaN is not thrown; instead the start point is set to 0, letting the later readXRef fail and the worker switch to recovery mode. This is a "do not die yet, leave it to the layer above" choice.

Linearized PDFs take another branch: the objects of the first page are placed at the start of the file, and startxref points at the end of the first object rather than at the last xref section. When PDFDocument detects a linearization dictionary it takes the end position of the first object directly, avoiding another read of the file tail just for the first screen. Both paths ultimately feed into setStartXRef.

Offsets inside startxref are relative to the PDF header. If the file has garbage bytes in front, checkHeader calls moveStart to adjust stream.start, so readXRef uniformly writes:

stream.pos = startXRef + stream.start;

readXRef: Walking Back Along /Prev

This is the most central traversal logic in the whole file:

  readXRef(recoveryMode = false) {
    const stream = this.stream;
    // Keep track of already parsed XRef tables, to prevent an infinite loop
    // when parsing corrupt PDF files where e.g. the /Prev entries create a
    // circular dependency between tables (fixes bug1393476.pdf).
    const startXRefParsedCache = new Set();
 
    while (this.startXRefQueue.length) {
      try {
        const startXRef = this.startXRefQueue[0];
 
        if (startXRefParsedCache.has(startXRef)) {
          warn("readXRef - skipping XRef table since it was already parsed.");
          this.startXRefQueue.shift();
          continue;
        }
        startXRefParsedCache.add(startXRef);
 
        stream.pos = startXRef + stream.start;
 
        const parser = new Parser({
          lexer: new Lexer(stream),
          xref: this,
          allowStreams: true,
        });
        let obj = parser.getObj();
        let dict;
 
        // Get dictionary
        if (isCmd(obj, "xref")) {
          // Parse end-of-file XRef
          dict = this.processXRefTable(parser);
          this.topDict ||= dict;
 
          // Recursively get other XRefs 'XRefStm', if any
          obj = dict.get("XRefStm");
          if (Number.isInteger(obj) && !this.#xrefStms.has(obj)) {
            // ignore previously loaded xref streams
            // (possible infinite recursion)
            this.#xrefStms.add(obj);
            this.startXRefQueue.push(obj);
          }
        } else if (Number.isInteger(obj)) {
          // Parse in-stream XRef
          if (
            !Number.isInteger(parser.getObj()) ||
            !isCmd(parser.getObj(), "obj") ||
            !((obj = parser.getObj()) instanceof BaseStream)
          ) {
            throw new FormatError("Invalid XRef stream");
          }
          dict = this.processXRefStream(obj);
          this.topDict ||= dict;
 
          if (!dict) {
            throw new FormatError("Failed to read XRef stream");
          }
        } else {
          throw new FormatError("Invalid XRef stream header");
        }
 
        this.#xrefSectionOffsets.add(startXRef);
 
        // Recursively get previous dictionary, if any
        obj = dict.get("Prev");
        if (Number.isInteger(obj)) {
          this.startXRefQueue.push(obj);
        } else if (obj instanceof Ref) {
          // The spec says Prev must not be a reference, i.e. "/Prev NNN"
          // This is a fallback for non-compliant PDFs, i.e. "/Prev NNN 0 R"
          this.startXRefQueue.push(obj.num);
        }
      } catch (e) {
        if (e instanceof MissingDataException) {
          throw e;
        }
        this.#xrefSectionsComplete = false;
        info("(while reading XRef): " + e);
      }
      this.startXRefQueue.shift();
    }
 
    if (this.topDict) {
      return this.topDict;
    }
    if (recoveryMode) {
      return undefined;
    }
    throw new XRefParseException();
  }

Although the comments say recursively, the implementation is actually queue-driven iteration. Each round takes the offset at the head of the queue; after parsing, newly discovered offsets are appended to the tail, until the queue is empty. The benefit: recovery mode, hybrid files and MissingDataException retries can all share the same state without maintaining a recursion stack.

The /Prev Chain and "Newer Sections Win"

The first token tells you the type: the xref command takes the classic table path, an integer takes the N 0 obj form of an xref stream, and anything else is an immediate error. Both branches end up producing a dictionary — for the classic table it is the dictionary after trailer, for an xref stream it is the stream's own dictionary — so the following logic can treat them uniformly.

/Prev points at the xref section of the previous revision; repeated incremental updates form a chain back toward the file header. readXRef starts from the newest and appends older sections to the queue in turn. Entries are written with ??=:

this.#entries[first + i] ??= entry;

That single line determines the correctness of the whole traversal order: the newest entry written first is never overwritten by an older entry parsed later. The queue must therefore stay "new first, old last"; the same applies to the object cache — once the newest section's objects enter #cacheMap, older sections pointing at the same object number are never consulted again. PDF's incremental update semantics thus land on the write order of the data structures.

Hybrid-Reference Files and /XRefStm

Since PDF 1.5 introduced xref streams, there is a class of "hybrid-reference files": the file uses a classic xref table as its main index, but the trailer carries an extra /XRefStm pointing at an xref stream in the same revision. That way an old reader can still open the file with only the classic table, while a new reader gets the entries for compressed objects from the xref stream.

The handling logic is only four lines:

obj = dict.get('XRefStm');
if (Number.isInteger(obj) && !this.#xrefStms.has(obj)) {
  this.#xrefStms.add(obj);
  this.startXRefQueue.push(obj);
}

The division of labour between the two sets must be kept straight:

Because #XRefStm is enqueued before /Prev, the queue order is "current section → xref stream of the same revision → previous revision". Entries already written by the classic table win; the xref stream only supplies type 2 (compressed object) records that the classic table cannot express. This is exactly why hybrid files work: two indexes stacked together, each responsible for part of the objects.

Fault Tolerance for Corrupt Files

The catch branch of readXRef does only three things:

1. MissingDataException is rethrown as-is. It does not mean "the file is broken" but "these bytes are not loaded yet", so the layer above must fetch more data and retry.
2. Other exceptions only log one info line and set #xrefSectionsComplete to false.
3. Continue with the next section in the queue.

This means a single corrupt xref section does not terminate the whole document: as long as another section can supply Root, the document still opens. #xrefSectionsComplete is later used by countUpdatesAfter to decide "whether the number of incremental updates can be counted reliably" — when the file is incomplete it returns null instead of a possibly wrong number.

If the queue runs out without a topDict, non-recovery mode throws XRefParseException, which the worker catches and switches to indexObjects; recovery mode returns undefined and lets the caller decide the next step.

Design note: readXRef separates "traversal" from "interpretation": the loop only takes offsets in order, enqueues and deduplicates, while the actual table/stream parsing is left to processXRefTable and processXRefStream. That way MissingDataException handling is centralized in one place, and the resume state of the two formats (_tableState and streamState) is maintained separately without interference. Fault tolerance is layered too: one bad section only degrades one section, and only "the bytes have not arrived" is a hard error that must be rethrown.

The Classic xref Table

The classic table is plain text, structured like this:

xref
0 3
0000000000 65535 f
0000000015 00000 n
0000000123 00000 n
trailer
<< /Size 3 /Root 1 0 R >>
startxref
1234
%%EOF

processXRefTable connects "the table" and "the trailer dictionary":

  processXRefTable(parser) {
    this._tableState ??= {
      entryNum: 0,
      streamPos: parser.lexer.stream.pos,
      parserBuf1: parser.buf1,
      parserBuf2: parser.buf2,
    };
 
    const obj = this.readXRefTable(parser);
 
    if (!isCmd(obj, "trailer")) {
      throw new FormatError(
        "Invalid XRef table: could not find trailer dictionary"
      );
    }
    let dict = parser.getObj();
 
    if (dict instanceof BaseStream) {
      dict = dict.dict;
    }
    if (!(dict instanceof Dict)) {
      throw new FormatError(
        "Invalid XRef table: could not parse trailer dictionary"
      );
    }
    delete this._tableState;
 
    return dict;
  }

_tableState is the resume state. During chunked network loading, reading into the middle of the table may find that the required bytes have not arrived; MissingDataException tears down the whole call stack; once the data is complete and readXRef retries, processXRefTable sees _tableState already exists and reuses it, and readXRefTable restores the stream position and the parser's two buffered tokens, continuing from the entry where it stopped. Without this state, every missing chunk would reparse the table from the beginning.

The dictionary after trailer is read by parser.getObj(). The << ... >> in the comment is PDF dictionary syntax; the parser provides a key-based lookup interface. Files produced by pdflib may wrap the trailer as a stream, so the source checks for BaseStream and takes .dict.

readXRefTable

Only the inner loop is excerpted here; the earlier stream-position restore, the subsection header read and the trailing trailer check are all done by the same method:

for (let i = tableState.entryNum; i < count; i++) {
  tableState.streamPos = stream.pos;
  tableState.entryNum = i;
  tableState.parserBuf1 = parser.buf1;
  tableState.parserBuf2 = parser.buf2;
 
  const entry = {
    offset: parser.getObj(),
    gen: parser.getObj(),
    free: false,
    uncompressed: false,
  };
  const type = parser.getObj();
 
  if (type instanceof Cmd) {
    switch (type.cmd) {
      case 'f':
        entry.free = true;
        break;
      case 'n':
        entry.uncompressed = true;
        break;
    }
  }
 
  // Validate entry obj
  if (
    !Number.isInteger(entry.offset) ||
    !Number.isInteger(entry.gen) ||
    !(entry.free || entry.uncompressed)
  ) {
    throw new FormatError(`Invalid entry in XRef subsection: ${first}, ${count}`);
  }
 
  // The first xref table entry, i.e. obj 0, should be free. Attempting
  // to adjust an incorrect first obj # (fixes issue 3248 and 7229).
  if (i === 0 && entry.free && first === 1) {
    first = 0;
  }
  this.#entries[first + i] ??= entry;
}

Item by item:

1. Restore positions. Three states are restored together: stream.pos, parser.buf1 and parser.buf2. Restoring only the stream position is not enough — the parser is a streaming lexer and may already have read ahead the two tokens of the next record.
2. Outer loop over subsections. Each subsection header is "first object number + count", for example 23 2 means 2 records starting at object 23. firstEntryNum and entryCount are only read when a subsection starts; on re-entry, if these two fields are still present the read is skipped, avoiding reading an already-consumed header again.
3. Inner loop over records. Each record has exactly three parts: offset, generation, and f or n. entryNum records which record is being processed, and re-entry continues from it.
4. Type validation. Offset and generation must be integers, and at least one of f / n must hit; otherwise the whole table is judged corrupt.
5. First-object fixup. The spec requires object 0 to be free, but some generators write the first record as starting at 1. When i === 0, the entry is free and first === 1, the source changes first to 0, fixing the index. This is compatibility for issues 3248 and 7229.
6. ??= writes. The new section is parsed first and the old section later, so ??= guarantees the new section wins.
7. After a whole section is read, the subsection header fields are cleared, and the next loop iteration reads the next subsection, until the trailer command is reached and the loop breaks.
8. Final validation. If #entries[0] exists and is not free, the file structure does not conform to the spec, and a FormatError is thrown.

The classic table can only produce free and uncompressed entries; it cannot express compressed objects. Compressed objects depend on xref streams or the object stream index (see the next section and fetchCompressed).

Classic Table vs xref Stream

xref Streams

An xref stream turns the index itself into a stream object. processXRefStream establishes the resume state:

  processXRefStream(stream) {
    if (!("streamState" in this)) {
      // Stores state of the stream as we process it so we can resume
      // from middle of stream in case of missing data error
      const { dict, pos } = stream;
      const byteWidths = dict.get("W");
      const range = dict.get("Index") || [0, dict.get("Size")];
 
      this.streamState = {
        entryRanges: range,
        byteWidths,
        entryNum: 0,
        streamPos: pos,
      };
    }
    this.readXRefStream(stream);
    delete this.streamState;
 
    return stream.dict;
  }

Three elements:

• /W: an array of three integers giving the byte widths of type, field2 and field3.
• /Index: a list of ranges in pairs, defaulting to [0, Size], meaning starting at object 0 for Size records.
• entryRanges is a mutable array; after each range is read it is consumed with splice(0, 2), so the remaining ranges survive a re-entry.

readXRefStream reads byte by byte according to the widths and assembles big-endian integers by hand:

  readXRefStream(stream) {
    const streamState = this.streamState;
    stream.pos = streamState.streamPos;
 
    const [typeFieldWidth, offsetFieldWidth, generationFieldWidth] =
      streamState.byteWidths;
 
    const entryRanges = streamState.entryRanges;
    while (entryRanges.length > 0) {
      const [first, n] = entryRanges;
 
      if (!Number.isInteger(first) || !Number.isInteger(n)) {
        throw new FormatError(`Invalid XRef range fields: ${first}, ${n}`);
      }
      if (
        !Number.isInteger(typeFieldWidth) ||
        !Number.isInteger(offsetFieldWidth) ||
        !Number.isInteger(generationFieldWidth)
      ) {
        throw new FormatError(
          `Invalid XRef entry fields length: ${first}, ${n}`
        );
      }
      for (let i = streamState.entryNum; i < n; ++i) {
        streamState.entryNum = i;
        streamState.streamPos = stream.pos;
 
        let type = 0,
          offset = 0,
          generation = 0;
        for (let j = 0; j < typeFieldWidth; ++j) {
          const typeByte = stream.getByte();
          if (typeByte === -1) {
            throw new FormatError("Invalid XRef byteWidths 'type'.");
          }
          type = (type << 8) | typeByte;
        }
        // if type field is absent, its default value is 1
        if (typeFieldWidth === 0) {
          type = 1;
        }
        for (let j = 0; j < offsetFieldWidth; ++j) {
          const offsetByte = stream.getByte();
          if (offsetByte === -1) {
            throw new FormatError("Invalid XRef byteWidths 'offset'.");
          }
          offset = (offset << 8) | offsetByte;
        }
        for (let j = 0; j < generationFieldWidth; ++j) {
          const generationByte = stream.getByte();
          if (generationByte === -1) {
            throw new FormatError("Invalid XRef byteWidths 'generation'.");
          }
          generation = (generation << 8) | generationByte;
        }
        const entry = {
          offset,
          gen: generation,
          free: false,
          uncompressed: false,
        };
        switch (type) {
          case 0:
            entry.free = true;
            break;
          case 1:
            entry.uncompressed = true;
            break;
          case 2:
            break;
          default:
            throw new FormatError(`Invalid XRef entry type: ${type}`);
        }
        this.#entries[first + i] ??= entry;
      }
 
      streamState.entryNum = 0;
      streamState.streamPos = stream.pos;
      entryRanges.splice(0, 2);
    }
  }

A few points that are easy to overlook:

1. A width can be 0. When the type field width is 0 the row has no type byte, and the spec says the default is 1 (normal object). This is for compatibility with old writers that only record offset and generation.
2. An offset can exceed 4 bytes. The loop accumulates by width; (offset << 8) | byte can assemble safe integers of up to 8 bytes and does not truncate large files.
3. Reaching the end of the file throws. When stream.getByte() returns -1 it throws three different errors for 'type', 'offset' and 'generation', making it easy to locate which field is incomplete.
4. type 2 sets no boolean. free and uncompressed both stay false, so the entry naturally falls into the "compressed object" branch, and fetch calls fetchCompressed.
5. ??= still gives the new section priority. Exactly the same as the classic table.

/W, /Index and Field Meanings

/Index is paired [first, count, first, count, ...]. A file can have several non-contiguous object number ranges, for example one range for body objects and another for deleted objects. The source consumes them pair by pair with entryRanges.splice(0, 2), while entryNum records the index within the current range — together these two states are everything needed to resume. Note that the xref stream dictionary has no key such as /Fields; the names and order of the three fields are defined entirely by /W; "field2" and "field3" are merely descriptive names for the different meanings the same position has under different types.

Predictor: the xref Stream Is Already Decompressed Before It Reaches XRef

An xref stream almost always uses /Filter /FlateDecode with a PNG predictor in /DecodeParms:

<< /Type /XRef
   /Size 100
   /W [1 4 2]
   /Index [0 100]
   /Filter /FlateDecode
   /DecodeParms << /Columns 7 /Predictor 12 >>
   /Root 1 0 R
   /Length 1234 >>

There is no decompression or de-prediction code in readXRefStream, because the stream it receives is already decoded bytes. When readXRef creates the Parser it passes allowStreams: true; when the parser parses a stream object it calls makeFilter to put a decoder on it:

        case "Fl":
        case "FlateDecode":
          if (params) {
            return new PredictorStream(
              new FlateStream(stream, maybeLength),
              maybeLength,
              params
            );
          }
          return new FlateStream(stream, maybeLength);

PredictorStream then decides how to restore rows based on /Predictor:

  constructor(str, maybeLength, params) {
    super(maybeLength);
 
    if (!(params instanceof Dict)) {
      return str; // no prediction
    }
    const predictor = (this.predictor = params.get("Predictor") || 1);
 
    if (predictor <= 1) {
      return str; // no prediction
    }
    if (predictor !== 2 && (predictor < 10 || predictor > 15)) {
      throw new FormatError(`Unsupported predictor: ${predictor}`);
    }
 
    this.readBlock = predictor === 2 ? this.readBlockTiff : this.readBlockPng;

There is an implicit constraint here: /Columns must equal W[0] + W[1] + W[2] (usually further multiplied by /Colors and the bytes per sample). If the writer gets this wrong, de-prediction splits rows at the wrong width, and the type, offset and generation read from the xref stream are all shifted. readXRefStream can only see "illegal field width" or "illegal entry type"; the real root cause is in the predictor parameters.

Design note: An xref stream is a fixed-length binary record, not a PDF object. PDF.js does not construct an object for each record and does not use the generic getObj; it shifts and concatenates directly according to the /W widths. This path bypasses the lexer's tokenization overhead, and the difference is obvious on documents with tens of thousands of objects; the price is having to handle the edge cases of zero widths, offsets longer than 4 bytes and truncated streams yourself. Splitting "format parsing" and "byte decoding" into layers (PredictorStream upstream, readXRefStream downstream) lets both sides be tested independently.

Recovery Mode: indexObjects

When readXRef cannot obtain even one usable topDict, the worker retries once in recovery mode:

loadDocument(false).then(onSuccess, function (reason) {
  ensureNotTerminated();
 
  // Try again with recoveryMode == true
  if (!(reason instanceof XRefParseException)) {
    onFailure(reason);
    return;
  }
  pdfManager.requestLoadedStream().then(function () {
    ensureNotTerminated();
 
    loadDocument(true).then(onSuccess, onFailure);
  }, onFailure);
});

Recovery mode no longer trusts any xref and scans the whole file:

// Clear out any existing entries, since they may be bogus.
this.#entries.length = 0;
this.#cacheMap.clear();
 
const stream = this.stream;
stream.pos = 0;
const buffer = stream.getBytes(),
  bufferStr = bytesToString(buffer),
  length = buffer.length;
let position = stream.start;
const trailers = [],
  xrefStms = [];

The scanner advances token by token and cares about only four kinds of things:

Object entry offsets are relative to the PDF header:

this.#entries[num] = {
  offset: position - stream.start,
  gen,
  free: false,
  uncompressed: true,
};

This uses unconditional assignment, not ??=. The scan goes from the file header toward the file tail, and an object appearing later belongs to a later revision, so it should overwrite an identically numbered object appearing earlier. Before overwriting there is one more safeguard: if an existing entry has the same generation, it first tries to parse the new position with a temporary parser, and only overwrites if an object can be parsed out (issue13783); otherwise the old entry is kept, avoiding replacing a good index with a bad offset.

Finding a Usable trailer

After the scan, trailers may hold a pile of candidate dictionaries. The source pre-parses them all first to determine whether the file is encrypted; then runs two validation rounds in the order "candidates → set the fallback flag → candidates":

The validation criteria are: Root must be a dictionary, Pages inside Root must be a dictionary, and Count inside Pages is an integer. A candidate that satisfies these and (if the file is encrypted) carries Encrypt plus an ID wins outright. If none carries an ID, the last usable one is kept; failing that, topDict is used; failing that, it walks #entries looking for a dictionary containing Root; finally it throws Invalid PDF structure..

_generationFallback is switched on here. When all candidates in the first round fail, the second round lets fetchUncompressed fall back to an older generation when the generation does not match — this thread explains why there is a seemingly odd retry branch inside fetchUncompressed.

Scanning xref Streams

The scan phase notices objects whose content contains /XRef not followed by a letter and records them in the xrefStms array; after the scan they are queued one by one and readXRef(true) is called. This step lets recovery mode also use the type 2 records in binary xref streams. Otherwise compressed objects could only be discovered indirectly through the object number table inside object streams, at much higher cost.

Design note: Recovery mode is not "another parser" but rebuild the index, then take the same retrieval path. indexObjects is only responsible for finding object numbers, offsets and trailer candidates and writing them back to #entries; afterwards fetch, fetchUncompressed and fetchCompressed are reused entirely. This design confines "corrupt file compatibility" to the index layer, and does not let fault-tolerance logic seep into every object read site.

Retrieving Objects: the Three fetch Entry Points

XRef exposes not one retrieval method but three, for different call sites:

The first two are thin synchronous wrappers:

  fetchIfRef(obj, suppressEncryption = false) {
    return obj instanceof Ref ? this.fetch(obj, suppressEncryption) : obj;
  }

The async version retries on MissingDataException:

  async fetchIfRefAsync(obj, suppressEncryption) {
    return obj instanceof Ref ? this.fetchAsync(obj, suppressEncryption) : obj;
  }
 
  async fetchAsync(ref, suppressEncryption) {
    try {
      return this.fetch(ref, suppressEncryption);
    } catch (ex) {
      if (!(ex instanceof MissingDataException)) {
        throw ex;
      }
      await this.pdfManager.requestRange(ex.begin, ex.end);
      return this.fetchAsync(ref, suppressEncryption);
    }
  }

The Main fetch Flow

  fetch(ref, suppressEncryption = false) {
    if (!(ref instanceof Ref)) {
      throw new Error("ref object is not a reference");
    }
    const num = ref.num;
 
    // The XRef cache is populated with objects which are obtained through
    // `Parser.getObj`, and indirectly via `Lexer.getObj`. Neither of these
    // methods should ever return `undefined` (note the `assert` calls below).
    const cacheEntry = this.#cacheMap.get(num);
    if (cacheEntry !== undefined) {
      // In documents with Object Streams, it's possible that cached `Dict`s
      // have not been assigned an `objId` yet (see e.g. issue3115r.pdf).
      if (cacheEntry instanceof Dict && !cacheEntry.objId) {
        cacheEntry.objId = ref.toString();
      }
      return cacheEntry;
    }
    let xrefEntry = this.getEntry(num);
 
    if (xrefEntry === null) {
      // The referenced entry can be free.
      return xrefEntry;
    }
    // Prevent circular references, in corrupt PDF documents, from hanging the
    // worker-thread. This relies, implicitly, on the parsing being synchronous.
    if (this.#pendingRefs.has(ref)) {
      this.#pendingRefs.remove(ref);
 
      warn(`Ignoring circular reference: ${ref}.`);
      return CIRCULAR_REF;
    }
    this.#pendingRefs.put(ref);
 
    try {
      xrefEntry = xrefEntry.uncompressed
        ? this.fetchUncompressed(ref, xrefEntry, suppressEncryption)
        : this.fetchCompressed(ref, xrefEntry, suppressEncryption);
      this.#pendingRefs.remove(ref);
    } catch (ex) {
      this.#pendingRefs.remove(ref);
      throw ex;
    }
    if (xrefEntry instanceof Dict) {
      xrefEntry.objId = ref.toString();
    } else if (xrefEntry instanceof BaseStream) {
      xrefEntry.dict.objId = ref.toString();
    }
    return xrefEntry;
  }

Broken down by state, a lookup lands in one of four cases:

Some notes:

1. The cache takes priority over the entry. Once an object is parsed, later references to the same object number hit #cacheMap and never consult the entry or read bytes again. Incremental updates do update entries, but old object cache entries are kept; this is an optimization PDF.js makes under the assumption "the document does not change", and the editor explicitly resets the relevant caches when rewriting a PDF.
2. A free reference returns null, not an error. Referencing a free object is legal in PDF and means "this object does not exist"; the layer above treats null as a default value.
3. CIRCULAR_REF is a symbol sentinel. A corrupt file can construct a cycle where 1 0 R points at itself, or two dictionaries point at each other. #pendingRefs registers on entry and removes on exit; encountering the same ref again means we are going in circles, so Symbol("CIRCULAR_REF") is returned to let parsing continue. The comment stresses that this mechanism relies on "parsing is synchronous" — if fetch awaited in the middle, the on-stack mark would not hold.
4. objId backfill. A dictionary parsed out of an object stream does not know which reference it corresponds to at parse time, so objId is filled in on a cache hit; after fetch completes it also writes objId for a Dict or "the dictionary of a stream". Debugging and cycle detection both rely on this field.
5. Streams are not cached. fetchUncompressed writes #cacheMap only for non-BaseStream values, and fetchCompressed continues when it meets a stream. A stream is a stateful object with a read cursor; caching it would make several callers share the cursor; and decoded stream data is usually large, so keeping it resident in memory is a bad trade.

Design note: fetch is synchronous, reentrant, and has no Promise. Cycle detection relies on #pendingRefs, memoization on #cacheMap, and missing bytes on the outermost fetchAsync retry with requestRange. Together, the three let an object graph of arbitrary depth be traversed in a single synchronous call; asynchrony appears only in the one case "the bytes have not arrived". Separating "parsing logic" from "data loading" is the precondition for the worker side supporting both local files and chunked network streams.

fetchUncompressed: Reading 5 0 obj by Offset

Retrieving a normal object:

  fetchUncompressed(ref, xrefEntry, suppressEncryption = false) {
    const gen = ref.gen;
    let num = ref.num;
    if (xrefEntry.gen !== gen) {
      const msg = `Inconsistent generation in XRef: ${ref}`;
      // Try falling back to a *previous* generation (fixes issue15577.pdf).
      if (this._generationFallback && xrefEntry.gen < gen) {
        warn(msg);
        return this.fetchUncompressed(
          Ref.get(num, xrefEntry.gen),
          xrefEntry,
          suppressEncryption
        );
      }
      throw new XRefEntryException(msg);
    }
    const stream = this.stream.makeSubStream(
      xrefEntry.offset + this.stream.start
    );
    const parser = new Parser({
      lexer: new Lexer(stream),
      xref: this,
      allowStreams: true,
    });
    const obj1 = parser.getObj();
    const obj2 = parser.getObj();
    const obj3 = parser.getObj();
 
    if (obj1 !== num || obj2 !== gen || !(obj3 instanceof Cmd)) {
      throw new XRefEntryException(`Bad (uncompressed) XRef entry: ${ref}`);
    }
    if (obj3.cmd !== "obj") {
      // some bad PDFs use "obj1234" and really mean 1234
      if (obj3.cmd.startsWith("obj")) {
        num = parseInt(obj3.cmd.substring(3), 10);
        if (!Number.isNaN(num)) {
          return num;
        }
      }
      throw new XRefEntryException(`Bad (uncompressed) XRef entry: ${ref}`);
    }
    xrefEntry =
      this.encrypt && !suppressEncryption
        ? parser.getObj(this.encrypt.createCipherTransform(num, gen))
        : parser.getObj();
    if (!(xrefEntry instanceof BaseStream)) {
      if (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) {
        assert(
          xrefEntry !== undefined,
          'fetchUncompressed: The "xrefEntry" cannot be undefined.'
        );
      }
      this.#cacheMap.set(num, xrefEntry);
    }
    return xrefEntry;
  }

Item by item:

1. The generation must match. A PDF reference carries a generation, and so does an xref entry; a mismatch means the reference is stale. By default an XRefEntryException is thrown; only when recovery mode has switched on _generationFallback and the entry's generation is older does it retry recursively with the entry's generation. This is the fallback for corrupt files of the kind where "the reference writes a new generation but the file only has the old one".
2. The substream starts at the offset. makeSubStream(offset + stream.start) does not copy bytes; it just gives the same underlying data a different read start. This is also why PDF.js can work on chunked network streams: when the substream reads an unloaded chunk it throws MissingDataException just the same.
3. Validate the 5 0 obj header. Three tokens are read in a row and must equal the object number, the generation, and the obj command in order. If they do not match, the xref offset points at the wrong place, and an XRefEntryException is thrown directly.
4. obj1234 compatibility. Some generators write the command as obj1234 (with the object number right after it), and the parser lexes it into a single Cmd. The source strips the obj prefix and returns the number — which actually returns the object number rather than an object; this is a historical compatibility branch.
5. Decryption. When Encrypt exists and decryption is not suppressed, a CipherTransform is passed to the parser, and string and stream contents derive keys from the object number and generation. suppressEncryption is used for the Encrypt dictionary in the trailer, avoiding the deadlock of "decrypting the encryption dictionary requires the encryption dictionary" (the source comment mentions issue7665).
6. Only non-stream objects are cached. Dict, arrays, numbers and strings can be shared safely; streams have a cursor, and caching would share state.

fetchCompressed: Decompress the ObjStm, Then Unpack

For a compressed object's entry, offset is the object stream's object number and gen is the object's index inside the stream:

  fetchCompressed(ref, xrefEntry, suppressEncryption = false) {
    const tableOffset = xrefEntry.offset;
    const stream = this.fetch(Ref.get(tableOffset, 0));
    if (!(stream instanceof BaseStream)) {
      throw new FormatError("bad ObjStm stream");
    }
    const first = stream.dict.get("First");
    const n = stream.dict.get("N");
    if (!Number.isInteger(first) || !Number.isInteger(n)) {
      throw new FormatError("invalid first and n parameters for ObjStm stream");
    }
    let parser = new Parser({
      lexer: new Lexer(stream),
      xref: this,
      allowStreams: true,
    });
    const nums = new Array(n);
    const offsets = new Array(n);
    // read the object numbers to populate cache
    for (let i = 0; i < n; ++i) {
      const num = parser.getObj();
      if (!Number.isInteger(num)) {
        throw new FormatError(
          `invalid object number in the ObjStm stream: ${num}`
        );
      }
      const offset = parser.getObj();
      if (!Number.isInteger(offset)) {
        throw new FormatError(
          `invalid object offset in the ObjStm stream: ${offset}`
        );
      }
      nums[i] = num;
 
      // The entry in the xref table is the object number followed by the index.
      // So if index (gen number) is not the same as the index (i), we fix it
      // (fixes bug 1978317).
      const entry = this.getEntry(num);
      if (entry?.offset === tableOffset && entry.gen !== i) {
        entry.gen = i;
      }
 
      offsets[i] = offset;
    }

The layout of an object stream is "header table + object body":

ObjStm stream (after decompression)
┌──────────────────────────────┐
│ N=3  First=24                │  stream dictionary
│ 10 0   11 12   12 25         │  header: object number + relative offset
├──────────────────────────────┤  ← First points here
│ <object 10><object 11><object 12>  │  object body
└──────────────────────────────┘

First is the offset of the first object body relative to the start of the stream; the header table itself occupies First bytes. The header table has N pairs of "object number, relative offset". The length of each object is determined by the next object's offset; the last object is read to the end of the stream.

The body part builds a substream for each object and parses it; the parsed results are validated against the entry and written into #cacheMap; finally the result is taken out by index:

    xrefEntry = entries[xrefEntry.gen];
    if (xrefEntry === undefined) {
      throw new XRefEntryException(`Bad (compressed) XRef entry: ${ref}`);
    }
    return xrefEntry;
  }

Key design points:

1. Recursively fetch the object stream. this.fetch(Ref.get(tableOffset, 0)) fetches the ObjStm itself. Per the spec an ObjStm must be a normal object (type 1), so this recursion is at most one level deep, and "an ObjStm inside an ObjStm" cannot occur.
2. Unpack all objects at once. Although only the one designated by xrefEntry.gen is needed, the source still parses and caches all N objects. The point of an object stream is batch compression; usually the fonts, annotations and resources of the same page are crammed into the same stream, so one unpack also hits a large number of later references.
3. Fix a wrong gen. In an xref stream, type 2's field3 is supposed to be the object's index inside the ObjStm. Some files get this wrong, and the source uses the header table order i to fix the entry (bug 1978317). After the fix, entry.gen === i is used to decide whether to cache, avoiding writing the pre-fix wrong mapping into the cache.
4. suppressEncryption is unused here. ObjStm decryption happens wholesale when the stream itself is fetched; objects inside the stream do not go through CipherTransform separately. The parameter is kept to match fetchUncompressed's signature. This is a point that is easy to get wrong when reading the source.
5. Out of range is an error. entries[xrefEntry.gen] being undefined means the xref points at a nonexistent index; an XRefEntryException is thrown, and the layer above decides whether to ignore it.

trailer, topDict and the Root Dictionary

parse turns the xref's output into a trailer usable by the document:

  parse(recoveryMode = false) {
    this.#parsedWithRecovery = recoveryMode;
    let trailerDict;
    if (!recoveryMode) {
      trailerDict = this.readXRef();
    } else {
      warn("Indexing all PDF objects");
      trailerDict = this.indexObjects();
    }
    trailerDict.assignXref(this);
    this.trailer = trailerDict;

assignXref(this) binds the trailer dictionary to the XRef. When Dict.get meets a Ref value it calls xref.fetch directly:

      value = this.#map.get(key2);
    }
    if (value instanceof Ref && this.xref) {
      return isAsync
        ? this.xref.fetchAsync(value, this.suppressEncryption)
        : this.xref.fetch(value, this.suppressEncryption);
    }

So trailerDict.get("Root") does not return the reference 1 0 R but the actual Catalog dictionary. Likewise Root.get("Pages"), Pages.get("Kids") and so on, each level retrieved lazily — only nodes that are actually visited are parsed. This is why the "object model" can be built on top of "the index".

The difference between topDict and trailer:

topDict ||= dict assigns only the first time; trailer dictionaries from older sections never overwrite it. The PDF convention is that the trailer of the latest revision contains complete information; older trailers are only used for backtracking, not as the final result. If some generator writes /Root only in an older section, the Root validation in parse fails, the worker immediately switches to recovery mode, and indexObjects picks a candidate again.

Encryption is also handled here. trailer.get("Encrypt") may throw MissingDataException because bytes are missing; in that case it is rethrown as-is; other errors are only warned about. After the Encrypt dictionary is obtained:

if (encrypt instanceof Dict) {
  const ids = trailerDict.get('ID');
  const fileId = ids?.length ? ids[0] : '';
  // The 'Encrypt' dictionary itself should not be encrypted, and by
  // setting `suppressEncryption` we can prevent an infinite loop inside
  // of `XRef_fetchUncompressed` if the dictionary contains indirect
  // objects (fixes issue7665.pdf).
  encrypt.suppressEncryption = true;
  this.encrypt = new CipherTransformFactory(encrypt, fileId, this.pdfManager.password);
}

ID[0] participates in key derivation as the file identifier; pdfManager.password is the password the user entered. The Encrypt dictionary itself is marked suppressEncryption, preventing decryption from being triggered again when fetching indirect objects inside it.

Finally getCatalogObj just returns the validated root dictionary:

  getCatalogObj() {
    return this.root;
  }

this.root is saved in parse after the double validation of Root and Pages. The actual Catalog class is created by PDFDocument.parse after xref.parse; page lookup (getPage) also happens in Catalog / PDFDocument, and XRef takes no part in page tree traversal.

Laziness, Caching and Lifetime

XRef's performance comes from "on demand": the parse phase only builds the index and does not parse objects; bytes are read only when an object is first accessed. The whole chain has three cache layers:

The object layer has no eviction policy; it lives as long as the document. That is reasonable for reading: once a page is rendered, its fonts and resources are going to be reused anyway; and the genuinely large objects (streams) never enter the cache. When the user triggers cleanup, PDFDocument.cleanup forwards to Catalog.cleanup:

  async cleanup(manuallyTriggered = false) {
    return this.catalog
      ? this.catalog.cleanup(manuallyTriggered)
      : clearGlobalCaches();
  }

Catalog.cleanup clears document-level caches such as fontCache, globalImageCache and pageDictCache, and does not touch XRef.#cacheMap. XRef itself has no cleanup or statistics method either; the two lines at the start of indexObjects (clear #entries, clear #cacheMap) are its only proactive release of the index and object cache. Stream lifetime is managed by PDFManager: a local file is an in-memory Stream, a network file is a ChunkedStream, and terminate aborts them all together.

countUpdatesAfter

The only method with a "statistics" flavour is countUpdatesAfter:

  countUpdatesAfter(offset) {
    if (this.#parsedWithRecovery || !this.#xrefSectionsComplete) {
      return null;
    }
    const relativeOffset = offset - this.stream.start;
    let count = 0;
    for (const sectionOffset of this.#xrefSectionOffsets) {
      if (
        sectionOffset >= relativeOffset &&
        !this.#xrefStms.has(sectionOffset)
      ) {
        count++;
      }
    }
    return count;
  }

It counts "how many xref sections exist after a given byte position", which is used for digital signature validation: if an incremental update section exists after the range covered by the signature, the document has been modified. There are two conservative decisions:

• When recovery mode was used (#parsedWithRecovery), return null, because the section offsets rebuilt by indexObjects do not represent the real revision structure.
• When some section failed to parse (#xrefSectionsComplete is false), return null, because the count is incomplete.

Offsets in #xrefStms are excluded, because in a hybrid file the xref stream and the main table belong to the same revision and must not count as two updates. That detail directly determines whether signature validation produces a false positive.

Conclusion

XRef is PDF.js's object addressing layer: upward it gives Catalog, PDFDocument and PartialEvaluator a fetch(ref), and downward it strings startxref, xref tables, xref streams and object streams into a fault-tolerant read chain.

Where It Sits in the Overall Chain

In one sentence: PDFManager supplies the bytes, XRef turns the bytes into objects addressable by number, and Document and Evaluator organise those objects into pages and drawing instructions.

Five Design Principles

1. Index and cache are separated. #entries is the authority on disk layout; #cacheMap is the memory of parse results. Their lifetimes and responsibilities differ; recovery mode can rebuild only the index, and a cache hit needs no entry lookup.
2. New sections win, old sections fill gaps. readXRef starts from the newest startxref and walks older along /Prev; #entries[first + i] ??= guarantees the new section is not overwritten by the old one. In hybrid files /XRefStm only supplies compressed objects and does not steal records the classic table already has.
3. Synchronous retrieval, asynchronous byte supply. fetch is synchronous and reentrant, with cycle detection in #pendingRefs; only MissingDataException is caught by fetchAsync, which calls requestRange and retries from the start. Parsing logic and data loading are fully decoupled.
4. Fault tolerance is layered. A single broken xref section only marks #xrefSectionsComplete = false; when nothing is usable it switches to a full indexObjects scan; a generation mismatch falls back to an older generation in recovery mode; a circular reference returns CIRCULAR_REF instead of hanging. Each layer only handles the errors it can handle.
5. Binary records are read directly by width. xref streams do not go through generic object parsing but shift and concatenate according to /W; predictor and Flate decoding are done upstream by PredictorStream. Documents with tens of thousands of objects therefore lose a large amount of tokenization overhead.

At this point the loop "how objects are located and retrieved" is complete. This article deliberately skipped the very bottom layer, though: how the bytes of 12 0 obj are actually cut into tokens and assembled into a Dict. The next article returns to src/core/parser.js to look at how Lexer and Parser complete "bytes → PDF objects"; only after that does Catalog get its turn to organise a pile of dictionaries into a page tree, name tree and resource tables.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.