PDF.js Deep Dive 11: Parser

Introduction

This is the eleventh article in the PDF.js series. It covers the Lexer and Parser classes in the object syntax layer, src/core/parser.js (1,484 lines).

The previous articles looked at PDF from different levels:

• Article 6, PDFManager.initDocument, answers "where does the file start reading from" — find startxref, read the trailer, hand the cross-reference table to XRef.
• Article 10, XRef, answers "at which offset does an object with a given number live" — maintain entries, handle incremental updates, turn offsets into readable byte ranges.
• Article 7, PartialEvaluator, answers "what is the page content stream saying" — translate operator sequences like q 1 0 0 1 0 0 cm /F1 12 Tf ... into an operatorList.

These three things look unrelated, but at the lowest level they all depend on the same capability: turning a run of bytes into PDF base objects (numbers, strings, names, arrays, dictionaries, indirect references, streams), plus booleans and null. The reason an offset in an xref entry can become a Dict, and the reason q in a content stream can be recognized as an operator, is entirely these two classes.

One sentence summary of the division of labor: Lexer is the tokenizer; it only splits the byte stream into tokens. Parser is the parser; it uses two-token lookahead and recursive descent to assemble tokens into an object graph, and takes over the raw bytes when it encounters the stream keyword.

Design note: PDF's grammar is small enough to hand-write — no nested comments, no operator precedence, no ambiguity requiring backtracking. So PDF.js uses no parser generator and no parser combinator: just two classes, one while loop, and one recursive method. Both readability and performance come from that smallness.

Module Overview

parser.js has a very regular structure, six blocks:

The relationship between the two classes can be pinned down with one table first; every detail later is an expansion of this table:

Lexical Basics

specialChars and character classification

The PDF spec divides characters into three classes: whitespace, delimiters, and regular characters. PDF.js does not write an isXxx function for each class, nor use a regular expression; it uses a lookup table:

// A '1' in this array means the character is white space. A '1' or
// '2' means the character ends a name or command.
// prettier-ignore
const specialChars = [
  1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, // 0x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 1x
  1, 0, 0, 0, 0, 2, 0, 0, 2, 2, 0, 0, 0, 0, 0, 2, // 2x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 0, // 3x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 4x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 0, 0, // 5x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 6x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 0, 0, // 7x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 8x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 9x
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // ax
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // bx
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // cx
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // dx
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // ex
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0  // fx
];

Indexes are byte values, 256 entries total. 1 means whitespace, 2 means delimiter, 0 means regular character. Conclusions from reading row by row:

• The 0x00 row has five 1s (0x00, 0x09, 0x0a, 0x0c, 0x0d), and 0x20 in the next row is whitespace too. NUL and form feed are also treated as whitespace — a lenient policy, since the spec only requires the latter.
• 0x25 is %, 0x28/0x29 are parentheses, 0x2f is slash, 0x3c/0x3e are angle brackets, 0x5b/0x5d are square brackets, 0x7b/0x7d are curly braces. All of them are 2.
• Therefore "is it a delimiter" is specialChars[ch] === 2, "is it whitespace" is specialChars[ch] === 1, and "is it valid inside a name/command" is !specialChars[ch].

Two common misconceptions need clearing up:

• In this version of core_utils.js there is no isDelimiter; delimiter checks are handled entirely by this table. There is also no isNumber; number detection is hard-coded in the switch of Lexer.getObj and the character ranges of getNumber.
• isWhiteSpace does exist, but accepts only four bytes — stricter than the table:

// src/core/core_utils.js
function isWhiteSpace(ch) {
  return ch === 0x20 || ch === 0x09 || ch === 0x0d || ch === 0x0a;
}

It is used by findDefaultInlineStreamEnd, findASCII85DecodeInlineStreamEnd, and #findStreamLength to decide "is the byte after EI or endstream a valid whitespace boundary". The table is for the lexing loop, the function is for stream boundaries; the two sets are deliberately not the same.

Name and Cmd checks are equally simple, from primitives.js:

function isName(v, name) {
  return v instanceof Name && (name === undefined || v.name === name);
}
 
function isCmd(v, cmd) {
  return v instanceof Cmd && (cmd === undefined || v.cmd === cmd);
}

Cmd / Name / EOF singletons

The tokenizer has to return an object every time it encounters an operator. If it did new Cmd("Tj") each time, one page content stream would produce tens of thousands of short-lived objects. PDF.js's solution is three cache tables (excerpt; type assertions for the test environment omitted):

const EOF = Symbol('EOF');
 
let CmdCache = Object.create(null);
let NameCache = Object.create(null);
let RefCache = Object.create(null);
 
class Name {
  constructor(name) {
    this.name = name;
  }
 
  static get(name) {
    // eslint-disable-next-line no-restricted-syntax
    return (NameCache[name] ||= new Name(name));
  }
}
 
class Cmd {
  constructor(cmd) {
    this.cmd = cmd;
  }
 
  static get(cmd) {
    // eslint-disable-next-line no-restricted-syntax
    return (CmdCache[cmd] ||= new Cmd(cmd));
  }
}

Three points:

1. EOF uses a Symbol. Parser writes this.buf1 === EOF all over the place, and only "the same Symbol reference" satisfies it, so no valid token can be misjudged. A Symbol is also inherently unique and cannot be constructed from user data.
2. The ||= cache. Cmd.get("Tj") creates an instance the first time and returns the same one forever after. So the two instanceof checks and the string comparisons in isCmd(x, "Tj") all happen on stable small objects, and the cache does not grow — there are only a few dozen distinct operator names.
3. Same for Name. /Type can appear tens of thousands of times in a document, and Name.get("Type") creates only one instance. Dictionary keys are stored as strings, but every Name value is normalized.

Ref and Dict data structures

Ref is the triple for an indirect reference, "object number + generation". Its string form is precomputed and cached:

class Ref {
  #str;
 
  constructor(str, num, gen) {
    this.#str = str;
    this.num = num;
    this.gen = gen;
  }
 
  toString() {
    return this.#str;
  }
 
  static get(num, gen) {
    const str = gen === 0 ? `${num}R` : `${num}R${gen}`;
    return (RefCache[str] ||= new Ref(str, num, gen));
  }
}

The reason toString() returns #str directly is that it sits on the hash paths of RefSet, RefMap, and XRef, where concatenating ${num}R${gen} every time is wasteful. Ref.get caches too, so Ref.get(12, 0) has just one instance across the whole document.

Dict is essentially a wrapper around a Map, but it does three things Map does not: automatic dereferencing, short/long key names, and encryption suppression. Structure excerpt:

class Dict {
  #map = new Map();
 
  objId = null;
 
  suppressEncryption = false;
 
  get(key1, key2) {
    return this.#getValue(/* isAsync = */ false, key1, key2);
  }
 
  set(key, value) {
    this.#map.set(key, value);
  }
}

Three design points worth expanding:

• Keys are strings, without the slash. Lexer.getName returns Name("Type"), Parser takes buf1.name and then does dict.set(key, ...). Hence dict.get("Type"), not dict.get("/Type").
• get(key1, key2) supports abbreviations. The source uses dict.get("F", "Filter") and dict.get("DP", "DecodeParms") extensively: look up the short key first, fall back to the long key. In #getValue there is even an unreachable assertion requiring key2.length >= key1.length, cementing the "short first, long second" ordering.
• get dereferences automatically. When #getValue finds the value is a Ref and this.xref exists, it calls xref.fetch directly; getting the raw value requires getRaw. suppressEncryption is passed all the way down to fetch, used for reading metadata that should not be decrypted.

As for "deduplication of Name when calling dict.set", the accurate statement is: Dict.set itself does not deduplicate; it stores the string key as-is. Deduplication happens upstream in Lexer.getName — all name tokens go through Name.get, so values with the same name share an instance. The key is just an ordinary string, and strings are interned by the JS engine anyway.

The Lexer Class

Lexer starts at line 954 and has only six fields: stream, currentChar, strBuf, knownCommands, _hexStringNumWarn, beginInlineImagePos. It has no recursion at all; every method serves "read one token".

constructor

class Lexer {
  constructor(stream, knownCommands = null) {
    this.stream = stream;
    this.nextChar();
 
    // While lexing, we build up many strings one char at a time. Using += for
    // this can result in lots of garbage strings. It's better to build an
    // array of single-char strings and then join() them together at the end.
    // And reusing a single array (i.e. |this.strBuf|) over and over for this
    // purpose uses less memory than using a new array for each string.
    this.strBuf = [];
 
    // The PDFs might have "glued" commands with other commands, operands or
    // literals, e.g. "q1". The knownCommands is a dictionary of the valid
    // commands and their prefixes. The prefixes are built the following way:
    // if there a command that is a prefix of the other valid command or
    // literal (e.g. 'f' and 'false') the following prefixes must be included,
    // 'fa', 'fal', 'fals'. The prefixes are not needed, if the command has no
    // other commands or literals as a prefix. The knowCommands is optional.
    this.knownCommands = knownCommands;
 
    this._hexStringNumWarn = 0;
    this.beginInlineImagePos = -1;
  }

The three comments already explain things; two more points:

• strBuf is a reused array. Every string or name read does strBuf.length = 0 first, then pushes character by character, then join(""). This produces fewer intermediate strings than += and avoids allocating a new array per token.
• knownCommands is the cure for "glued commands". PDF allows things like q1; looking only at characters, q1 would be read as one command. Given the command table, once q is confirmed as a known command and the next character cannot extend it into another known command, it stops immediately and leaves 1 for the next round. The prefix entries in the table with null values, fa, fals, nu, nul, exist to serve false / null.
• beginInlineImagePos is initialized to -1; it records the current position only when BI is read, for inline image caching.

nextChar / peekChar

  nextChar() {
    return (this.currentChar = this.stream.getByte());
  }
 
  peekChar() {
    return this.stream.peekByte();
  }

The lexer's "single character buffer" is currentChar: nextChar advances one byte and updates the field, peekChar looks without moving. stream.getByte() returns 0–255 or -1 (end of file), so all loops are written as >= 0 or !== -1. It does not buffer a whole chunk of bytes and does no slice; each token depends only on the current and next byte.

getNumber

The difficulty in reading numbers is not integers, but that "the integer part before the dot can be omitted", "exponents do not exist", and "a minus sign may appear in odd places". getNumber uses a single divideBy variable to express both "is there a decimal point" and "how many digits follow it":

    let baseValue = ch - 0x30; // '0'
 
    while ((ch = this.nextChar()) >= 0) {
      if (ch >= /* '0' = */ 0x30 && ch <= /* '9' = */ 0x39) {
        const currentDigit = ch - 0x30; // '0'
        if (divideBy !== 0) {
          // We are after a point.
          divideBy *= 10;
        }
        baseValue = baseValue * 10 + currentDigit;
      } else if (ch === /* '.' = */ 0x2e) {
        if (divideBy === 0) {
          divideBy = 1;
        } else {
          // A number can have only one dot.
          break;
        }
      } else if (ch === /* '-' = */ 0x2d) {
        // Ignore minus signs in the middle of numbers to match
        // Adobe's behavior.
        warn("Badly formatted number: minus sign in the middle");
      } else {
        // The last character doesn't belong to us.
        break;
      }
    }
 
    if (divideBy !== 0) {
      baseValue /= divideBy;
    }
    return sign * baseValue;
  }

The values of divideBy are interesting: when the dot is read after the integer part it is set to 1, then multiplied by 10 for each digit read; if the dot comes first (.5), it is set directly to 10. The final baseValue /= divideBy performs the scaling in one step, with no floating-point accumulation error, and there is no need to maintain both "decimal place count" and "fractional part".

The start of the method also handles several legacy notations:

• --5 applies only one minus sign, giving -5 (two consecutive minus signs are ignored), matching Adobe Reader.
• + is skipped directly.
• A newline between the sign and the first digit is ignored (- newline 5 is still -5), because Adobe Reader behaves this way.
• If the first valid character is not a digit, dot, or sign, then when it is whitespace, (, <, or end of file, an info is logged and 0 is returned; otherwise a FormatError is thrown. The former lenient path corresponds to the real broken documents listed in the source comments, such as issue9252 and issue15604.

getString

PDF strings are wrapped in parentheses, may contain nested parentheses, and use backslash escapes. The escape branches are the most interesting part of the method:

            case 0x6e: // 'n'
              strBuf.push("\n");
              break;
            case 0x72: // 'r'
              strBuf.push("\r");
              break;
            case 0x74: // 't'
              strBuf.push("\t");
              break;
            case 0x62: // 'b'
              strBuf.push("\b");
              break;
            case 0x66: // 'f'
              strBuf.push("\f");
              break;
            case 0x5c: // '\'
            case 0x28: // '('
            case 0x29: // ')'
              strBuf.push(String.fromCharCode(ch));
              break;
            case 0x30: // '0'
            case 0x31: // '1'
            case 0x32: // '2'
            case 0x33: // '3'
            case 0x34: // '4'
            case 0x35: // '5'
            case 0x36: // '6'
            case 0x37: // '7'
              let x = ch & 0x0f;
              ch = this.nextChar();
              charBuffered = true;
              if (ch >= /* '0' = */ 0x30 && ch <= /* '7' = */ 0x37) {
                x = (x << 3) + (ch & 0x0f);
                ch = this.nextChar();
                if (ch >= /* '0' = */ 0x30 && ch /* '7' = */ <= 0x37) {
                  charBuffered = false;
                  x = (x << 3) + (ch & 0x0f);
                }
              }
              strBuf.push(String.fromCharCode(x));
              break;

Three easy-to-miss points:

1. charBuffered. An octal escape may already have read 1–2 extra characters; at that point ch does not belong to the next round, so a flag must prevent the outer loop from calling nextChar() again, or characters would be swallowed.
2. Line continuation. A backslash followed by CR or CRLF means "line continuation" per the PDF spec and produces no character. Inside case 0x0d it also peeks to see whether an LF follows, to avoid processing CRLF as two separate cases.
3. Nested parenthesis counting. numParen starts at 1, increments on ( and decrements on ), and only finishes when it reaches 0, consuming the closing parenthesis. So (a(b)c) is one complete string rather than being truncated at the first ).

Design note: getString returns a JS string where "each character corresponds to one raw byte". It does not do UTF-16BE BOM detection or PDFDocEncoding mapping — those belong to text semantics, not lexing. The actual decoding lives in stringToPDFString in src/core/string_utils.js, which dispatches on the three BOMs FE FF, FF FE, EF BB BF to utf-16be / utf-16le / utf-8 and falls back to the PDFDocEncoding translation table. The benefit of layering: the parsing stage never guesses encodings, and the same bytes can be interpreted differently by different semantic layers.

getName

Names start with a slash and end at any delimiter or whitespace. # escapes bytes that cannot appear directly:

  getName() {
    let ch, previousCh;
    const strBuf = this.strBuf;
    strBuf.length = 0;
 
    while ((ch = this.nextChar()) >= 0 && !specialChars[ch]) {
      if (ch === /* '#' = */ 0x23) {
        ch = this.nextChar();
        if (specialChars[ch]) {
          warn(
            "Lexer_getName: " +
              "NUMBER SIGN (#) should be followed by a hexadecimal number."
          );
          strBuf.push("#");
          break;
        }
        const x = toHexDigit(ch);
        if (x !== -1) {
          previousCh = ch;
          ch = this.nextChar();
          const x2 = toHexDigit(ch);
          if (x2 === -1) {
            warn(
              `Lexer_getName: Illegal digit (${String.fromCharCode(ch)}) ` +
                "in hexadecimal number."
            );
            strBuf.push("#", String.fromCharCode(previousCh));
            if (specialChars[ch]) {
              break;
            }
            strBuf.push(String.fromCharCode(ch));
            continue;
          }
          strBuf.push(String.fromCharCode((x << 4) | x2));
        } else {
          strBuf.push("#", String.fromCharCode(ch));
        }
      } else {
        strBuf.push(String.fromCharCode(ch));
      }
    }
    if (strBuf.length > 127) {
      warn(`Name token is longer than allowed by the spec: ${strBuf.length}`);
    }
    return Name.get(strBuf.join(""));
  }

#xx must be exactly two hex digits. When only one digit is decoded, or the second is illegal, PDF.js does not throw; it pushes # and the original character back into the result and keeps reading — to be compatible with bad names in real documents. Over 127 bytes it only warns, without truncating or throwing, because the spec limits names to 127 but longer names exist in practice.

The key part at the end is Name.get: it strings together "byte sequence → string → normalized instance", guaranteeing that /Type is the same object across the document.

toHexDigit and getHexString

Hex strings start with < and end with >. toHexDigit first converts a character to 0–15: '0'–'9' take the low four bits directly, 'A'–'F' and 'a'–'f' take the low four bits plus 9, and illegal characters return -1. No branch table and no parseInt.

  getHexString() {
    const strBuf = this.strBuf;
    strBuf.length = 0;
    let ch = this.currentChar;
    let firstDigit = -1,
      digit = -1;
    this._hexStringNumWarn = 0;
 
    while (true) {
      if (ch < 0) {
        warn("Unterminated hex string");
        break;
      } else if (ch === /* '>' = */ 0x3e) {
        this.nextChar();
        break;
      } else if (specialChars[ch] === 1) {
        ch = this.nextChar();
        continue;
      } else {
        digit = toHexDigit(ch);
        if (digit === -1) {
          this._hexStringWarn(ch);
        } else if (firstDigit === -1) {
          firstDigit = digit;
        } else {
          strBuf.push(String.fromCharCode((firstDigit << 4) | digit));
          firstDigit = -1;
        }
        ch = this.nextChar();
      }
    }
 
    // According to the PDF spec, section "7.3.4.3 Hexadecimal Strings":
    //  "If the final digit of a hexadecimal string is missing—that is, if there
    //   is an odd number of digits—the final digit shall be assumed to be 0."
    if (firstDigit !== -1) {
      strBuf.push(String.fromCharCode(firstDigit << 4));
    }
    return strBuf.join("");
  }

With an odd number of digits the low nibble is padded with 0, so <4> equals <40>. Whitespace is ignored inside hex strings (PDF allows arbitrary whitespace separators), and illegal characters are only warned about and skipped. _hexStringWarn also rate-limits: at most 5 illegal-character warnings per hex string, then silence — the comment says explicitly this is to prevent broken documents from flooding the console and slowing everything down.

getObj

getObj is the lexer's entry point: it first skips whitespace and comments, then dispatches on the first byte:

  getObj() {
    // Skip whitespace and comments.
    let comment = false;
    let ch = this.currentChar;
    while (true) {
      if (ch < 0) {
        return EOF;
      }
      if (comment) {
        if (ch === /* LF = */ 0x0a || ch === /* CR = */ 0x0d) {
          comment = false;
        }
      } else if (ch === /* '%' = */ 0x25) {
        comment = true;
      } else if (specialChars[ch] !== 1) {
        break;
      }
      ch = this.nextChar();
    }
 
    // Start reading a token.
    switch (ch | 0) {
      case 0x30: // '0'
      case 0x31: // '1'
      case 0x32: // '2'
      case 0x33: // '3'
      case 0x34: // '4'
      case 0x35: // '5'
      case 0x36: // '6'
      case 0x37: // '7'
      case 0x38: // '8'
      case 0x39: // '9'
      case 0x2b: // '+'
      case 0x2d: // '-'
      case 0x2e: // '.'
        return this.getNumber();
      case 0x28: // '('
        return this.getString();
      case 0x2f: // '/'
        return this.getName();
      // array punctuation
      case 0x5b: // '['
        this.nextChar();
        return Cmd.get("[");
      case 0x5d: // ']'
        this.nextChar();
        return Cmd.get("]");
      // hex string or dict punctuation
      case 0x3c: // '<'
        ch = this.nextChar();
        if (ch === 0x3c) {
          // dict punctuation
          this.nextChar();
          return Cmd.get("<<");
        }
        return this.getHexString();
      // dict punctuation
      case 0x3e: // '>'
        ch = this.nextChar();
        if (ch === 0x3e) {
          this.nextChar();
          return Cmd.get(">>");
        }
        return Cmd.get(">");
      case 0x7b: // '{'
        this.nextChar();
        return Cmd.get("{");
      case 0x7d: // '}'
        this.nextChar();
        return Cmd.get("}");
      case 0x29: // ')'
        // Consume the current character in order to avoid permanently hanging
        // the worker thread if `Lexer.getObj` is called from within a loop
        // containing try-catch statements, since we would otherwise attempt
        // to parse the *same* character over and over (fixes issue8061.pdf).
        this.nextChar();
        throw new FormatError(`Illegal character: ${ch}`);
    }

The output forms can be listed in a table:

case 0x29 is a defense worth mentioning separately: a closing parenthesis at the position where a new token is expected is a syntax error, but if it throws without advancing, and the caller retries in a try/catch loop, it would read the same byte forever and hang the worker thread (issue8061). So nextChar() is called before throwing.

The other half of the non-delimiter logic reads commands:

    // Start reading a command.
    let str = String.fromCharCode(ch);
    // A valid command cannot start with a non-visible ASCII character,
    // and the next character may be (the start of) a valid command.
    if (ch < 0x20 || ch > 0x7f) {
      const nextCh = this.peekChar();
      if (nextCh >= 0x20 && nextCh <= 0x7f) {
        this.nextChar();
        return Cmd.get(str);
      }
    }
    const knownCommands = this.knownCommands;
    let knownCommandFound = knownCommands?.[str] !== undefined;
    while ((ch = this.nextChar()) >= 0 && !specialChars[ch]) {
      // Stop if a known command is found and next character does not make
      // the string a command.
      const possibleCommand = str + String.fromCharCode(ch);
      if (knownCommandFound && knownCommands[possibleCommand] === undefined) {
        break;
      }
      if (str.length === 128) {
        throw new FormatError(`Command token too long: ${str.length}`);
      }
      str = possibleCommand;
      knownCommandFound = knownCommands?.[str] !== undefined;
    }
    if (str === "true") {
      return true;
    }
    if (str === "false") {
      return false;
    }
    if (str === "null") {
      return null;
    }
 
    if (str === "BI") {
      // Keep track of the current stream position, since it's needed in order
      // to correctly cache inline images; see `Parser.makeInlineImage`.
      this.beginInlineImagePos = this.stream.pos;
    }
 
    return Cmd.get(str);
  }

This section solves three real-world problems:

1. Binary garbage. If the first byte is not visible ASCII (less than 0x20 or greater than 0x7f) but the next byte is visible ASCII, it is returned as a single-character command. That way control bytes in broken documents do not pollute subsequent tokens.
2. Glued tokens. Once knownCommands matches, it stops as soon as the next character cannot extend the current string into another known command. So false is not split into f + alse.
3. Command length limit. It throws after 128 characters, preventing bad data from growing without bound.

true / false / null are downgraded to booleans and null at this stage rather than Cmd, so Parser does not need to know about them. BI, meanwhile, records beginInlineImagePos, preparing the "dictionary start" for inline image caching.

skipToNextLine

It jumps from the end of the stream keyword to after the first line of data, handling CRLF, lone LF, and lone CR line endings correctly. makeStream then uses stream.pos - 1 as the data start — at that point currentChar holds the first byte after the newline, and pos has already moved one past it.

The Parser Class

Parser is a recursive-descent parser. It does not touch bytes directly; it maintains a sliding window of two tokens.

constructor / refill / shift / tryShift

  constructor({ lexer, xref, allowStreams = false, recoveryMode = false }) {
    this.lexer = lexer;
    this.xref = xref;
    this.allowStreams = allowStreams;
    this.recoveryMode = recoveryMode;
 
    this.refill();
  }
 
  refill() {
    this.buf1 = this.lexer.getObj();
    this.buf2 = this.lexer.getObj();
  }
 
  shift() {
    if (this.buf2 instanceof Cmd && this.buf2.cmd === "ID") {
      this.buf1 = this.buf2;
      this.buf2 = null;
    } else {
      this.buf1 = this.buf2;
      this.buf2 = this.lexer.getObj();
    }
  }
 
  tryShift() {
    try {
      this.shift();
      return true;
    } catch (e) {
      if (e instanceof MissingDataException) {
        throw e;
      }
      // Upon failure, the caller should reset this.lexer.pos to a known good
      // state and call this.shift() twice to reset the buffers.
      return false;
    }
  }

Why only two tokens of lookahead? Because PDF's grammar only needs two cells:

• Array and dictionary terminators are a single Cmd; looking at buf1 is enough.
• 1 0 R needs three cells, but can be split into two steps: when buf1 = 1, look at buf1 (which becomes 0 after the move) and buf2 (R), and if it holds, call shift twice. In practice only "current + next" two cells are needed.
• Whether a dictionary is followed by stream is checked against buf2 when the dictionary is done.

shift()'s special handling of ID is easiest to miss: an inline image dictionary ends with ID, and raw binary follows. If shift called lexer.getObj() as usual, the lexer would read the image data as tokens and go completely off track. So when buf2 is ID, it is only moved to buf1 and buf2 is set to null — lookahead stops here, and control passes to makeInlineImage.

tryShift() exists for stream-length tolerance. It allows rolling back after a failed shift: the caller restores lexer.pos and calls shift twice to rebuild both buffers. MissingDataException must be rethrown as-is, because it signals "data has not finished downloading", not a syntax error.

getObj

  getObj(cipherTransform = null) {
    const buf1 = this.buf1;
    this.shift();
 
    if (buf1 instanceof Cmd) {
      switch (buf1.cmd) {
        case "BI": // inline image
          return this.makeInlineImage(cipherTransform);
        case "[": // array
          const array = [];
          while (!isCmd(this.buf1, "]") && this.buf1 !== EOF) {
            array.push(this.getObj(cipherTransform));
          }
          if (this.buf1 === EOF) {
            if (this.recoveryMode) {
              return array;
            }
            throw new ParserEOFException("End of file inside array.");
          }
          this.shift();
          return array;
        case "<<": // dictionary or stream
          const dict = new Dict(this.xref);
          while (!isCmd(this.buf1, ">>") && this.buf1 !== EOF) {
            if (!(this.buf1 instanceof Name)) {
              info("Malformed dictionary: key must be a name object");
              this.shift();
              continue;
            }
 
            const key = this.buf1.name;
            this.shift();
            if (this.buf1 === EOF) {
              break;
            }
            dict.set(key, this.getObj(cipherTransform));
          }
          if (this.buf1 === EOF) {
            if (this.recoveryMode) {
              return dict;
            }
            throw new ParserEOFException("End of file inside dictionary.");
          }
 
          // Stream objects are not allowed inside content streams or
          // object streams.
          if (isCmd(this.buf2, "stream")) {
            return this.allowStreams
              ? this.makeStream(dict, cipherTransform)
              : dict;
          }
          this.shift();
          return dict;
        default: // simple object
          return buf1;
      }
    }
 
    if (Number.isInteger(buf1)) {
      // indirect reference or integer
      if (Number.isInteger(this.buf1) && isCmd(this.buf2, "R")) {
        const ref = Ref.get(buf1, this.buf1);
        this.shift();
        this.shift();
        return ref;
      }
      return buf1;
    }
 
    if (typeof buf1 === "string") {
      return cipherTransform ? cipherTransform.decryptString(buf1) : buf1;
    }
 
    // simple object
    return buf1;
  }

The entire recursive-descent rule set is in these 76 lines:

1. Arrays: loop recursively until ]. On EOF, recoveryMode returns what has been read so far; otherwise it throws ParserEOFException. This guarantees broken documents can still be opened in "tolerant mode", while errors are not silently swallowed in normal mode.
2. Dictionaries: keys must be Name. On an illegal key it logs info, calls shift() to skip one token, and continues — not throw. This is PDF.js's consistent attitude toward broken documents: guess when you can.
3. Dictionary vs stream: after the dictionary closes, check whether buf2 is stream. Only when allowStreams is true is makeStream called; nested streams are not allowed inside content streams and ObjStm, so here only the Dict is returned and the caller handles the following tokens.
4. Ref recognition: buf1 is an integer, the moved buf1 is also an integer, and buf2 is R — the three form "object number + generation + R". Note it uses the after-move buf1, so the three-cell token is neatly decomposed by the two-cell window.
5. String decryption: only strings go through cipherTransform.decryptString. Encryption of numbers, names, and dictionaries is handled at the stream level or the XRef level.

Design note: getObj has no "token type enum + state machine"; it is "take the current token, then dispatch on its actual type". Because Lexer has already turned [, <<, >> into Cmd, Parser needs only three switch (buf1.cmd) branches; all other types fall through to instanceof Cmd, Number.isInteger, typeof === "string". One less mapping layer means one less place to be inconsistent.

obj / endobj and the division of labor with callers

Parser itself knows nothing about obj and endobj. The xref entry locates the stream at the start of 12 0 obj; XRef calls getObj() three times in a row to read the object number, generation, and the obj Cmd, and only after validation does it call getObj() a fourth time for the real object; endobj is overwritten by the next offset-based lookup. Garbage bytes are handled in two places: Lexer skips comments and illegal characters, while Parser falls back with tryShift, skips non-Name dictionary keys, and rescans bad lengths with #findStreamLength.

makeStream

The syntax of a stream object is "dictionary + stream + raw bytes + endstream". The dictionary is built by getObj; the bytes after stream must not pass through the lexer and must be sliced by length directly in Parser:

  makeStream(dict, cipherTransform) {
    const lexer = this.lexer;
    let stream = lexer.stream;
 
    // Get the stream's start position.
    lexer.skipToNextLine();
    const startPos = stream.pos - 1;
 
    // Get the length.
    let length = dict.get("Length");
    if (!Number.isInteger(length)) {
      info(`Bad length "${length && length.toString()}" in stream.`);
      length = 0;
    }
 
    // Skip over the stream data.
    stream.pos = startPos + length;
    lexer.nextChar();
 
    // Shift '>>' and check whether the new object marks the end of the stream.
    if (this.tryShift() && isCmd(this.buf2, "endstream")) {
      this.shift(); // 'stream'
    } else {
      // Bad stream length, scanning for endstream command.
      length = this.#findStreamLength(startPos);
      if (length < 0) {
        throw new FormatError("Missing endstream command.");
      }
 
      lexer.nextChar();
      this.shift();
      this.shift();
    }
    this.shift(); // 'endstream'
 
    stream = stream.makeSubStream(startPos, length, dict);
    const filter = dict.get("F", "Filter");
    // Streams that explicitly use `/Crypt` are decrypted in the filter chain,
    // so avoid applying the default stream cipher up-front.
    if (cipherTransform && !this.#hasCryptFilter(filter)) {
      stream = cipherTransform.createStream(stream, length);
    }
    stream = this.filter(stream, dict, length, cipherTransform);
    stream.dict = dict;
    return stream;
  }

The core here is "trust /Length first, scan on failure":

• Per the spec, /Length may be an indirect reference; dict.get dereferences it automatically. If it is an integer, jump to the end of the data using it.
• After jumping, lexer.nextChar() syncs the single-character buffer, and tryShift() probes "is the next token endstream". Success means the length is trustworthy, and shift() consumes it directly.
• On failure, #findStreamLength scans from startPos for the endstream byte sequence. If not found, it throws FormatError. This step is for documents whose /Length is wrong (commonly 0, or pointing to the wrong offset).
• On either path, shift() finally skips endstream, then makeSubStream(startPos, length, dict) builds a bounded view. Subsequent decryption and the filter chain operate on this view; the original stream position is unaffected.

The interplay between #hasCryptFilter and cipherTransform is also subtle: if the filter chain explicitly contains /Crypt, the default stream cipher must not be applied up front, or the data would be decrypted twice. The check is "the name equals Crypt" or "any array element equals Crypt", and array elements may be Refs.

findStreamLength (private)

  #findStreamLength(startPos) {
    const { stream } = this.lexer;
    stream.pos = startPos;
 
    const SCAN_BLOCK_LENGTH = 2048;
    const signatureLength = "endstream".length;
 
    const END_SIGNATURE = new Uint8Array([0x65, 0x6e, 0x64]);
 
    // Ideally we'd directly search for "endstream", however there are corrupt
    // PDF documents where the command is incomplete; hence we search for:
    //  1. The normal case.
    //  2. The misspelled case (fixes issue18122.pdf).
    //  3. The truncated case (fixes issue10004.pdf).
 
    while (stream.pos < stream.end) {
      const scanBytes = stream.peekBytes(SCAN_BLOCK_LENGTH);
      const scanLength = scanBytes.length - signatureLength;
 
      if (scanLength <= 0) {
        break;
      }

The scanning strategy is "block scan + prefix match":

1. peekBytes(2048) each time; the whole file is not copied. peekBytes returns the current buffer window, and after scanning only stream.pos += scanLength is needed.
2. Match the three-byte end first. On a hit, continue checking the suffix: the complete stream, the misspelled steam, and the truncated strea are all accepted; the latter two also require the first byte after the suffix to be whitespace, to avoid false hits inside image data.
3. Return stream.pos - startPos, the data length; -1 if not found.

Design note: /Length is a "claim", endstream is the "fact". When the claim is trustworthy it is a zero-cost skip; when it is not, the parser degrades to a bounded scan, and the scan only ever happens in a 2KB window. This "fast path first, slow path second, and even the slow path has an upper bound" structure is why the parser still converges on malformed documents.

filter / makeFilter

/Filter in the stream dictionary determines how the raw bytes are decoded. filter expands the filter chain, and makeFilter maps each name to the corresponding Stream subclass:

  filter(stream, dict, length, cipherTransform = null) {
    let filter = dict.get("F", "Filter");
    let params = dict.get("DP", "DecodeParms");
 
    if (filter instanceof Name) {
      if (Array.isArray(params)) {
        warn("/DecodeParms should not be an Array, when /Filter is a Name.");
      }
      return this.makeFilter(
        stream,
        filter.name,
        length,
        params,
        cipherTransform
      );
    }

• length only applies to the first layer. The raw byte length describes the data in the file; after the first decode layer the length changes, so the loop immediately sets maybeLength = null.
• DecodeParms pairs with Filter by position. The arrays need not be the same length; missing entries are treated as null; elements may be Refs, dereferenced with #fetchIfRef.

makeFilter covers every standard filter in the PDF spec:

makeFilter also applies two protections: maybeLength === 0 returns NullStream directly (the comment notes the length may be wrong, so it only dares to skip when it is definitely 0); in catch, aside from MissingDataException, which must be rethrown, all other exceptions are warned and then return NullStream, ensuring one bad stream does not fail an entire page.

Inline Images

The syntax of an inline image is BI key value ... ID raw bytes EI. It has no /Length and no separate stream object; all data is embedded in the content stream. This is the least "grammatical" part of parser.js: boundaries are guessed, and wrong guesses are caught by validation.

makeInlineImage

  makeInlineImage(cipherTransform) {
    const lexer = this.lexer;
    const stream = lexer.stream;
 
    const dict = new Dict(this.xref);
    let dictLength;
    while (!isCmd(this.buf1, "ID") && this.buf1 !== EOF) {
      if (!(this.buf1 instanceof Name)) {
        throw new FormatError("Dictionary key must be a name object");
      }
      const key = this.buf1.name;
      this.shift();
      if (this.buf1 === EOF) {
        break;
      }
      dict.set(key, this.getObj(cipherTransform));
    }
    if (lexer.beginInlineImagePos !== -1) {
      dictLength = stream.pos - lexer.beginInlineImagePos;
    }

The dictionary loop is almost the same as an ordinary dictionary; the difference is that the terminator is ID instead of >>, and an illegal key throws immediately (there is no room for recovery in an inline image, since the meaning of the following bytes is already uncertain). Once the dictionary is read, it immediately dispatches on filterName to choose a detection method:

const filter = dict.get('F', 'Filter');
let filterName;
if (filter instanceof Name) {
  filterName = filter.name;
} else if (Array.isArray(filter)) {
  const filterZero = this.#fetchIfRef(filter[0]);
  if (filterZero instanceof Name) {
    filterName = filterZero.name;
  }
}

The code only looks at the first name; for an array it takes filter[0] and allows it to be a Ref (#fetchIfRef). The reason is practical: later filters (for example FlateDecode) need to know the boundary of the compressed data before they can decode, whereas if the first layer is DCT, ASCII85, or ASCIIHex, each has a clear terminator and the end of the data can be found directly; otherwise it can only fall back to generic EI guessing.

// Parse image stream.
const startPos = stream.pos;
let length;
switch (filterName) {
  case 'DCT':
  case 'DCTDecode':
    length = this.findDCTDecodeInlineStreamEnd(stream);
    break;
  case 'A85':
  case 'ASCII85Decode':
    length = this.findASCII85DecodeInlineStreamEnd(stream);
    break;
  case 'AHx':
  case 'ASCIIHexDecode':
    length = this.findASCIIHexDecodeInlineStreamEnd(stream);
    break;
  default:
    length = this.findDefaultInlineStreamEnd(stream);
}

After finding the length, the code also tries to cache the entire inline image: the bytes from lexer.beginInlineImagePos to the end of the data serve as the key (getInlineImageCacheKey includes the original length to prevent collisions), and only images within MAX_LENGTH_TO_CACHE = 1000 bytes with a dictionary length greater than 0 are cached. On a hit, no decoding is needed: cacheEntry.reset() and synthesize an EI for Parser; on a miss, the normal path below is taken.

findDefaultInlineStreamEnd

This is the most typical "heuristic boundary". The spec only says the data is followed by EI, but the two bytes EI can easily appear inside binary data. So the state machine looks for E, then I, then requires whitespace after it:

  findDefaultInlineStreamEnd(stream) {
    const E = 0x45,
      I = 0x49,
      SPACE = 0x20,
      LF = 0xa,
      CR = 0xd,
      NUL = 0x0;
    const { knownCommands } = this.lexer,
      startPos = stream.pos,
      n = 15;
    let state = 0,
      ch,
      maybeEIPos;
    while ((ch = stream.getByte()) !== -1) {
      if (state === 0) {
        state = ch === E ? 1 : 0;
      } else if (state === 1) {
        state = ch === I ? 2 : 0;
      }

After reaching state === 2, it first checks whether whitespace follows EI, then calls peekBytes(15) to check that the following bytes are all visible ASCII:

        if (ch === SPACE || ch === LF || ch === CR) {
          maybeEIPos = stream.pos;
          // Let's check that the next `n` bytes are ASCII... just to be sure.
          const followingBytes = stream.peekBytes(n);
 
          const ii = followingBytes.length;
          if (ii === 0) {
            break; // The end of the stream was reached, nothing to check.
          }

In the ASCII check there is a special case for NUL: a single NUL is allowed, consecutive NULs are treated as binary. The comment explains why — some generators write NULs outside inline images, so forbidding them entirely would cause false negatives, but allowing them unconditionally might truncate the image.

The most critical validation is "use a temporary Lexer to try reading the following operators":

          const tmpLexer = new Lexer(
            new Stream(stream.peekBytes(5 * n)),
            knownCommands
          );
          // Reduce the number of (potential) warning messages.
          tmpLexer._hexStringWarn = () => {};
          let numArgs = 0;
 
          while (true) {
            const nextObj = tmpLexer.getObj();
 
            if (nextObj === EOF) {
              state = 0; // No valid command found, resetting the state.
              break;
            }
            if (nextObj instanceof Cmd) {
              const knownCommand = knownCommands[nextObj.cmd];
              if (!knownCommand) {
                // Not a valid command, i.e. the inline image data *itself*
                // contains an "EI" sequence. Resetting the state.
                state = 0;
                break;
              } else if (
                knownCommand.variableArgs
                  ? numArgs <= knownCommand.numArgs
                  : numArgs === knownCommand.numArgs
              ) {
                break; // Valid command found.
              }
              numArgs = 0;
              continue;
            }
            numArgs++;
          }
 
          if (state === 2) {
            break; // Finished!
          }

This step upgrades "the bytes after EI look like ASCII" to "the bytes after EI really do form a legal operator with the correct number of arguments". For example, if the image data happens to contain EI followed by x and then garbage, the temporary lexer reads an unknown command, the state resets to zero, and scanning continues. Taking 75 bytes (5 * n) instead of 15 is to accommodate multi-argument operators like cm.

If the end of the file is reached without confirmation, PDF.js warns and falls back to the position of "the last EI seen". After the endpoint is determined, a one-byte offset must also be corrected:

    let endOffset = 4;
    stream.skip(-endOffset); // Set the stream position to just before "EI".
    ch = stream.peekByte();
    stream.skip(endOffset); // ... and remember to reset the stream position.
 
    // Ensure that we don't accidentally truncate the inline image, when the
    // data is immediately followed by the "EI" marker (fixes issue10388.pdf).
    if (!isWhiteSpace(ch)) {
      endOffset--;
    }
    return stream.pos - endOffset - startPos;
  }

The dispute between 4 and 3 for endOffset is essentially "does the whitespace before EI count as image data": assume first that "whitespace + EI + whitespace" is four bytes; if the fourth byte from the end is not whitespace, then it is only "EI + whitespace", three bytes, so subtract one. This step ensures image data immediately adjacent to EI does not lose its last byte (issue10388).

Design note: the boundary of an inline image has no authoritative declaration, only three kinds of weak evidence — the terminator, the ASCII-ness of the following bytes, and the legality of the following operators. PDF.js chains the three into a progressively stricter validation chain, and keeps "the last EI seen" as a recovery point. This is the most pragmatic way to "parse something the spec does not define".

Other find* methods and inlineStreamSkipEI

DCT, ASCII85, and ASCIIHex have clear terminators, so there is no need to guess EI:

findDCTDecodeInlineStreamEnd does not only look for 0xFFD9; it also skips APPn, DQT, SOF, SOS, and other segments by JPEG segment length, avoiding mistaking an FF D9 byte inside a segment for the end. The ASCII85 implementation tolerates whitespace between ~ and >, and broken documents that write only ~ before EI.

All three specialized methods call inlineStreamSkipEI on success:

  inlineStreamSkipEI(stream) {
    const E = 0x45,
      I = 0x49;
    let state = 0,
      ch;
    while ((ch = stream.getByte()) !== -1) {
      if (state === 0) {
        state = ch === E ? 1 : 0;
      } else if (state === 1) {
        state = ch === I ? 2 : 0;
      } else if (state === 2) {
        break;
      }
    }
  }

Because the specialized methods stop after EOI / ~> / >, EI is still ahead, so the two-byte state machine must run once more to consume it before Parser can continue reading subsequent operators.

A function name that is easy to get wrong is worth clearing up here: this version of the source has no findLengthOfInt. The inline-image-related find* methods are only the four in the table above, plus inlineStreamSkipEI and #findStreamLength.

Object Streams and Reuse

Parser does not serve only ordinary indirect objects. The same class is reused in four scenarios:

Reading object streams demonstrates the generality best: XRef.fetchCompressed first decompresses the ObjStm, uses one Parser to read N groups of "object number + offset", then slices a bounded substream per object using the difference between adjacent offsets, and creates a Parser for each to call getObj(). No dedicated syntax was written for ObjStm.

Content streams are the reverse: Lexer receives EvaluatorPreprocessor.opMap, so glued commands are split correctly; allowStreams stays at its default false, and even if the stream keyword appears in a content stream it is treated as an ordinary dictionary, so following bytes are never swallowed by mistake.

The linearization dictionary is a smaller example: Linearization.create builds a Parser with xref: null, calls getObj() four times in a row to read "object number, generation, obj, dictionary", then validates with isCmd(obj3, "obj") and linDict instanceof Dict. With a null xref, Dict does not attempt to dereference, which is exactly right for the moment "there is no xref yet, only the file header can be read".

Summary

parser.js is the foundation of the entire pipeline: the xref offsets found in Article 6, the entries maintained in Article 10, and the operators executed in Article 7 all ultimately land here to turn bytes into objects.

Its place in the pipeline

Five core design principles

1. Lexing and parsing are fully separated. Lexer does not know whether parentheses are balanced; Parser does not know how many bytes a number is made of. The separation lets Lexer be reused in four places (content streams, ObjStm, the linearization dictionary, inline image validation) and Parser in four places (xref, ObjStm, content streams, linearization dictionary).
2. Two cells of lookahead are enough to express PDF syntax. buf1 / buf2 plus shift recognize 1 0 R using "the moved buf1 + buf2"; the stream after a dictionary also needs only one cell. No backtracking, no token array, constant memory.
3. Trust the declaration when it is trustworthy, probe when it is not. When /Length is correct, stream.pos = startPos + length directly; when it is wrong, degrade to a 2KB block scan for endstream. Inline images have no declaration at all, so they rely on the three-level evidence chain of EI + ASCII check + operator validation.
4. The lexing layer does no semantic decoding. getString only maps bytes one-to-one to characters; UTF-16BE BOM and PDFDocEncoding are left to stringToPDFString. Thus the same bytes can be interpreted differently by different semantic layers, and the parser never has to guess encodings.
5. Reuse instead of allocate. The three caches for Cmd, Name, and Ref, the reused strBuf, the single-character buffer, and the inline image cache all aim to compress "high-frequency short-lived objects" into "low-frequency stable objects". This directly determines PDF.js's memory curve on documents with hundreds of pages.

Series wrap-up

At this point, the main "read" side of PDF.js is complete: bytes → tokens → objects → document model → operatorList → pixels. Each layer depends only on the stable interface of the layer below, and each is interruptible and resumable. This layering lets the worker start parsing when only part of the file has been downloaded, and keeps the main thread away from raw PDF syntax entirely. Details such as Catalog's page tree and name tree, encrypted streams, and font subsetting can be broken down further, but they all hang off some node of this main pipeline and do not change the layering itself.

StreamPDF's design principle: anything that can be handled up front is never deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.