PDF.js Source Walkthrough: encodings

Introduction

The previous article covered fonts_utils.js, which turns "a character code + a name array" into a glyph id. But where the name array comes from, and which glyph name a character code corresponds to, has not been explained yet. The answer is in src/core/encodings.js (301 lines): it stores the predefined single-byte encoding tables of the PDF specification, translating character codes from 0 to 255 into glyph names.

Its place in the chain is very early: evaluator.js reads /Encoding and /Differences from the font dictionary and decides the base encoding name; getEncoding turns the name into a table; fonts.js, type1_font.js and cff_font.js then use that table to build the mapping from character code to glyph id. Without this table there would be no way to answer "which glyph does this byte in Tj draw".

In one sentence: it is the encoding table surface of the font layer and only handles the "code to glyph name" hop; it neither parses glyph outlines nor decides Unicode.

Module Overview

encodings.js has no imports and no classes; the whole file splits into five blocks:

Its place in the font chain:

// Font dictionary /Encoding, /Differences
//   → evaluator.js            parses out baseEncodingName and differences
//   → encodings.js            getEncoding(name) → 256-entry encoding table     (this article)
//   → fonts_utils.js          type1FontGlyphMapping(table + glyphNames) → glyph id
//   → fonts.js                buildToFontChar(table + glyphlist) → Unicode
//   → font_renderer.js        fetches outlines by glyph id

Version note: This article follows the version of encodings.js in the current repository. Two points differ from older material. First, this version has no PDFDocEncoding: the decoding of PDF text strings is handled by PDFStringTranslateTable in string_utils.js, and getEncoding("PDFDocEncoding") returns null. Second, the getExtendedEncoding and the "base table + extension table factory" pattern common in older versions do not exist here: all seven tables are literal 256-entry arrays, with the 0x80 to 0x9F high range written directly into the array. In addition, although MacExpertEncoding is defined in the file, it does not appear in the export block and can only be obtained through getEncoding("MacExpertEncoding").

Where the Encoding Tables Sit in the Font Chain

A character code in PDF has three different interpretation paths, and encodings.js handles only the first:

The three paths do not replace each other. A CID font can work normally without any table from encodings.js; a simple font can draw glyphs without /ToUnicode, though the text copied out may be garbled. That is the boundary of encodings.js: it only describes "which glyph name this byte corresponds to visually", not what that glyph name is ultimately pronounced as.

The Shape of the Tables: 256 Entries, Index Is the Code Point

All seven tables are ordinary JavaScript arrays with a fixed length of 256, and the index is the character code:

// prettier-ignore
const StandardEncoding = [
  "", "", "", "", "", "", "", "", "", "", "", "", "", "", "",
  // ...
  "space", "exclam", "quotedbl", "numbersign", "dollar", "percent",
  "ampersand", "quoteright", "parenleft", "parenright", "asterisk", "plus",
  "comma", "hyphen", "period", "slash", "zero", "one", "two", "three",
  // ...
  "asciicircum", "underscore", "quoteleft", "a", "b", "c", "d", "e", "f",
  "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u",
  // ...
  "exclamdown", "cent", "sterling", "fraction", "yen", "florin", "section",
  "currency", "quotesingle", "quotedblleft", "guillemotleft", "guilsinglleft",
  // ...
  "Lslash", "Oslash", "OE", "ordmasculine", "", "", "", "", "", "ae",
  "", "", "", "dotlessi", "", "", "lslash", "oslash", "oe", "germandbls", "",
  "", "", ""];

The empty string "" means the code point has no definition in that encoding. Using an empty string instead of undefined or null has two direct benefits: the array length is always 256, so any index can be read safely; and callers can uniformly test !== "" to decide "whether this code point has a glyph name", without distinguishing "the array is too short" from "the value is missing".

// prettier-ignore is not decoration. All seven tables are hand-laid data, line by line; Prettier would re-wrap them by maximum line width and destroy the readability of "code points of the same range grouped together". Turning formatting off keeps code point ranges recognisable at a glance.

By code point range, all tables share the same structure:

The Three Western Encodings

StandardEncoding, MacRomanEncoding and WinAnsiEncoding cover the same ASCII range, and their differences concentrate in a few punctuation marks and the whole high range. Look first at the high segment of WinAnsi:

  "asciitilde", "bullet", "Euro", "bullet", "quotesinglbase", "florin",
  "quotedblbase", "ellipsis", "dagger", "daggerdbl", "circumflex",
  "perthousand", "Scaron", "guilsinglleft", "OE", "bullet", "Zcaron",
  "bullet", "bullet", "quoteleft", "quoteright", "quotedblleft",
  "quotedblright", "bullet", "endash", "emdash", "tilde", "trademark",
  "scaron", "guilsinglright", "oe", "bullet", "zcaron", "Ydieresis", "space",

This segment starts at 0x7E. 0x80 is Euro, 0x82 is quotesinglbase, 0x85 is ellipsis, and 0x91 to 0x94 are curly quotes. The repeated bullet in the middle is not a real glyph but a placeholder for undefined positions in the Windows code page; PDF.js copies this mapping as is in order to stay consistent with PDFs generated by Acrobat and Word.

MacRoman puts accented letters in the same range:

  "", "Adieresis", "Aring", "Ccedilla", "Eacute", "Ntilde", "Odieresis",
  "Udieresis", "aacute", "agrave", "acircumflex", "adieresis", "atilde",
  "aring", "ccedilla", "eacute", "egrave", "ecircumflex", "edieresis",
  // ...
  "Omega", "ae", "oslash", "questiondown", "exclamdown", "logicalnot",
  "radical", "florin", "approxequal", "Delta", "guillemotleft",
  "guillemotright", "ellipsis", "space", "Agrave", "Atilde", "Otilde", "OE",
  // ...
  "Icircumflex", "Idieresis", "Igrave", "Oacute", "Ocircumflex", "apple",
  "Ograve", "Uacute", "Ucircumflex", "Ugrave", "dotlessi", "circumflex",

Note the space at 0xCA and the apple at 0xF0: MacRoman uses space for a non-breaking space, and apple is the Apple logo glyph. These names are meaningful only in Mac fonts and usually fall back to .notdef on other fonts.

Comparing the three tables on the same set of code points makes the differences clearer:

This table explains a common question: why text exported from Word can turn into curly quotes when copied out of PDF.js. Because WinAnsi defines 0x91 to 0x94 as typographic quotes, while StandardEncoding leaves the same positions undefined; pick the wrong encoding and the names differ, and the final Unicode differs too.

Design note: The encoding tables only describe "code to glyph name", not "name to Unicode". Unicode resolution is left to glyphlist.js, so a single table can serve both glyph positioning (name to glyph id) and text extraction (name to Unicode) without maintaining two copies of the data for two paths. The price is that names must be standard enough; non-standard names such as uniXXXX still need recoverGlyphName as a safety net.

Symbol and Expert Encodings

The last four tables serve special fonts, and their naming style is clearly different from the Western tables. The Symbol font uses Greek letters and mathematical symbols:

  "space", "exclam", "universal", "numbersign", "existential", "percent",
  "ampersand", "suchthat", "parenleft", "parenright", "asteriskmath", "plus",
  "comma", "minus", "period", "slash", "zero", "one", "two", "three", "four",
  // ...
  "congruent", "Alpha", "Beta", "Chi", "Delta", "Epsilon", "Phi", "Gamma",
  "Eta", "Iota", "theta1", "Kappa", "Lambda", "Mu", "Nu", "Omicron", "Pi",
  "Theta", "Rho", "Sigma", "Tau", "Upsilon", "sigma1", "Omega", "Xi", "Psi",
  "Zeta", "bracketleft", "therefore", "bracketright", "perpendicular",
  // ...
  "Euro", "Upsilon1", "minute", "lessequal", "fraction", "infinity", "florin",
  "club", "diamond", "heart", "spade", "arrowboth", "arrowleft", "arrowup",
  "arrowright", "arrowdown", "degree", "plusminus", "second", "greaterequal",

universal, existential, suchthat and therefore are the standard glyph names of mathematical logic symbols; the trailing 1 in theta1, sigma1 and Upsilon1 means "another glyph of the same letter", not the digit 1. club, diamond, heart and spade are playing-card suits, which shows that this table covers more than mathematics.

fonts.js also uses it to construct the glyph id encoding of Symbol fonts in reverse:

const getSymbolGlyphIdEncoding = getLookupTableFactory(t => {
  let glyphId = 3;
  for (const [firstCharCode, lastCharCode] of [
    [0x20, 0x7e],
    [0xa1, 0xfe],
  ]) {
    for (let charCode = firstCharCode; charCode <= lastCharCode; charCode++) {
      t[glyphId++] = SymbolSetEncoding[charCode];
    }
  }
}, /* useArray = */ true);

This code walks 0x20 to 0x7E and 0xA1 to 0xFE in order and writes the glyph name of each code point to an increasing glyphId. It exploits the fact that glyphs in a Symbol font are ordered by code point, while skipping the gap from 0x7F to 0xA0 and glyph ids 0 to 2 for .notdef, .null and nonmarkingreturn. Here the same table is used as a "code point order", and the index is no longer a glyph id but maintained by an external counter.

ZapfDingbats is more radical: the glyph names are Adobe numbers with no semantics:

  "space", "a1", "a2", "a202", "a3", "a4", "a5", "a119", "a118", "a117",
  "a11", "a12", "a13", "a14", "a15", "a16", "a105", "a17", "a18", "a19",
  // ...
  "a57", "a58", "a59", "a60", "a61", "a62", "a63", "a64", "a65", "a66",
  "a67", "a68", "a69", "a70", "a71", "a72", "a73", "a74", "a203", "a75",

The numbers are not in increasing code point order: a202 appears after a2, and a203 sits between a74 and a75. These numbers must be kept as they are, because the post table and the built-in encoding of the ZapfDingbats font are aligned by them.

These aNN names are also not in the ordinary glyph table. glyphlist.js exports getDingbatsGlyphsUnicode separately for them, alongside getGlyphsUnicode; fonts.js uses the former when building the Unicode mapping of ZapfDingbats. The encoding table and the glyph table must be used as a pair; using the wrong table makes a1 resolve to an empty value and text extraction then loses the character.

The Expert family targets professional typesetting and holds small caps, old-style figures, super- and subscripts, and ligatures:

  "space", "exclamsmall", "Hungarumlautsmall", "", "dollaroldstyle",
  "dollarsuperior", "ampersandsmall", "Acutesmall", "parenleftsuperior",
  "parenrightsuperior", "twodotenleader", "onedotenleader", "comma",
  "hyphen", "period", "fraction", "zerooldstyle", "oneoldstyle",
  // ...
  "Asmall", "Bsmall", "Csmall", "Dsmall", "Esmall", "Fsmall", "Gsmall",
  "Hsmall", "Ismall", "Jsmall", "Ksmall", "Lsmall", "Msmall", "Nsmall",
  // ...
  "questiondownsmall", "oneeighth", "threeeighths", "fiveeighths",
  "seveneighths", "onethird", "twothirds", "", "", "zerosuperior",

ExpertEncoding and MacExpertEncoding cover similar content with different code point arrangements. MacExpertEncoding has centoldstyle at 0x23, while ExpertEncoding has an empty string at the same position:

  "space", "exclamsmall", "Hungarumlautsmall", "centoldstyle",
  "dollaroldstyle", "dollarsuperior", "ampersandsmall", "Acutesmall",
  "parenleftsuperior", "parenrightsuperior", "twodotenleader",
  "onedotenleader", "comma", "hyphen", "period", "fraction", "zerooldstyle",

Names such as Ethsmall also land on different code points in the two tables. Neither table can be guessed; you have to look them up by name.

Where Encoding Names Come From: The Three Forms of /Encoding

The encoding tables themselves never appear in a PDF. What appears in a PDF is the /Encoding entry, which may be a name or a dictionary. evaluator.js is responsible for converging these forms into "base encoding name + differences array":

if (dict.has('Encoding')) {
  encoding = dict.get('Encoding');
  if (encoding instanceof Dict) {
    baseEncodingName = encoding.get('BaseEncoding');
    baseEncodingName = baseEncodingName instanceof Name ? baseEncodingName.name : null;
    // Load the differences between the base and original
    if (encoding.has('Differences')) {
      const diffEncoding = encoding.get('Differences');
      let index = 0;
      for (const entry of diffEncoding) {
        const data = xref.fetchIfRef(entry);
        if (typeof data === 'number') {
          index = data;
        } else if (data instanceof Name) {
          differences[index++] = data.name;
        } else {
          throw new FormatError(`Invalid entry in 'Differences' array: ${data}`);
        }
      }
    }
  } else if (encoding instanceof Name) {
    baseEncodingName = encoding.name;
  } else {
    // ...
  }
  // According to table 114 if the encoding is a named encoding it must be
  // one of these predefined encodings.
  if (
    baseEncodingName !== 'MacRomanEncoding' &&
    baseEncodingName !== 'MacExpertEncoding' &&
    baseEncodingName !== 'WinAnsiEncoding'
  ) {
    baseEncodingName = null;
  }
}

The /Differences parsing uses the sparse-array style of "a number resets the index, a name is written and increments it": a numeric entry in the array means the code point where the next name goes, and after a name is written the index goes up by one. So 120 /a /b means a at 120 and b at 121, without filling all 256 entries in between.

The last whitelist is the easiest part to overlook: even if /Encoding is a legal name such as /StandardEncoding, as long as it is not one of MacRomanEncoding, MacExpertEncoding and WinAnsiEncoding, baseEncodingName is also set to null. In the end there are only four forms:

This also explains why getEncoding needs seven branches while evaluator.js in practice only passes three names: the other four tables are referenced directly by the font program parsers (type1_parser.js, cff_parser.js), or taken by fonts.js in the TrueType branch according to an explicit whitelist. The dispatch function serves all callers, and the whitelist constrains only one of the paths.

getEncoding: Dispatching Seven Names

The seven tables are exposed to the outside through a very short switch:

function getEncoding(encodingName) {
  switch (encodingName) {
    case 'WinAnsiEncoding':
      return WinAnsiEncoding;
    case 'StandardEncoding':
      return StandardEncoding;
    case 'MacRomanEncoding':
      return MacRomanEncoding;
    case 'SymbolSetEncoding':
      return SymbolSetEncoding;
    case 'ZapfDingbatsEncoding':
      return ZapfDingbatsEncoding;
    case 'ExpertEncoding':
      return ExpertEncoding;
    case 'MacExpertEncoding':
      return MacExpertEncoding;
    default:
      return null;
  }
}

Its contract has only two rules: exact name matching, and null for unknown names. It does no case folding, no aliases and no prefix matching. These names are fixed in the PDF specification; a typo is a typo, and returning null is safer than guessing an approximation.

How callers handle null is not uniform, and this is worth noting when reading the source:

The exported surface is smaller than the defined surface, which is the detail of this table most easily overlooked:

When cff_parser.js needs ExpertEncoding it imports it directly; no file imports MacExpertEncoding, which exists only as one branch of the switch. This asymmetry is not an oversight: the predefined charsets of CFF use only Expert and Standard, and MacExpert is obtained through getEncoding only when a font dictionary declares it; a smaller export surface keeps dependencies clearer.

Consumers: Default Encoding, Built-in Encoding and ToUnicode

evaluator.js turns the encoding name into a property and leaves the real choice to the glyph-mapping stage:

    if (baseEncodingName) {
      properties.defaultEncoding = getEncoding(baseEncodingName);
    } else {
      let isSymbolicFont = !!(properties.flags & FontFlags.Symbolic);
      const isNonsymbolicFont = !!(properties.flags & FontFlags.Nonsymbolic);
      // ...
      // According to "Table 114" in section "9.6.6.1 General" (under
      // "9.6.6 Character Encoding") of the PDF specification, a Nonsymbolic
      // font should use the `StandardEncoding` if no encoding is specified.
      encoding = StandardEncoding;

This code corresponds to a default rule in the specification: a non-symbolic font without /Encoding uses StandardEncoding. properties.defaultEncoding is only a "candidate base"; later type1FontGlyphMapping combines it with /Differences and the font program's built-in encoding to decide the final mapping.

fonts.js, in turn, converts the encoding table into a Unicode table for text extraction:

function buildToFontChar(encoding, glyphsUnicodeMap, differences) {
  const toFontChar = [];
  let unicode;
  for (let i = 0, ii = encoding.length; i < ii; i++) {
    unicode = getUnicodeForGlyph(encoding[i], glyphsUnicodeMap);
    if (unicode !== -1) {
      toFontChar[i] = unicode;
    }
  }
  for (const charCode in differences) {
    unicode = getUnicodeForGlyph(differences[charCode], glyphsUnicodeMap);
    if (unicode !== -1) {
      toFontChar[+charCode] = unicode;
    }
  }
  return toFontChar;
}

The same encoding array is passed to glyphNames.indexOf in type1FontGlyphMapping and to a glyphsUnicodeMap lookup here. The two consumers share the data but not the result: the former gets glyph ids, the latter gets Unicode.

The predefined charsets of CFF also use these two tables directly, but in the opposite direction:

if (pos === 0 || pos === 1) {
  predefined = true;
  format = pos;
  const baseEncoding = pos ? ExpertEncoding : StandardEncoding;
  for (i = 0, ii = charset.length; i < ii; i++) {
    const index = baseEncoding.indexOf(charset[i]);
    if (index !== -1) {
      encoding[index] = i;
    }
  }
}

The CFF charset is "an array of names ordered by glyph id"; here indexOf is used to look the code point up in reverse, and that code point is then mapped to the current glyph id. In other words, the same table can be used forwards (code point to name) and backwards (name to code point).

Text extraction and glyph positioning are two parallel chains and must not be merged into one priority:

When /ToUnicode exists, text extraction uses it directly and encodings.js plays no part at all; only when there is no /ToUnicode does the encoding table turn code points into glyph names, which are then looked up in glyphlist for Unicode. This is why removing /ToUnicode does not affect display but garbles the copied text.

Design note: The encoding tables deliberately contain no "code to Unicode" correspondence. Unicode is a variable interpretation result; the same glyph may be given different code points by the ToUnicode of different fonts. The glyph name is the stable intermediate layer. Extracting this layer on its own lets ToUnicode override it without modifying any predefined table.

Conclusion

encodings.js is the first-hop translation table of the font layer: bytes first become glyph names, and only then can they become glyph ids or Unicode. It does not parse font programs and does not take part in drawing, yet it decides "what a given byte should actually look like".

Where It Sits in the Overall Chain

Chained in one sentence: evaluator picks the encoding name, encodings.js turns the name into a table, fonts.js turns the table into glyph ids or Unicode, and the drawing layer only knows glyph ids.

Five Design Principles

1. Separate data from logic: of the 301 lines, the vast majority are array literals and the only function is getEncoding. The tables can be checked against the specification byte by byte, the logic is one switch, and there are no hidden branches.
2. The index is the code point: no Map, no string-key lookup, one addressing operation table[charCode] does the job. The value range of a single-byte encoding is fixed, and an array is the container that fits best.
3. The empty string is a legal placeholder: undefined code points are written as "" and the array length is always 256. Callers test !== "" for the presence of a name and never have to handle out-of-bounds access.
4. The name is the only external contract: getEncoding accepts only seven exact names and returns null for unknown input. A mistyped encoding name does not silently hit an approximate table but makes the upper layer take an explicit default fallback.
5. One layer does one thing: this file does not parse CMaps, does not do Unicode mapping and does not decide the priority of /Differences. The encoding table only answers "which glyph name this byte corresponds to"; every other decision is left to fonts_utils.js and fonts.js.

With encodings.js understood, going back to type1FontGlyphMapping in fonts_utils.js lets you connect the three sources "base encoding + /Differences + built-in encoding"; looking further up at buildToFontChar in fonts.js also makes it clear why the same code point gets different Unicode values in different fonts. The next article can continue deeper into the font formats, looking at how a Type1 charstring is interpreted into outlines.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.