PDF.js Source Walkthrough: binary_cmap

Introduction

The previous article covered cmap.js: a text CMap describes its mapping with operators such as begincodespacerange, begincidrange and beginbfrange, and CMap expands them into arrays indexed by character code. But the built-in CMaps that ship with PDF.js are not those text files; they are the .bcmap binary packages under external/bcmaps.

Why is there a binary version? The built-in CMaps cover Adobe-GB1, Adobe-CNS1, Adobe-Japan1, Adobe-Korea1 and various EUC, RKSJ, UTF-16 and UTF-32 encodings, with names running into the hundreds, and a text mapping is easily hundreds of thousands of lines long. If every PDF opening had to download and parse a text CMap, both the network and lexing costs would be high. src/core/binary_cmap.js is the reader for this batch of .bcmap files: it uses big-endian bytes, varints and delta encoding to compress the same mapping into a smaller byte stream, then restores it into a CMap in a way that is close to "reading a linear array".

In one sentence: binary_cmap.js is the built-in data decoder behind CMapFactory, translating the binary records of a .bcmap into the codespaceRanges and _map of the previous article.

Module Overview

binary_cmap.js is 323 lines and splits into four blocks:

Its place in the overall chain can be drawn like this:

// CMapFactory.create({ encoding: Name })
//   → createBuiltInCMap(name)
//   → fetchBuiltInCMap(name)           returns { cMapData, isCompressed }
//   → new BinaryCMapReader().process   ← this article
//   → CMap.addCodespaceRange / mapOne / mapCidRange / mapBfRange
//   → evaluator: CID fonts or ToUnicode

Compared with parsing a text CMap, there is no Lexer, no Parser and no operator dispatch here. The input is one contiguous Uint8Array, every record header is a single byte, and every number is either a base-128 varint or fixed-length big-endian bytes. The parser only has to take fields by bit position and loop by type, without building intermediate tokens.

Why a Binary CMap Is Faster

Start by putting the two forms side by side:

The difference is not only "smaller size" but also a shorter parse path:

1. No lexing. Every hex string in a text CMap has to be scanned by Lexer, decoded and turned into a string; the binary reader calls getByte() and assembles numbers bit by bit.
2. Ranges are expressed as deltas. When a begincidrange block contains hundreds of items, the text form repeats low high dst; the binary form writes the full value only for the first item, and later items write the start delta relative to the previous end plus the end delta. Those numbers are usually small, so a varint takes only one or two bytes.
3. Reused fixed buffers. start, end, char, charCode and tmp are all 16-byte Uint8Arrays allocated once at the start of process and overwritten inside the loop, producing no per-item temporary objects.
4. Cheaper network transfer. Built-in CMaps are static resources, and a compressed .bcmap is an order of magnitude smaller than the text form; together with builtInCMapCache, the same data is fetched only once per session.

There is a comment in fetchBuiltInCMap in evaluator.js: "Only compressed CMaps are (currently) supported here." When useWorkerFetch is on, the worker fetches data only along the .bcmap path and sets isCompressed to true. In other words, built-in CMaps almost always take the binary path of this article in the default deployment; the text path is mainly left for embedded streams or an explicit .cmap configuration.

BinaryCMapStream: Six Read Primitives

BinaryCMapStream extends Stream from stream.js and adds only one reused buffer and six read methods:

const MAX_NUM_SIZE = 16;
const MAX_ENCODED_NUM_SIZE = 19; // ceil(MAX_NUM_SIZE * 7 / 8)
 
class BinaryCMapStream extends Stream {
  tmpBuf = new Uint8Array(MAX_ENCODED_NUM_SIZE);
 
  constructor(data) {
    super(
      /* arrayBuffer = */ data,
      /* start = */ 0,
      /* length = */ data.length,
      /* dict = */ null
    );
  }

MAX_NUM_SIZE means a fixed-length number occupies at most 16 bytes, corresponding to 128 bits; MAX_ENCODED_NUM_SIZE is 19 bytes, because each original byte can encode into at most 8 groups of 7 bits, and rounding up needs 19 bytes of varint space. tmpBuf is only a transit buffer for readHexNumber and readHexSigned, so a new array is not created on every call.

readNumber: base-128 varint

  readNumber() {
    let n = 0;
    let last;
    do {
      const b = this.getByte();
      if (b < 0) {
        throw new FormatError("unexpected EOF in bcmap");
      }
      last = !(b & 0x80);
      n = (n << 7) | (b & 0x7f);
    } while (!last);
    return n;
  }

The highest bit of each byte is the "continue" flag and the low 7 bits are data. Reading ends when a byte with a clear highest bit is reached, and the accumulated n is shifted left by 7 bits before the new 7 bits are added. Small numbers therefore take one byte and large numbers grow as needed; 0x4207 is split into 0x81, 0x84 and 0x07 and reassembled on read.

b < 0 is the value getByte() returns at the end of the stream. A .bcmap is a static resource, so truncation usually means a damaged file or an incomplete download, and the source throws FormatError directly rather than quietly padding with zeros.

readSigned: Zigzag Encoding

  readSigned() {
    const n = this.readNumber();
    return n & 1 ? ~(n >>> 1) : n >>> 1;
  }

Signed numbers use zigzag encoding: a non-negative number is written as 2 * n and a negative number as -2 * n - 1. Decoding looks at the lowest bit: a low bit of 1 means negative and is restored with ~(n >>> 1); a low bit of 0 means non-negative and is shifted right by one directly. The mapping is:

This way both small positive and small negative numbers need only a tiny varint, which suits the deltas of CIDs and target strings.

readHex: Fixed-Length Raw Bytes

  readHex(num, size) {
    num.set(this.getBytes(size + 1));
  }

size is "byte count minus one", so size + 1 bytes are actually read and written straight into the Uint8Array supplied by the caller. It performs no numeric conversion and preserves the raw big-endian order, which hexToInt or hexToStr interprets later.

readHexNumber: A varint Written into a Fixed-Length Buffer

  readHexNumber(num, size) {
    let last;
    const stack = this.tmpBuf;
    let sp = 0;
    do {
      const b = this.getByte();
      if (b < 0) {
        throw new FormatError("unexpected EOF in bcmap");
      }
      last = !(b & 0x80);
      stack[sp++] = b & 0x7f;
    } while (!last);
    let i = size,
      buffer = 0,
      bufferSize = 0;
    while (i >= 0) {
      while (bufferSize < 8 && stack.length > 0) {
        buffer |= stack[--sp] << bufferSize;
        bufferSize += 7;
      }
      num[i] = buffer & 255;
      i--;
      buffer >>= 8;
      bufferSize -= 8;
    }
  }

It reads a varint, but the result is not a JavaScript number; it is a fixed-length big-endian byte array. The process has two stages:

1. Push the low 7 bits of every byte into tmpBuf, with the high-order group first.
2. Fill num from back to front: pop 7 bits from the top of the stack each time, pack them into buffer, and write one byte as soon as 8 bits are complete. Because num is also written back to front, the result is the big-endian representation.

There is one easily misread detail in this code: the inner while condition is written as stack.length > 0 rather than sp > 0. stack is the 19-byte reused Uint8Array, so the condition is almost always true; after sp reaches zero, stack[--sp] still executes, at which point stack[-1] returns undefined and the shift operation treats it as 0. The loop therefore keeps filling high bits with zeros until bufferSize reaches 8. The width of a fixed-length number is usually greater than the actual byte count of the varint, and this "out of range" access happens to supply the leading zeros, which is also why it can fill num completely from back to front.

readHexSigned: Fixed-Length Signed Numbers

  readHexSigned(num, size) {
    this.readHexNumber(num, size);
    const sign = num[size] & 1 ? 255 : 0;
    let c = 0;
    for (let i = 0; i <= size; i++) {
      c = ((c & 1) << 8) | num[i];
      num[i] = (c >> 1) ^ sign;
    }
  }

Fixed-length signed numbers also use zigzag: the lowest bit is the sign bit. sign is 255 or 0, and a negative number means the result must be bitwise inverted. The loop starts at the highest byte; c = ((c & 1) << 8) | num[i] carries the lowest bit of the previous byte into the current byte, then shifts the whole thing right by one, which is equivalent to "divide the entire big-endian number by 2 while keeping bits across byte boundaries". Finally the xor with sign completes the two's-complement restoration of a negative number.

readString serves metadata records: it reads a varint length first, then reads UTF-16 code units one by one. It only serves comments and usecmap names and never takes part in the mapping table, so the implementation is straightforward.

The purposes of the six primitives can be compared side by side:

Design note: Every "fixed-length hex number" goes uniformly through Uint8Array rather than a 32-bit JavaScript number. CMap source codes can be up to 4 bytes and target strings can be longer; using fixed-length byte arrays avoids 32-bit overflow and lets carry operations such as addHex and incHex be implemented byte by byte. Only values that certainly cannot exceed 32 bits, such as CIDs and record counts, go through readNumber.

File Format: Header Byte and Record Headers

process reads the first byte first:

  async process(data, cMap, extend) {
    const stream = new BinaryCMapStream(data);
    const header = stream.getByte();
    cMap.vertical = !!(header & 1);

The lowest bit of the header byte is WMode: 1 means vertical writing. The compression tool writes the header byte as (inputData.type << 1) | inputData.wmode, so bit 1 and bit 2 encode the CMap type. The reader does not use those two bits: it decides whether to write an integer CID or a string target purely by record type (cidchar, bfchar and so on). In other words, the same reader can decode both a CID-type .bcmap and a ToUnicode-type .bcmap; the type is determined by the file contents and by the name the caller chose, and the reader does not need to judge it again.

After that comes a run of records, each starting with at least one header byte:

    let b;
    while ((b = stream.getByte()) >= 0) {
      const type = b >> 5;
      if (type === 7) {
        // metadata, e.g. comment or usecmap
        switch (b & 0x1f) {
          case 0:
            stream.readString(); // skipping comment
            break;
          case 1:
            useCMap = stream.readString();
            break;
        }
        continue;
      }
      const sequence = !!(b & 0x10);
      const dataSize = b & 15;
 
      if (dataSize + 1 > MAX_NUM_SIZE) {
        throw new Error("BinaryCMapReader.process: Invalid dataSize.");
      }
 
      const ucs2DataSize = 1;
      const subitemsCount = stream.readNumber();

A record header byte is split into three fields:

Records of type 7 use the low 5 bits to distinguish metadata: 0 is a comment and 1 is usecmap. A comment is read and discarded; usecmap is stored in a local variable and handled after all records have been read. A data record then reads a "subitem count" with readNumber and enters the matching branch by type.

Packing type, sequence and dataSize into one byte is the most central compression trick of this format: a record usually needs only one header byte plus a few varints, whereas the text form has to write a whole line of keywords and hex strings.

The semantics of the six data records are as follows:

Six Kinds of Data Records

codespacerange: Full Values First, Deltas After

        case 0: // codespacerange
          stream.readHex(start, dataSize);
          stream.readHexNumber(end, dataSize);
          addHex(end, start, dataSize);
          cMap.addCodespaceRange(
            dataSize + 1,
            hexToInt(start, dataSize),
            hexToInt(end, dataSize)
          );
          for (let i = 1; i < subitemsCount; i++) {
            incHex(end, dataSize);
            stream.readHexNumber(start, dataSize);
            addHex(start, end, dataSize);
            stream.readHexNumber(end, dataSize);
            addHex(end, start, dataSize);
            cMap.addCodespaceRange(
              dataSize + 1,
              hexToInt(start, dataSize),
              hexToInt(end, dataSize)
            );
          }
          break;

The first item reads the raw bytes of start directly, while end is read as a delta relative to start, so the real end value only appears after addHex(end, start, dataSize). Later items first increment end to become the new starting point, then read startDelta and endDelta, and two addHex calls produce start and end respectively.

dataSize + 1 is the code length. A .bcmap expresses 1 to 16 bytes with a single 4-bit field in the record header, which exactly covers the maximum 4-byte code length of PDF CMaps and also accommodates wider target strings.

notdefrange: Read and Discard

Type 1 has the same structure as codespace, except that each item carries an extra code. The reader reads start, end and code, then writes nothing at all. notdefrange describes "these character codes are undefined, use the notdef glyph". The glyph layer of PDF.js does not depend on this table to decide that a character is missing; a missing character simply falls back when _map has no entry. So the only thing needed here is to consume the bytes and keep later records aligned.

cidchar: Single-Point CID

        case 2: // cidchar
          stream.readHex(char, dataSize);
          code = stream.readNumber();
          cMap.mapOne(hexToInt(char, dataSize), code);
          for (let i = 1; i < subitemsCount; i++) {
            incHex(char, dataSize);
            if (!sequence) {
              stream.readHexNumber(tmp, dataSize);
              addHex(char, tmp, dataSize);
            }
            code = stream.readSigned() + (code + 1);
            cMap.mapOne(hexToInt(char, dataSize), code);
          }
          break;

The first item is a pair of character code and CID. Later items call incHex(char) first; if sequence is 1 the character codes simply increase consecutively and no delta needs to be read, while if it is 0 a charDelta is read and added to the current value. The CID delta is always readSigned(), based on the previous CID plus one, hence code = delta + (code + 1).

sequence is a product of the compression tool optimize.mjs: when the character codes inside one record block are strictly consecutive, all deltas are dropped and only a flag bit remains.

cidrange: Range CID

        case 3: // cidrange
          stream.readHex(start, dataSize);
          stream.readHexNumber(end, dataSize);
          addHex(end, start, dataSize);
          code = stream.readNumber();
          cMap.mapCidRange(
            hexToInt(start, dataSize),
            hexToInt(end, dataSize),
            code
          );
          for (let i = 1; i < subitemsCount; i++) {
            incHex(end, dataSize);
            if (!sequence) {
              stream.readHexNumber(start, dataSize);
              addHex(start, end, dataSize);
            } else {
              start.set(end);
            }
            stream.readHexNumber(end, dataSize);
            addHex(end, start, dataSize);
            code = stream.readNumber();
            cMap.mapCidRange(
              hexToInt(start, dataSize),
              hexToInt(end, dataSize),
              code
            );
          }
          break;

The first item corresponds exactly to begincidrange in a text CMap: start, end and the starting CID. Later items are handled in two ways:

• sequence is 1: start.set(end), that is the new start is the previous end after incrementing.
• sequence is 0: another startDelta is read and start is set to "the previous end plus one" plus the delta.

end is always computed from endDelta relative to start. Finally mapCidRange is called to expand the whole range into an item-by-item array; this step uses the same method as a text CMap, so downstream receives exactly the same CMap shape.

bfchar: Single-Point Unicode

        case 4: // bfchar
          stream.readHex(char, ucs2DataSize);
          stream.readHex(charCode, dataSize);
          cMap.mapOne(
            hexToInt(char, ucs2DataSize),
            hexToStr(charCode, dataSize)
          );
          for (let i = 1; i < subitemsCount; i++) {
            incHex(char, ucs2DataSize);
            if (!sequence) {
              stream.readHexNumber(tmp, ucs2DataSize);
              addHex(char, tmp, ucs2DataSize);
            }
            incHex(charCode, dataSize);
            stream.readHexSigned(tmp, dataSize);
            addHex(charCode, tmp, dataSize);
            cMap.mapOne(
              hexToInt(char, ucs2DataSize),
              hexToStr(charCode, dataSize)
            );
          }
          break;

There are two widths here: ucs2DataSize is fixed at 1, meaning a 2-byte source code; the target string uses the dataSize from the record header, which may be 2 bytes, 4 bytes or longer. The first item reads the raw bytes of the source and the target string; later items apply the same delta handling to the source as cidchar, and to the target string they first call incHex and then add the delta from readHexSigned.

hexToStr turns a fixed-length byte array into a string with one character per byte, which exactly matches how bf targets are represented inside CMap. The source special-cases the two most common UTF-16 target string lengths, 2 and 4 bytes, building the string by index directly to avoid subarray() creating a new view object; the comment "This code is hot" shows that this path is called over and over again in CJK documents.

bfrange: Range Unicode

        case 5: // bfrange
          stream.readHex(start, ucs2DataSize);
          stream.readHexNumber(end, ucs2DataSize);
          addHex(end, start, ucs2DataSize);
          stream.readHex(charCode, dataSize);
          cMap.mapBfRange(
            hexToInt(start, ucs2DataSize),
            hexToInt(end, ucs2DataSize),
            hexToStr(charCode, dataSize)
          );
          for (let i = 1; i < subitemsCount; i++) {
            incHex(end, ucs2DataSize);
            if (!sequence) {
              stream.readHexNumber(start, ucs2DataSize);
              addHex(start, end, ucs2DataSize);
            } else {
              start.set(end);
            }
            stream.readHexNumber(end, ucs2DataSize);
            addHex(end, start, ucs2DataSize);
            stream.readHex(charCode, dataSize);
            cMap.mapBfRange(
              hexToInt(start, ucs2DataSize),
              hexToInt(end, ucs2DataSize),
              hexToStr(charCode, dataSize)
            );
          }
          break;

The structure is almost identical to cidrange, except that the source range is fixed at 2 bytes and the target changes from "starting CID" to "starting string". mapBfRange is responsible for incrementing the starting string by its last byte and filling _map item by item; the binary format only has to deliver the starting string and the range, and the expansion logic is shared completely with text CMaps.

Looking at these case blocks together, they all do the same thing: read the start, read the end, read the target, write into CMap. The first item is read in full with readHex, later items restore deltas with incHex and addHex; the target is either a varint or fixed-length bytes; finally everything goes to mapOne, mapCidRange or mapBfRange. Not one case in the whole reader needs backtracking, and there is no nested grammar structure.

Design note: The binary format is not "a text CMap translated into bytes line by line" but records regrouped into blocks and differenced. Inside one block, the start, end and target are all encoded relative to the previous item; consecutive intervals drop every delta through the sequence bit; small numbers use varints and large numbers use fixed-length big-endian bytes. The reader loop is therefore very regular: every case is "read the first item, loop over later items, call the same write method", with no nested grammar analysis and no backtracking.

useCMap and How It Joins the Factory

usecmap is a metadata record in the binary form, not part of a data block:

      if (type === 7) {
        // metadata, e.g. comment or usecmap
        switch (b & 0x1f) {
          case 0:
            stream.readString(); // skipping comment
            break;
          case 1:
            useCMap = stream.readString();
            break;
        }
        continue;
      }

The reader only stores the name in useCMap and calls back after all records have been processed:

    if (useCMap) {
      return extend(useCMap);
    }
    return cMap;
  }

extend is not logic implemented by BinaryCMapReader itself; it is the closure passed in by createBuiltInCMap:

if (isCompressed) {
  return new BinaryCMapReader().process(cMapData, cMap, useCMap =>
    extendCMap(cMap, fetchBuiltInCMap, useCMap),
  );
}

So the inheritance chain still lands in extendCMap from the previous article: load the parent CMap by name, clone the parent buckets when the child has no codespaces, then merge the parent entries one by one without overwriting keys the child already has. The binary reader only writes "its own records" into cMap, and parent loading and merging reuse the logic of cmap.js completely.

This chain can be drawn like this:

// BinaryCMapReader.process
//   → read codespace / cid / bf records and write them into the current cMap
//   → on a metadata usecmap record, remember the name
//   → after the records end, call extend(useCMap)
//   → extendCMap: createBuiltInCMap(parent name)
//   → the parent CMap may be text or another .bcmap
//   → merge parent entries and return the current cMap

A parent CMap may itself carry a usecmap, forming a finite chain of built-in inheritance. Each layer is one createBuiltInCMap call, data is fetched by name through fetchBuiltInCMap, and builtInCMapCache caches only raw bytes, never a parsed CMap.

The relation with CMapFactory can be summarised in one table:

The last row deserves emphasis: a CMap embedded in a PDF is never parsed as binary, even if its bytes look like a .bcmap. The binary path serves built-in resources only, because only built-in resources have a fixed naming and packaging process; the contents of an embedded stream are untrusted, and the text parser is better suited to per-record tolerance.

Boundaries and Performance

The failure points of binary_cmap.js are highly concentrated:

Two constants are the heart of these boundaries: MAX_NUM_SIZE bounds dataSize + 1, and MAX_ENCODED_NUM_SIZE bounds the capacity of tmpBuf. The former guarantees that the 16-byte arrays start, end, char and charCode are never written out of bounds, and the latter guarantees that the 7-bit groups of a varint never exceed 19. Both are preconditions of "allocate first, reuse later".

Performance can be summarised in four points:

1. Zero tokens. There is no Lexer character classification, string construction or command objects, only getByte() and shifts.
2. Delta encoding. Only the first item in a block is written in full and later items write small integer deltas; the sequence bit then drops the deltas of consecutive intervals entirely.
3. Fixed buffers. Five Uint8Arrays are reused across the whole process, so no object is allocated per item no matter how large the range.
4. Shared with the write methods. mapCidRange and mapBfRange expand straight into _map, so downstream queries are still a single array index; the binary form only optimises "how data is delivered", never "how it is queried".

Design note: The speed of a binary CMap comes from eliminating the general-purpose parser. A text CMap must go through the PDF lexer because it shares a grammar with other PDF objects; a .bcmap is a private format, and the reader knows the exact meaning of every byte, so it can take fields directly by bit position. The format design and the reader are hard-wired one-to-one: the three header fields, the six types and the delta rules are all agreed between the compression tool and the reader, with no version negotiation and no burden of generality. This is exactly why it is faster than the text path, and also why it can only be used for built-in resources.

Conclusion

binary_cmap.js is PDF.js's built-in CMap decoder: downward it splits the byte stream of a .bcmap into codespace, cid and bf records, and upward it produces a CMap shaped exactly like the one from a text CMap. It takes no part in parsing embedded streams and does not decide how the font fetches glyphs; it only restores pre-compressed data into _map as quickly as possible.

Where It Sits in the Overall Chain

In one sentence: fetchBuiltInCMap retrieves the compressed bytes, BinaryCMapReader restores the mapping record by record, extendCMap fills in the parent mapping, and the font layer and the text layer finally receive the same CMap interface.

Four Design Principles

1. A private format deserves a dedicated reader: a .bcmap does not need to be compatible with PDF syntax, so the reader can take fields by bit position and skip the lexer and general object construction. The more specialised the format, the shorter the parse path.
2. Delta beats repetition: only the first item in a record block is written in full and later items write deltas relative to the previous one; consecutive intervals drop their deltas through the sequence bit. The compression ratio comes from "repeated information appears only once", not from a general-purpose compression algorithm.
3. Fixed length and variable length each do their job: character codes and target strings use fixed-length big-endian bytes so widths and carries are correct; counts, CIDs and small deltas use varints so small numbers take one byte. Mixing the two avoids overflow without wasting space on small values.
4. Decoding is separate from writing: the reader only takes records apart, and the actual range expansion still calls CMap.mapCidRange and mapBfRange. The binary path and the text path share the same write and query logic, so downstream never needs to know which format its CMap came from.

At this point the two entry blocks of the font layer are complete: cmap.js defines the mapping and its parsing, and binary_cmap.js handles the fast decoding of built-in data. Next we can go deeper into the font itself and see how fonts.js uses CIDs, width tables and outline data to turn characters into glyphs that can actually be drawn.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.