PDF.js Source Walkthrough: metadata_parser

Introduction

This is the article in the PDF.js source walkthrough series about XMP metadata. It covers src/core/metadata_parser.js (146 lines).

Besides the title, author and creation date in the /Info dictionary, a PDF document can carry a Metadata stream. The content of that stream is XMP, which is essentially RDF/XML. Viewers use it to read richer information: dc:title, dc:creator, xmp:CreateDate, pdf:Producer, and vendor-specific fields.

XMP in real files is often not well formed: there may be junk bytes before the stream, Ghostscript writes UTF-16BE text as octal escapes, namespace prefixes vary in case, and arrays are mixed with language alternatives. The job of metadata_parser.js is not to be a general XML parser but to repair this possibly damaged XMP as far as possible and flatten it into a table from field name to a string or an array of strings. It preserves no DOM, does no XPath, and keeps no namespace objects.

Module Overview

Where it sits in the chain:

Catalog.getMetadata
  → XRef.fetch(Metadata stream)
  → BaseStream.getString
  → stringToUTF8String
  → new MetadataParser(data)
  → _repair
  → SimpleXMLParser.parseFromString
  → _parse
  → serializable
  → Metadata (display layer)
  → app.js / pdf_document_properties.js

Version note: This article follows exactly the version of metadata_parser.js in the current repository. MetadataParser does not extend SimpleXMLParser; it composes one: the constructor creates a parser and hands data to it. There are no beginElement / endElement methods in the source; the corresponding parser callbacks are onBeginElement / onEndElement. There are also no onclosetag / onopentag callbacks, because text and CDATA go through onText and onCdata respectively. Likewise there is no parse() method returning xmp and xdc dictionaries; this version has only serializable, which returns a Map and a string. The _metadata, xmp and xdc fields common in older versions do not exist here.

Entry Point: The Constructor and _repair

The constructor is short, but the order matters: repair first, then parse.

class MetadataParser {
  constructor(data) {
    // Ghostscript may produce invalid metadata, so try to repair that first.
    data = this._repair(data);
 
    // Convert the string to an XML document.
    const parser = new SimpleXMLParser({ lowerCaseName: true });
    const xmlDocument = parser.parseFromString(data);
 
    this._metadataMap = new Map();
    this._data = data;
 
    if (xmlDocument) {
      this._parse(xmlDocument);
    }
  }

The four actions are: repair the text, create a parser that lowercases element names, build a lightweight DOM, and initialize an empty map and traverse the DOM if it exists. Two points are easy to miss.

First, this._data holds the repaired text, not the original text passed by the caller. serializable.rawData is therefore the output of _repair; if repair changed any characters, it no longer matches the bytes in the stream character for character.

Second, lowerCaseName: true applies to all element names. The parser lowercases names in the start-tag callback, so the source compares rdf:rdf, rdf:description, rdf:bag and rdf:li, and field names are normalized to dc:title, xmp:createdate and pdf:producer. Consumers must use lowercase keys.

The _repair Method

_repair is a chain of regular-expression replacements that handles only two known kinds of bad data.

  _repair(data) {
    // Start by removing any "junk" before the first tag (see issue 10395).
    return data
      .replace(/^[^<]+/, "")
      .replaceAll(/>\\376\\377([^<]+)/g, function (all, codes) {
        const bytes = codes
          .replaceAll(/\\([0-3])([0-7])([0-7])/g, function (code, d1, d2, d3) {
            return String.fromCharCode(d1 * 64 + d2 * 8 + d3 * 1);
          })
          .replaceAll(/&(amp|apos|gt|lt|quot);/g, function (str, name) {
            switch (name) {
              case "amp":
                return "&";
              case "apos":
                return "'";
              case "gt":
                return ">";
              case "lt":
                return "<";
              case "quot":
                return '"';
            }
            throw new Error(`_repair: ${name} isn't defined.`);
          });
 
        const charBuf = [">"];
        for (let i = 0, ii = bytes.length; i < ii; i += 2) {
          const code = bytes.charCodeAt(i) * 256 + bytes.charCodeAt(i + 1);
          if (
            code >= /* Space = */ 32 &&
            code < /* Delete = */ 127 &&
            code !== /* '<' = */ 60 &&
            code !== /* '>' = */ 62 &&
            code !== /* '&' = */ 38
          ) {
            charBuf.push(String.fromCharCode(code));
          } else {
            charBuf.push(
              "&#x" + (0x10000 + code).toString(16).substring(1) + ";"
            );
          }
        }
        return charBuf.join("");
      });
  }

Broken into six steps in execution order:

Step 1 corresponds to issue 10395. The regex matches from the start to the first < and removes the entire description in front of it. If the data contains no <, the result is an empty string and the later parse naturally produces an empty table.

Step 2 targets Ghostscript. It writes UTF-16BE text with octal escapes such as \376\377: \376 is 0xFE and \377 is 0xFF, and together they form the BOM. The regex captures the >, the two escapes, and everything after them up to the next <. The > is placed alone as the first element of charBuf and preserved as is.

Step 3 restores each octal escape to a one-byte character. d1, d2 and d3 are strings, and multiplication converts them to numbers: the first digit times 64, the second times 8, the third times 1. The first digit ranges from 0 to 3 and the last two from 0 to 7, so the maximum is 0o377, exactly one byte.

Step 4 decodes entities. Restoring octal before decoding entities means a literal & becomes & and is not later mistaken for markup. Only the five predefined entities are handled; any other name throws.

Step 5 interprets two characters at a time. charCodeAt(i) * 256 + charCodeAt(i + 1) combines high and low bytes into one 16-bit code unit. It assumes an even length; for an odd length the last pair reads NaN and produces an invalid entity, and the source has no guard for that extreme case.

Step 6 decides how each code unit is written back to XML:

60, 62 and 38 are <, > and &. Written raw, XML parsers would read them as the start of a tag, the end of a tag, or the start of an entity, so they must be escaped.

The numeric entity construction is worth a closer look:

'&#x' + (0x10000 + code).toString(16).substring(1) + ';';

code is at most 0xFFFF; adding 0x10000 and converting to hexadecimal always yields five digits whose first digit is 1; substring(1) strips the prefix and leaves exactly four. This is a zero-padding trick that does not rely on padStart.

Design note: _repair fixes only the two known kinds of damage and does not try to understand XML structure. Writing every non-ASCII code unit as a numeric character reference turns the encoding problem into plain ASCII text, so the later parser need not care whether the original bytes were UTF-8, UTF-16BE or PDFDocEncoding. The repair boundary is narrow, but it covers the two most common failures in real files.

How SimpleXMLParser Cooperates

MetadataParser uses SimpleXMLParser from src/core/xml_parser.js. It does not implement the browser DOM; it only builds element nodes, text nodes and parent-child relationships. The parser's state is kept in two arrays: _currentFragment points at the array into which new nodes should currently go, and _stack holds the outer arrays. A start tag pushes and switches, an end tag pops and switches back, and the whole tree is therefore built out of nested arrays.

The two callbacks that really determine node shape are:

  onBeginElement(name, attributes, isEmpty) {
    if (this._lowerCaseName) {
      name = name.toLowerCase();
    }
    const node = new SimpleDOMNode(name);
    node.childNodes = [];
    if (this._hasAttributes) {
      node.attributes = attributes;
    }
    this._currentFragment.push(node);
    if (isEmpty) {
      return;
    }
    this._stack.push(this._currentFragment);
    this._currentFragment = node.childNodes;
  }
 
  onEndElement(name) {
    this._currentFragment = this._stack.pop() || [];
    const lastElement = this._currentFragment.at(-1);
    if (!lastElement) {
      return null;
    }
    for (const childNode of lastElement.childNodes) {
      childNode.parentNode = lastElement;
    }
    return lastElement;
  }

A few characteristics are worth listing:

hasAttributes is not passed at construction, so elements have no attributes field. Attributes such as xml:lang are dropped during parsing, which means language information cannot be preserved for language alternatives in dc:title.

Text nodes are created by onText and onCdata. onText first checks isWhitespaceString and returns immediately for pure whitespace; otherwise it creates a #text node and appends it to the current array. This matters for traversal: rdf:Description's child list contains no runs of whitespace #text, so the source only has to skip scattered non-whitespace text. onCdata has the same structure but does not filter whitespace, because whitespace inside CDATA may be meaningful.

SimpleDOMNode.textContent concatenates recursively:

  get textContent() {
    return !this.childNodes
      ? this.nodeValue || ""
      : this.childNodes.map(child => child.textContent).join("");
  }

With no children it returns nodeValue; with children it concatenates the text of all of them. It inserts no separator, so a field with several nested rdf:li elements becomes one continuous string rather than an array. Array semantics must be handled separately by _parseArray.

parseFromString sets an error code and returns undefined on any syntax error; it also returns undefined when there is no root node. When MetadataParser sees undefined it does not call _parse, and the result is an empty table rather than an exception.

Design note: SimpleXMLParser is a good-enough parser. It does not validate tag pairing, expand custom entities, keep attributes or handle namespace declarations. It only guarantees that well-formed common XMP becomes a traversable tree; broken parts are either skipped or cause the whole call to return undefined. The worker therefore does not have to pull in a full XML library.

Locating rdf:RDF and rdf:Description

After obtaining xmlDocument, _parse first finds the RDF root and then iterates over each description node.

  _parse(xmlDocument) {
    let rdf = xmlDocument.documentElement;
 
    if (rdf.nodeName !== "rdf:rdf") {
      // Wrapped in <xmpmeta>
      rdf = rdf.firstChild;
      while (rdf && rdf.nodeName !== "rdf:rdf") {
        rdf = rdf.nextSibling;
      }
    }
 
    if (!rdf || rdf.nodeName !== "rdf:rdf" || !rdf.hasChildNodes()) {
      return;
    }
 
    for (const desc of rdf.childNodes) {
      if (desc.nodeName !== "rdf:description") {
        continue;
      }
 
      for (const entry of desc.childNodes) {
        const name = entry.nodeName;
        switch (name) {
          case "#text":
            continue;
          case "dc:creator":
          case "dc:subject":
            this._parseArray(entry);
            continue;
        }
        this._metadataMap.set(name, entry.textContent.trim());
      }
    }
  }

Standard XMP has two common outer shapes:

The second row is the source of the while loop in _parse. It scans only sibling nodes and does not recurse into subtrees. If there is another wrapper layer between xmpmeta and rdf:rdf, this code will not find the target and ultimately returns an empty table. Real files usually have only one wrapper.

Once found, there are three more checks: rdf exists, its name is rdf:rdf, and it has at least one child. If any check fails, the method returns immediately without attempting repairs.

Among the children of rdf:rdf, only rdf:description is processed; every other node is skipped. Inside a description node, each child is processed in turn. #text is explicitly skipped: onText only filtered pure whitespace above, so if a description node contains non-whitespace bare text it becomes a #text node, and the source does not want to treat that as a field name.

The switch special-cases only two names:

Every other field goes through the same single write. This means a dc:title written as an rdf:Alt multilingual array is concatenated into one string rather than an array; the same is true for dc:description; single-value fields such as xmp:CreateDate and pdf:Producer become plain text; nested structures are concatenated recursively and the hierarchy is lost.

This is special-casing by field name, not inferring by value type. The source does not look at rdf:Bag, rdf:Seq or rdf:Alt to decide whether to parse an array; it looks only at the field name. The benefit is an extremely short implementation; the cost is that only the two most common multi-value fields, dc:creator and dc:subject, come back as arrays.

_getSequence and _parseArray

Array fields are produced by two methods working together:

  _getSequence(entry) {
    const name = entry.nodeName;
    if (name !== "rdf:bag" && name !== "rdf:seq" && name !== "rdf:alt") {
      return null;
    }
    return entry.childNodes.filter(node => node.nodeName === "rdf:li");
  }
 
  _parseArray(entry) {
    if (!entry.hasChildNodes()) {
      return;
    }
    // Child must be a Bag (unordered array) or a Seq.
    const [seqNode] = entry.childNodes;
    const sequence = this._getSequence(seqNode) || [];
 
    this._metadataMap.set(
      entry.nodeName,
      sequence.map(node => node.textContent.trim())
    );
  }

_getSequence accepts three container names:

All three collect rdf:li in document order and ignore the unordered versus ordered distinction. The language attributes on Alt were already lost during parsing, so all that remains is an array of the text in each language.

_parseArray takes only the first child of entry as the container. const [seqNode] = entry.childNodes is destructuring, and every later child is ignored. Since whitespace text has been filtered, the first child is normally one of the three containers. If it is not, _getSequence returns null, || [] turns that into an empty array, and an empty array is written instead of skipping the field.

If entry has no children at all, the method returns immediately and the field does not appear in the Map. An empty array and a missing field are therefore two different outcomes, and consumers must distinguish them.

Values are handled uniformly with node.textContent.trim(). For rdf:li this is usually the language text; if there are nested tags inside, they are still concatenated recursively.

Design note: _parseArray checks container type and entry type separately. The container must be one of the three RDF collection types, and the entries must be rdf:li. But the source only checks the first child and does not handle several containers side by side. This is a minimal implementation aimed at common XMP, not a complete RDF parser.

External Interface: serializable and Consumers

The parse result is exposed through an accessor:

  get serializable() {
    return {
      parsedData: this._metadataMap,
      rawData: this._data,
    };
  }

The two fields have different purposes:

The worker does not hand this object directly to the main thread; it goes through Catalog.getMetadata:

const data = stringToUTF8String(stream.getString());
if (data) {
  metadata = new MetadataParser(data).serializable;
}

The Metadata wrapper on the main thread does three things: get(name) looks up the table, getRaw() returns the text, and the iterator exposes every key-value pair.

  get(name) {
    return this.#map.get(name) ?? null;
  }
 
  getRaw() {
    return this.#data;
  }

Key names were already lowercased in SimpleXMLParser, so consumers must use lowercase:

This also explains why lowerCaseName cannot simply be turned off: consumers are already hard-coded to lowercase keys. If the parser preserved original case, xmp:CreateDate and xmp:createdate would be two different keys.

Design note: serializable returns the structured result together with the raw text. Upper layers can use the Map for fast lookups and can also get the complete XMP when they need to display it or parse it again. Keys are lowercased during parsing, which pushes case-insensitivity into the worker so the main thread does not normalize on every comparison.

Fault-Tolerance Boundaries

The last row is the most notable. SimpleXMLParser's entity handling covers only the five predefined entities and numeric character references; every other entity name goes to onResolveEntity, whose default implementation returns the original string. It does not read a DTD and does not load external entities. Custom entities in XMP therefore cause no network request or file read, and they are not expanded into unexpected content.

Another boundary is multiple rdf:Description nodes. _parse iterates over all direct children and writes the fields from each description into the same Map. If two descriptions write the same field, the later one overwrites the earlier one; the source does not merge, compare or report an error.

Conclusion

metadata_parser.js is PDF.js's metadata flattening layer: downward it receives a possibly damaged XML text, upward it outputs a Map with uniform case and a limited set of value types.

Where It Sits in the Overall Chain

In one sentence: _repair turns bad text into parseable ASCII, SimpleXMLParser turns the text into a tree, _parse flattens the tree into a table, and Metadata wraps the table in an interface the main thread can query.

Four Design Principles

1. Repair before parsing. XMP damage usually happens at the text layer, not the structure layer. Handling leading junk, octal escapes and control characters before they reach the parser means the later state machine needs no branches for them.
2. Composition, not inheritance. MetadataParser needs an XML parser that can build a tree but does not need to expose all of the parser's callbacks. Composition lets it depend on a single entry point, parseFromString, and avoids mixing base-class state with its own fields.
3. Special-case by field name, not by value type. Only dc:creator and dc:subject take the array path; everything else uses textContent. The implementation is very short and covers the most common multi-value fields; consumers handle the rest as needed.
4. Fail down to an empty table rather than throwing. XML syntax errors, a missing RDF root and missing description nodes all end up as empty metadata. For a viewer, a document without XMP still opens normally, which is more reasonable than aborting the whole load over optional information.

The next article returns to the content stream side. XMP describes what a document is, while a default appearance string describes what annotations and form fields look like. default_appearance.js splits a string such as /Helvetica 0 Tf 0 g into font size and color, and also fabricates a fake font when no usable font data is available.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.