PDF.js Source Walkthrough: ascii_hex_stream

Introduction

This is one of the two shortest filters in the "bytes and streams" layer. The subject is AsciiHexStream in the worker-side src/core/ascii_hex_stream.js (79 lines).

The previous articles already built the skeleton of the decoding chain: base_stream defines the read contract, decode_stream provides the output buffer and the readBlock driver, flate_stream handles DEFLATE bit streams, and predictor_stream restores row differences. Each of them had a real algorithm to describe: Huffman trees, the LZ77 window, five predictor formulas. This article takes the opposite extreme: a filter that needs almost no algorithm at all.

In the PDF specification, one class of filters does not compress data; it only changes the representation. ASCIIHexDecode is the plainest of them: each byte becomes two hexadecimal characters, whitespace in between makes no difference, and a greater-than sign terminates the data. Its reason for existing is historical — early transport channels only guaranteed 7-bit safety, so binary streams had to be wrapped in printable text first. Today it shows up mainly in inline images, test files, and PDFs produced by some scanners.

The specification imposes only four requirements on this layer:

1. The input should contain nothing but hexadecimal characters and whitespace.
2. Every two characters are interpreted as one byte, high nibble first.
3. If a single nibble is left over at the end, it is padded with zeros.
4. The greater-than sign is the end marker; anything after it no longer belongs to this stream.

AsciiHexStream exists to turn text back into bytes according to those four rules. Its entry point in the filter chain is short:

        case "AHx":
        case "ASCIIHexDecode":
          return new AsciiHexStream(stream, maybeLength);

This comes from Parser.makeFilter. The upstream may be a plain Stream, or a stream already decrypted or chunked; maybeLength is the /Length from the stream dictionary and serves only as a hint. There is one easily overlooked branch: inline images do not go through this class. Parser.makeInlineImage first calls findASCIIHexDecodeInlineStreamEnd, which scans only for the end marker to determine the length, and then constructs the filter. Scanning and decoding are therefore split into two places, and this article covers the decoding one.

In one sentence, its job: turn a text stream of "two characters per byte" back into binary bytes, keep half a byte across block boundaries, and stop immediately at the end marker.

Module Overview

ascii_hex_stream.js is only 79 lines; stripping the copyright header and the import, the actual logic sits in two methods:

The whole data path can be written as:

Hexadecimal text block
  → AsciiHexStream.readBlock    (this file, at most 8000 input bytes per call)
  → DecodeStream.buffer         (output buffer, at most 4000 bytes)
  → getBytes / getByte          (keep driving readBlock as the caller needs)
  → Lexer / image decoder / XRef record parsing

One division of labour implied by inheritance deserves emphasis: a single readBlock call is only responsible for "decoding as much of one upstream block as possible"; it does not guarantee the amount the caller wants. The actual "collect enough bytes" work is done by DecodeStream.getBytes, which calls readBlock in a loop. This split means the class does not need to care how much the caller wants, only about the boundaries of its own block.

readBlock takes no parameters. Other streams in the filter chain may accept decoderOptions; this class ignores it entirely, because hexadecimal decoding has no tunable parameters.

Constructor: Halving the Size Hint

The constructor method

class AsciiHexStream extends DecodeStream {
  constructor(str, maybeLength) {
    // Most streams increase in size when decoded, but AsciiHex streams shrink
    // by 50%.
    if (maybeLength) {
      maybeLength *= 0.5;
    }
    super(maybeLength);
 
    this.stream = str;
    this.dict = str.dict;
 
    this.firstDigit = -1;
  }

The constructor does only four things, but each of them shapes later behaviour.

The first is the size hint. maybeLength is the /Length from the stream dictionary, meaning the number of encoded bytes, that is the total of hexadecimal characters plus whitespace. The number of decoded bytes equals the number of hexadecimal characters divided by two, and the number of hexadecimal characters cannot exceed the total, so "half the total" is an upper bound on the output. Multiplying by 0.5 is not an exact calculation; it is a conservative starting point.

That value goes into the DecodeStream constructor, where it is raised to "the smallest power of two not smaller than it" and becomes minBufferLength. ensureBuffer then doubles from there. In other words, /Length only decides how large the first allocation is, not how much can ultimately be decoded. One comment in Parser puts it bluntly: the /Length in a stream dictionary may be completely wrong — a non-empty stream may declare 0 — so it can only be treated as a hint.

The second is saving the upstream stream: this.stream = str. All input bytes come from str.getBytes; this class never caches the text.

The third is forwarding the dictionary: this.dict = str.dict. Filter chains are nested many layers deep, and the dictionary must be passed all the way down. The image decoder downstream relies on /Width, /Height and /ColorSpace from the dictionary to interpret pixels; if the chain breaks at this layer, the image cannot be restored. It is a single assignment, yet it is the precondition for the whole chain to work.

The fourth is initializing firstDigit = -1. This is cross-block state that holds "the half byte waiting for its partner". Valid values are 0 to 15; -1 means no nibble is currently waiting.

Design note: The size hint only affects "how large the first allocation is"; it does not affect correctness. The 0.5 factor lets common streams allocate exactly once; even if /Length is completely wrong, the geometric growth in ensureBuffer still catches up. Keeping the hint separate from the capacity guarantee is a practice shared by every decoding stream in this layer.

readBlock: Eight Thousand Bytes at a Time

The readBlock method

  readBlock() {
    const UPSTREAM_BLOCK_SIZE = 8000;
    const bytes = this.stream.getBytes(UPSTREAM_BLOCK_SIZE);
    if (!bytes.length) {
      this.eof = true;
      return;
    }
 
    const maxDecodeLength = (bytes.length + 1) >> 1;
    const buffer = this.ensureBuffer(this.bufferLength + maxDecodeLength);
    let bufferLength = this.bufferLength;

The dozen lines at the top decide the granularity and the memory behaviour of the whole method.

UPSTREAM_BLOCK_SIZE = 8000 is how many input bytes are requested from upstream per call. Three considerations went into that number:

• Upstream getBytes returns a view, not a copy, so the zero-copy cost of 8000 bytes is very low.
• One input block produces at most 4000 output bytes, so the memory increment of a single decode has a clear upper bound and one huge stream cannot eat all the memory at once.
• DecodeStream.getBytes calls readBlock repeatedly; the larger the block, the less call overhead. Beyond a point the returns diminish and the memory peak of a single block grows instead.

A bytes.length of 0 means upstream is already exhausted, so eof is set and the method returns. This is the termination condition and there is nothing more to do.

maxDecodeLength = (bytes.length + 1) >> 1 is the maximum number of bytes this block can produce. Adding one and shifting right is a round-up: if the previous block left a nibble behind, the first character of this block can combine with it into a full byte, so the theoretical maximum is half of "length plus one". A shift is used instead of Math.ceil: it avoids floating point and keeps the integer semantics uniform.

ensureBuffer(this.bufferLength + maxDecodeLength) grows the capacity once and for all to "existing output plus this block's maximum output". This step matters: because the loop's output is already bounded, no capacity check is needed for each byte written later. Growth happens only between blocks, never inside the per-character hot loop.

The last line, let bufferLength = this.bufferLength, copies the length into a local variable. The loop writes only to the local value on each iteration and writes it back to this.bufferLength before the method returns. The benefit is fewer property accesses, and more importantly that "how much this block decoded" is kept separate from "the object's current state", so a mid-loop growth cannot leave a half-updated state.

Design note: The output capacity estimate happens outside the loop; inside the loop there are only two jobs: classify the character and write the byte. This is the uniform shape of every DecodeStream subclass: ensureBuffer handles memory, readBlock handles content, and the two meet at the block boundary.

Working with getBytes

readBlock does not guarantee to produce the amount the caller wants in one go. What stitches multiple blocks together is the base class's read loop:

while (!this.eof && this.bufferLength < end) {
  this.readBlock(decoderOptions);
}

This comes from DecodeStream.getBytes. It first grows the buffer to the requested size, then calls readBlock repeatedly until the buffer covers the target position or eof becomes true. For this class each call contributes at most 4000 bytes, so a request for 100,000 bytes triggers more than twenty readBlock calls, each of which asks further upstream for 8000 bytes of text.

That also explains why eof must be set in both places — "upstream returned 0 bytes" and "the end marker was found": the base class's loop looks only at that boolean. Miss either one and getBytes keeps calling forever while readBlock returns empty every time, which is an infinite loop. The contract is simple, but every subclass has to honour it.

For the caller, the result may be shorter than requested: when the source text ends early, getBytes clamps the end to bufferLength and returns a shorter view. Parsers above advance by the actual length they got and never assume they receive exactly what they asked for.

Character Classification: Letting the ASCII Layout Be the Lookup Table

Classifying hexadecimal characters

    let firstDigit = this.firstDigit;
    for (const ch of bytes) {
      let digit;
      if (ch >= /* '0' = */ 0x30 && ch <= /* '9' = */ 0x39) {
        digit = ch & 0x0f;
      } else if (
        (ch >= /* 'A' = */ 0x41 && ch <= /* 'Z' = */ 0x46) ||
        (ch >= /* 'a' = */ 0x61 && ch <= /* 'z' = */ 0x66)
      ) {
        digit = (ch & 0x0f) + 9;
      } else if (ch === /* '>' = */ 0x3e) {
        this.eof = true;
        break;
      } else {
        // Probably whitespace, ignoring.
        continue;
      }

This branch is the only place in the whole file that needs to "compute", yet it computes very cheaply: two range comparisons, one mask, one addition — no lookup array and no conversion of the byte to a string.

The principle is the ASCII code layout:

Why add 9 rather than 10 for letters? Because the low four bits of A are 1, not 0: 0x41 & 0x0f equals 1, and to make A map to 10 the offset must be 9. F has low bits 6, and 6 plus 9 is 15. The offset can be a constant only because letters are contiguous in the ASCII table. Under a layout like EBCDIC the trick would not work at all.

The source comment on the upper bound for letters says 'Z', but the actual value 0x46 is the code for F. The test must stop at F: if it really let everything up to Z through, G to Z would be computed as values of 16 or more and the assembled bytes would be wrong. The comment is a typo; the code is right.

The else branch is commented "Probably whitespace", but its actual behaviour is broader than whitespace: any byte that is neither a hexadecimal character nor the end marker is skipped, including control characters the specification does not list as whitespace. It is a tolerant stance: better to let garbage through than to make a whole document fail over one stray byte. The price is that real corruption goes unnoticed and the decoded output can quietly come out short.

One more ordering detail: the end-marker test comes after the hexadecimal tests. The two ranges do not overlap numerically, so the order does not affect correctness; but putting the most common, fastest-to-hit branch first reduces the average number of comparisons. Over a loop of 8000 bytes that difference is measurable.

Design note: The continuity of ASCII is itself a free table. Digits and letters have contiguous low nibbles, so "classify" and "extract the value" collapse into a single mask. A lookup table would add a memory access; string comparison would be slower still. The bit operations here are not showing off — they leave the hot loop with nothing but arithmetic.

Assembling a Byte: Storing Only Half a Byte

if (firstDigit < 0) {
  firstDigit = digit;
} else {
  buffer[bufferLength++] = (firstDigit << 4) | digit;
  firstDigit = -1;
}

This is the whole decoder's state machine, and it has just two states: no nibble waiting and one nibble waiting. The source encodes the former as the integer -1 and the latter as 0 to 15, without introducing an extra boolean field.

High nibble first is a requirement of the specification: the first character is the high four bits, the second the low four bits. (firstDigit << 4) | digit shifts the waiting value left by four bits and fills the low bits with a bitwise OR. OR would work as well as addition, because the low four bits are all zero after the shift and both operators give the same result; OR just expresses "fill the bits" more directly.

Writing back takes bufferLength first and increments afterwards — the classic use-then-increment. buffer is the reference returned by ensureBuffer and may be a new array after growth; since growth happens before the loop, the loop carries no risk of the array being swapped underneath it.

This state machine also explains an easily missed fact: decoding progress is not aligned to blocks. A block boundary may fall exactly between two characters or between two bytes. The state machine only cares whether half a byte is pending; it does not care where the block ends.

Design note: Continuing across blocks requires no cached text, only one nibble. This is the classic case of saving state by meaning rather than by input: no matter how the upstream chops the text into blocks, the decoder holds a constant amount of information.

The Cost of the Per-Character Loop

for (const ch of bytes) iterates the Uint8Array iterator, yielding a number from 0 to 255 each time. An index loop would be closer to the machine, but readability wins here: the type is fixed and the length is known, and engines usually handle this kind of iteration well; even without optimisation, the extra per-iteration overhead is hidden behind the branching and byte writes that follow.

Classified by input byte, the cost of processing a single byte is:

The worst case is five comparisons and no memory allocation at all. An 8000-byte block performs at most forty thousand integer comparisons, negligible for a modern engine; the real cost is in writing output and fetching blocks upstream, not in the tests.

So why not use a 256-entry lookup table? A table compresses "classify" into a single memory read, but it has to be initialized, and the latency of random access is not necessarily lower than a handful of predictable branches. For input dominated by hexadecimal characters, branch prediction hits almost every time, so range tests are the more stable choice. With unusual input dominated by whitespace, neither matters, because the bottleneck is then simply the number of bytes skipped.

break exits only the current block; it does not rewind upstream. The bytes skipped afterwards have already been consumed, but the stream is over and they never belonged to this filter anyway. In the inline-image case the end marker is often followed immediately by /EI; that part is handled by another piece of Parser logic and is not the decoder's concern.

The Other Path: Inline Images

ASCIIHexDecode is the same filter, but an inline image does not use this class to determine its length. Parser.findASCIIHexDecodeInlineStreamEnd only scans for the end marker:

  findASCIIHexDecodeInlineStreamEnd(stream) {
    const GT = 0x3e;
    const startPos = stream.pos;
    let ch;
    while ((ch = stream.getByte()) !== -1) {
      if (ch === GT) {
        break;
      }
    }
    const length = stream.pos - startPos;
    if (ch === -1) {
      warn(
        "Inline ASCIIHexDecode image stream: " +
          "EOD marker not found, searching for /EI/ instead."
      );
      stream.skip(-length); // Reset the stream position.
      return this.findDefaultInlineStreamEnd(stream);
    }
    this.inlineStreamSkipEI(stream);
    return length;
  }

An inline image has no stream dictionary and therefore no /Length. Parser must first know how many bytes the image data occupies in the file before it can cut it out and build a sub-stream. The cheapest way to do that is not to decode but to "find the marker": read byte by byte until the greater-than sign, and the difference in position is the length. That scan does no nibble pairing, allocates no output buffer, and costs a single linear read.

After finding the marker it calls inlineStreamSkipEI to skip the /EI that follows. That is a small state machine: match the letters E and I in order, and return to the initial state on a mismatch, so that an accidental E or I inside the data is not mistaken for the terminator.

If no marker is found, the code first calls stream.skip(-length) to rewind to the start, then falls back to the generic findDefaultInlineStreamEnd. The rewind works because BaseStream.skip accepts negative numbers — another sign that the cursor has to be writable. The generic scan looks for /EI, which is less reliable than an explicit marker but still much better than reading the whole file through.

The split between the two paths is clear: the scan decides "how long the slice is", the decoder decides "what bytes it becomes". The scan is an approximation, the decode is an exact restoration, and they agree only on the same end marker. Note also that the returned length includes the end marker itself, so the downstream AsciiHexStream will still read the greater-than sign and treat it as the end — the scan does not consume it on the decoder's behalf, it merely counts it in the slice.

Design note: The same data format can have two implementations depending on purpose. A stream object uses "lazy decoding", with the length bounded by the dictionary or by the end marker; an inline image uses "scan first, decode later", because the boundary must be cut first. Keeping the two concerns apart means neither slows the other down.

End Marker and Zero Padding

Finishing Up

if (firstDigit >= 0 && this.eof) {
  // incomplete byte
  buffer[bufferLength++] = firstDigit << 4;
  firstDigit = -1;
}
this.firstDigit = firstDigit;
this.bufferLength = bufferLength;

Finishing happens in three steps.

The first handles the specification's "pad an odd nibble with zero". If a nibble remains when the loop ends and eof is already true, that nibble is shifted left four bits and written out with the low bits set to 0. For example, a trailing 2 produces 0x20, the space character. Padding happens only when eof is true; if the block merely ended, the nibble is left for the next block.

The second writes the state back to the object. Even when firstDigit is -1 this assignment must run, because it may overwrite the value left by the previous block. The third writes back the length.

The three places that can trigger eof do not behave identically:

The second and third rows deserve their own explanation. The early-return path on if (!bytes.length) only sets eof and exits; it never runs the padding logic. In other words, if a stream has no end marker and its last nibble happens to fall just before a block boundary, that nibble is silently dropped. The specification requires streams to end with an end marker, so a normal file never gets here; it only happens when the file is truncated or the generator is broken. This is a deliberate trade-off: adding a test for illegal input that practically never occurs is not worth it, and half a byte of data could not be restored correctly anyway.

Here is a concrete decode. The input is 48 65 6C 6C 6F 21 (spaces between characters are only for readability; real spaces are skipped):

If the last byte were replaced by a lone 2, the output would be 0x20. If the input were split into two blocks in the middle — say the first block is 48 6 and the second is 5 3E — firstDigit would hold 6 at the end of the first block, the second block would combine it with 5 into 0x65, and then hit the end marker to set eof. Only a single integer passes between the two readBlock calls.

Design note: The end marker is the sole authority; /Length is used only to estimate the buffer. Even if bytes remain in the same block after the marker, they are not processed. This rule makes the decoder naturally immune to a wrong length field — as long as the text contains an end marker, the decoded result is correct.

Differences from Other Filters

Put the three "uncompressed" or "simple encoding" filters side by side and the differences are obvious:

ASCII85Decode multiplies the hint by 0.8 while AsciiHexStream multiplies by 0.5, for the same reason: bring the first buffer allocation close to the real output. Ascii85Stream reads a full group of five characters at a time and uses isWhiteSpace to detect whitespace inside the group, so it needs no cross-block state; it stops at a tilde or at the end of upstream, and pads an incomplete final group with 0x21 + 84.

RunLengthDecode takes a completely different route: it handles binary bytes, skips no characters, and stores no half byte. ASCIIHex and RunLength are both short, yet their state management is opposite: one relies on a resident integer, the other on "read a whole run each time".

All three filters share one property: they do not change the meaning of the data, only its representation. Precisely for that reason none of them needs a dictionary, window or code table the way Flate does, and readBlock can be a few dozen lines long.

Conclusion

AsciiHexStream is PDF.js's text-to-binary filter: above it accepts any byte source, below it outputs a continuous run of binary bytes, and in between a single nibble of state carries the join across blocks.

Where It Sits in the Overall Chain

In one sentence: upstream supplies text blocks, readBlock joins every two characters into a byte, getBytes stitches blocks into one continuous byte stream, and parsers and image decoders only look at the result.

Five Design Principles

1. A hint is a hint; capacity relies on the fallback. maybeLength times 0.5 only sizes the first allocation; the doubling strategy in ensureBuffer is the real capacity guarantee. Separating the two lets the decoder tolerate a wrong length field.
2. Let the encoding layout do the classification. Digits and letters have contiguous low nibbles, so "is this a hexadecimal character" and "what value does it have" collapse into one mask plus one addition. No lookup table and no string conversion in the hot loop.
3. Save state by meaning, not by input. Continuing across blocks needs one integer from 0 to 15, plus -1 for empty. However large the block or however the text is split, the state size is constant.
4. Grow outside the loop, write bytes inside it. (bytes.length + 1) >> 1 gives a strict upper bound on this block's output, so the loop body never needs to check capacity.
5. An explicit end marker outranks the length field. Stop as soon as the marker is found, without relying on the accuracy of /Length; that is also why it can handle streams of indeterminate length such as inline images.

The next article takes a different angle: run_length_stream handles run-length encoding — "one header byte plus a run of data". It has no characters to skip and needs no half-byte state; its complexity lies in run semantics and buffer accumulation.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.