PDF.js Source Walkthrough: decode_stream

Introduction

This is the second article in the "Bytes and Streams" layer.

The previous article, chunked_stream, solved "where the bytes come from": how network slices become an addressable buffer on demand. This article covers the next leg — how those bytes get decoded.

PDF stream objects almost never hold raw data directly. A page content stream is usually Flate-compressed, a font file may be wrapped in ASCII85 and LZW, an image is one of DCT, JPX, JBIG2 or CCITT, and some streams go through an additional PNG or TIFF predictor. The parser never reads the bytes stored in the file directly; it reads the bytes left after every filter has been undone.

PDF.js organises all its decoders into one inheritance chain: DecodeStream is the common base class, and FlateStream, LZWStream, Ascii85Stream, RunLengthStream, PredictorStream, JpegStream and the rest extend it. The base class knows nothing about any specific algorithm; it provides exactly three things:

1. An output buffer that grows automatically.
2. A read interface driven by readBlock as its only source.
3. Two channels that materialise decoded output into an ordinary Stream: makeSubStream and clone.

A subclass only has to do one thing: implement readBlock, write the decoded result into the buffer (text decoders append piece by piece, image decoders often replace it wholesale), and set eof to true when the source data is exhausted.

In one sentence: DecodeStream is the trio of "output buffer, read cursor, abstract readBlock", and every filter is mounted on that skeleton.

Module Overview

decode_stream.js is 246 lines long and splits into six parts:

Where it sits in the chain:

The chain in one sentence: the byte source supplies encoded data, DecodeStream subclasses decode it chunk by chunk into the output buffer, and the parser only ever sees contiguous decoded bytes.

The Base-Class Contract: Implement Only readBlock

const emptyBuffer = new Uint8Array(0);
 
// Super class for the decoding streams.
class DecodeStream extends BaseStream {
  buffer = emptyBuffer;
 
  bufferLength = 0;
 
  eof = false;
 
  minBufferLength = 512;
 
  pos = 0;
 
  constructor(maybeMinBufferLength) {
    super();
    this._rawMinBufferLength = maybeMinBufferLength || 0;
 
    if (maybeMinBufferLength) {
      // Compute the first power of two that is as big as maybeMinBufferLength.
      while (this.minBufferLength < maybeMinBufferLength) {
        this.minBufferLength *= 2;
      }
    }
  }
 
  readBlock() {
    unreachable("Abstract method `readBlock` called");
  }

emptyBuffer is a shared zero-length array. The comment explains why it is shared instead of using null: many DecodeStream instances never produce any data during their lifetime, so one shared empty array saves memory and avoids writing a "buffer is empty" special case into every read method.

Each field has a clear role:

The constructor argument maybeMinBufferLength is a hint, not an exact length. Many decoding streams can obtain the compressed length from a dictionary, and the decompressed size is usually larger, so it is used as a starting point to avoid doubling up from 512 over and over. minBufferLength is raised to "the smallest power of two that is not less than the hint", so the later growth sequence is the tidy 512, 1024, 2048 and so on.

Design note: The base class defines the interface only, never the algorithm. The sole contract of readBlock is "produce a batch of bytes and update bufferLength, or set eof to true". The base class does not check how many bytes were produced, which filter produced them, or whether decoding finished in one go. That extremely narrow contract lets more than a dozen completely different decoders share the same reading, growing and derivation logic.

Buffer Growth: ensureBuffer

  ensureBuffer(requested) {
    const buffer = this.buffer;
    if (requested <= buffer.byteLength) {
      return buffer;
    }
    let size = this.minBufferLength;
    while (size < requested) {
      size *= 2;
    }
    const buffer2 = new Uint8Array(size);
    buffer2.set(buffer);
    return (this.buffer = buffer2);
  }

ensureBuffer is the only memory function a subclass needs to care about. It decides based on capacity, not logical length: as long as the currently allocated memory can hold the request, it reuses the buffer, no matter how many valid bytes it contains.

The growth policy is "start at minBufferLength and keep doubling until the request fits". That gives three properties:

• The number of allocations is O(log n); going from 512 bytes to tens of megabytes takes only a dozen or so allocations.
• Total copying is O(n): old contents are copied each time, but capacity doubles, so the amortised copy cost per byte is constant.
• Peak memory usually stays below twice the requested amount; when the request is smaller than 512, the lower bound minBufferLength applies.

Copying uses buffer2.set(buffer), which copies the old buffer's entire capacity, including the bytes after bufferLength that have never been used. Those bytes are garbage; copying them costs a little time and cannot affect correctness, and in exchange the implementation never has to distinguish "valid region" from "free region".

Design note: Separating capacity from logical length is what allows a decoding stream to append repeatedly. bufferLength grows as decoding progresses, while buffer.byteLength only jumps when the buffer cannot hold more. The read interface depends only on the former, the growth logic only on the latter, and the two never interfere.

Reading: Driving readBlock on Demand

  getByte() {
    const pos = this.pos;
    while (this.bufferLength <= pos) {
      if (this.eof) {
        return -1;
      }
      this.readBlock();
    }
    return this.buffer[this.pos++];
  }

getByte is the core of the whole mechanism. It uses a while rather than an if, because a single readBlock may produce no bytes at all:

• A Flate block may only update internal state.
• A predictor may have read a row but not yet accumulated enough output.
• An image decoder may decode the whole image at once, or may be waiting for asynchronous data.

The loop condition bufferLength <= pos means "there is no valid byte at the current cursor". As long as eof is not reached, the subclass keeps producing; once eof is true and there are still no bytes, -1 is returned, matching the end-of-stream semantics of Stream.

This also implies an obligation on every subclass: readBlock must either produce bytes or set eof. Otherwise the loop never terminates. The base class has no timeout or iteration guard, because it trusts every subclass to honour the contract.

  getBytes(length, decoderOptions = null) {
    const pos = this.pos;
    let end;
 
    if (length) {
      this.ensureBuffer(pos + length);
      end = pos + length;
 
      while (!this.eof && this.bufferLength < end) {
        this.readBlock(decoderOptions);
      }
      const bufEnd = this.bufferLength;
      if (end > bufEnd) {
        end = bufEnd;
      }
    } else {
      while (!this.eof) {
        this.readBlock(decoderOptions);
      }
      end = this.bufferLength;
    }
 
    this.pos = end;
    return this.buffer.subarray(pos, end);
  }

getBytes has two modes:

• With a length: grow the capacity to pos + length first, then decode in a loop until end is covered or eof is reached. If the source data ends early, end is clamped to bufferLength, so the caller gets a shorter span instead of an error.
• Without a length: decode all the way to eof and return every byte from the current position to the end.

Calling ensureBuffer(pos + length) up front folds this request's size into the capacity once, avoiding repeated checks and many small growth steps while decoding advances byte by byte. subarray is taken only after reading finishes, and the view points into the final buffer; even if some readBlock produced more bytes than requested and triggered another growth, the view is created after all appends, so it never points at the old array.

decoderOptions is passed through to readBlock unchanged. Ordinary text decoders ignore it; image decoders use it to control scaling, colour transforms and similar parameters. The base class does not interpret it, only forwards it.

  get isEmpty() {
    while (!this.eof && this.bufferLength === 0) {
      this.readBlock();
    }
    return this.bufferLength === 0;
  }

isEmpty does not simply compare lengths; it is a lazy check: as long as there is no output and the end has not been reached, it keeps decoding, until the first byte is produced or the stream truly ends. So when a caller asks "does this stream have any data", the answer reflects the decoded reality rather than the intermediate state "the buffer happens to be empty right now".

The price is that isEmpty may trigger a large amount of decoding work. It is therefore only appropriate when an empty-stream check is genuinely needed, and should not be called repeatedly on a hot path.

Images and Asynchronous Decoding

getImageData

Image streams differ from text streams: the parser wants the whole decoded pixel block, not bytes read while parsing. getImageData provides three paths for that:

  async getImageData(length, decoderOptions) {
    if (!this.canAsyncDecodeImageFromBuffer) {
      return this.isAsyncDecoder
        ? this.decodeImage(null, length, decoderOptions)
        : this.getBytes(length, decoderOptions);
    }
    const data = await this.stream.asyncGetBytes();
    return this.decodeImage(data, length, decoderOptions);
  }

canAsyncDecodeImageFromBuffer is implemented in JpegStream, JpxStream and Jbig2Stream as "is the upstream stream asynchronous". When it is true, the whole encoded block is fetched first and decoded afterwards, which avoids repeatedly interrupting the decoder while it waits for the network. A true isAsyncDecoder means the decoder itself is asynchronous (JpxStream, Jbig2Stream and CCITTFaxStream are), so null is passed and decodeImage fetches the whole block from upstream through this.bytes itself.

decodeImage is implemented by each image stream subclass and returns decoded pixel data. The base class does not implement it; it only chooses the path.

asyncGetBytesFromDecompressionStream

  async asyncGetBytesFromDecompressionStream(name) {
    this.stream.reset();
    const bytes = this.stream.isAsync
      ? await this.stream.asyncGetBytes()
      : this.stream.getBytes();
 
    try {
      const { readable, writable } = new DecompressionStream(name);
      const writer = writable.getWriter();
      await writer.ready;
 
      // We can't await writer.write() because it'll block until the reader
      // starts which happens few lines below.
      writer
        .write(bytes)
        .then(async () => {
          await writer.ready;
          await writer.close();
        })
        .catch(() => {});
 
      const chunks = [];
      let totalLength = 0;
 
      for await (const chunk of readable) {
        chunks.push(chunk);
        totalLength += chunk.byteLength;
      }
      const data = new Uint8Array(totalLength);
      let offset = 0;
      for (const chunk of chunks) {
        data.set(chunk, offset);
        offset += chunk.byteLength;
      }
      return { decompressed: data, compressed: bytes };
    } catch {
      return { decompressed: null, compressed: bytes };
    }
  }

This is a fast path that trades browser-native decompression for speed, in several steps:

1. Reset the upstream stream. this.stream.reset() sends the read position back to the start; if the upstream is asynchronous, it also waits for the whole encoded block to arrive.
2. Fetch the compressed bytes in one go. When isAsync is true it uses asyncGetBytes, otherwise the synchronous getBytes. Both return the complete data, so the upstream is never read again afterwards.
3. Construct the DecompressionStream. name is a format string such as "deflate" or "brotli", and the browser's decompression implementation picks the algorithm.
4. Wait for writer.ready, then write. Writing applies backpressure: when nobody is consuming the readable side, write does not complete. So the code deliberately does not await the write; it starts writing and immediately enters the read loop. The write chain completes naturally once the consumer starts, and closes the writer when done.
5. Collect and concatenate the output. for await yields decompressed slices one by one, the total length is accumulated, and a single Uint8Array is allocated and filled. Compared with pushing and growing repeatedly, this concatenation allocates only once.
6. Keep the compressed bytes on failure. The whole flow sits inside try; if DecompressionStream does not exist, the format is unsupported, or a stray byte at the end of the data breaks decompression, the method returns decompressed: null and hands back the compressed bytes it already has.

Step 6 is what makes this fast path safe to use. After receiving the empty result, the caller can fall back to its own JavaScript decoder without reading upstream again — the compressed bytes are already in hand.

FlateStream and BrotliStream do exactly that:

• On success, return the decompressed result directly.
• On failure, set #isAsync to false, build an ordinary Stream from compressed (Flate skips the two-byte header), reset() it and run their own incremental decoder; asyncGetBytes returns null, while their overridden getImageData switches to getBytes(length) when it gets an empty result.

Design note: A native fast path must be able to fail, able to fall back, and able to reuse its input. asyncGetBytesFromDecompressionStream builds all three into its return value: failure is expressed as null, the compressed bytes needed for fallback come back with the result, and reading the upstream from the start guarantees complete input. If any of these were missing, fallback would turn into a second read or into polluted state.

Substreams and Clones: Materialising Decoded Output

makeSubStream

  makeSubStream(start, length, dict = null) {
    if (length === undefined) {
      while (!this.eof) {
        this.readBlock();
      }
    } else {
      const end = start + length;
      while (this.bufferLength <= end && !this.eof) {
        this.readBlock();
      }
    }
    return new Stream(this.buffer, start, length, dict);
  }

There is no laziness in DecodeStream's makeSubStream: decoding must advance to the substream's end before an ordinary Stream can be constructed.

• When length is undefined, decode all the way to eof, then wrap the span from start to the end of the buffer as a stream.
• When length is given, decode until the buffer covers end, then construct [start, start + length).

The returned Stream shares the same underlying Uint8Array as the decoding stream and copies no data. It is effectively a snapshot of the decoded output: even if the decoding stream later grows again or swaps its array, the substream still holds the old buffer and what it sees never changes.

This is the exact opposite of the "live view" provided by ChunkedStream.makeSubStream:

clone

  clone() {
    // Make sure it has been fully read.
    while (!this.eof) {
      this.readBlock();
    }
    return new Stream(this.buffer, 0, this.bufferLength, this.dict?.clone());
  }

clone is even more demanding: the entire stream must be read. The new Stream it returns starts at 0, has a length equal to bufferLength, shares the buffer with the decoding stream, but gets its own read cursor. The dictionary is also shallow-cloned, so the two streams do not share one mutable dictionary.

Why read the whole stream? Because Stream needs a definite end, and the decoding stream's eof only becomes true once decoding truly finishes. Cloning after reading only part of it would give the new stream a temporary length that subsequent decoding would never reflect.

clone and makeSubStream cost the same: calling them completes the remaining decoding immediately. They suit the case where "the parser needs to store a stream and read it again from the start later", and are a poor fit for frequent calls that only read a small span.

StreamsSequenceStream: Joining Multiple Streams

Construction

class StreamsSequenceStream extends DecodeStream {
  constructor(streams, onError = null) {
    streams = streams.filter(s => s instanceof BaseStream && !s.isImageStream);
 
    let maybeLength = 0;
    for (const stream of streams) {
      maybeLength +=
        stream instanceof DecodeStream
          ? stream._rawMinBufferLength
          : stream.length;
    }
    super(maybeLength);
 
    this.streams = streams;
    this._onError = onError;
  }

Construction filters first, then estimates capacity:

• Only BaseStream instances are kept, and image streams are excluded along the way. Image stream data is usually handled separately as a transferable object, so mixing it into text stitching makes no sense.
• Capacity estimation adds a DecodeStream's _rawMinBufferLength to other streams' length. The former is the raw hint from construction, the latter a known length; both are estimates, and actual growth is still the job of ensureBuffer.

The estimate reaches minBufferLength through super(maybeLength), so the stitching stream starts with a capacity close to its final size and needs fewer intermediate growth steps.

readBlock

  readBlock() {
    const streams = this.streams;
    if (streams.length === 0) {
      this.eof = true;
      return;
    }
    const stream = streams.shift();
    let chunk;
    try {
      chunk = stream.getBytes();
    } catch (reason) {
      if (this._onError) {
        this._onError(reason, stream.dict?.objId);
        return;
      }
      throw reason;
    }
    const bufferLength = this.bufferLength;
    const newLength = bufferLength + chunk.length;
    const buffer = this.ensureBuffer(newLength);
    buffer.set(chunk, bufferLength);
    this.bufferLength = newLength;
  }

Each readBlock handles exactly one substream:

1. If the list is empty, set eof and stop.
2. shift a stream off the front and take all of its remaining bytes. getBytes() without a length means the whole substream is read in one go.
3. On error, if there is an onError callback, hand over the error and the object number and return immediately; that stream is dropped, and the next readBlock moves on to the following one.
4. On success, append chunk to the end of the output buffer and update bufferLength.

shift ensures each substream is consumed exactly once, which also means stitching is "read and discard": a stream that has been merged into the output is referenced no more. getBytes() reads each substream in full, so the stitching stream's memory grows with the output rather than retaining every substream.

The error behaviour deserves its own table:

The presence of onError means one bad stream inside a page's content streams does not bring down the whole page. When PDFDocument assembles page content, it passes in an error handler, reports the object number of the failing substream, and carries on stitching the rest.

  getBaseStreams() {
    const baseStreamsBuf = [];
    for (const stream of this.streams) {
      const baseStreams = stream.getBaseStreams();
      if (baseStreams) {
        baseStreamsBuf.push(...baseStreams);
      }
    }
    return baseStreamsBuf.length > 0 ? baseStreamsBuf : null;
  }

getBaseStreams iterates over substreams that have not yet been consumed. Streams already shifted out no longer participate, because their bytes are already part of the output and their original source no longer means anything to the layer above. When there is no traceable source it returns null, matching the base class's default semantics.

readBlock Strategies of the Subclasses

StreamsSequenceStream is only one of many implementations. Putting the common subclasses side by side shows how elastic the readBlock contract is:

Design note: The stitching stream folds "many streams" into "one stream" by treating substreams as data blocks instead of maintaining many cursors. Each substream is discarded as it is consumed, and the output buffer keeps only the final result. The parser above never needs to know that the original page content was split into several pieces.

Conclusion

decode_stream.js is PDF.js's decoding foundation: upward it turns the various filters into contiguous bytes, and downward it reuses the reading and derivation semantics of Stream.

Where It Sits in the Overall Chain

In one sentence: the byte source supplies encoded data, DecodeStream decodes it into a buffer through the uniform readBlock contract, and the parser turns that buffer into objects.

Five Design Principles

1. One abstraction point; subclasses write only readBlock. The base class knows no algorithm and handles only buffering, reading and derivation, so more than a dozen filters can share one set of read semantics.
2. Capacity and length are separate, with geometric growth. buffer.byteLength manages memory, bufferLength manages valid data; growth doubles from minBufferLength, and the amortised copy cost is constant.
3. Reads return views with zero copying. getBytes returns a subarray directly, and makeSubStream and clone share the underlying buffer; when an independent cursor is needed, a new Stream wrapper is created without copying bytes.
4. Asynchronous decoding comes first, with a fallback on failure. getImageData prefers the asynchronous path, and when DecompressionStream fails, the compressed bytes are handed back to the caller so a JavaScript decoder can take over without a second read.
5. Materialise before deriving. makeSubStream and clone both read all the data they need before handing it to an ordinary Stream; the stitching stream swallows one substream at a time. Laziness stays at the "decode chunk by chunk" level and never spreads into derived objects.

This wraps up the two articles on "Bytes and Streams": chunked_stream turns network bytes into an addressable buffer, and decode_stream turns encoded bytes into a decoded buffer. The next article returns to parser.js to see how Lexer cuts tokens out of that buffer, and how Parser assembles those tokens into PDF objects.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.