PDF.js Source Walkthrough: base_stream

Introduction

This is the first article in the PDF.js source walkthrough series to reach the "bytes and streams" layer. It covers the BaseStream class in src/core/base_stream.js (168 lines).

Article 10 on XRef already explained how objects are located and retrieved, and article 11 on Parser explained how bytes are tokenized and then assembled into Dict, arrays and stream objects. Both articles stand at the height of "objects" and assume there is a stable byte source underneath. This article fills in that implicit premise: what exactly does a stream promise the parser.

BaseStream itself does almost no actual reading work. It is an abstract base class written in plain JavaScript: it defines the names and semantics of getByte, getBytes, reset, makeSubStream and the rest, and leaves the implementation to three kinds of subclass:

• Stream: opens a window over an ArrayBuffer that already lives in memory;
• DecodeStream: decodes compressed or encrypted bytes while buffering them;
• ChunkedStream: loads network chunks on demand, and throws MissingDataException when a read touches a range that has not been downloaded.

In one sentence, its job is to be the single facade for every byte source, so that Lexer, Parser, XRef and PartialEvaluator never need to know whether they received a local file, a network chunk, or a Flate decompression result.

Design note: PDF.js does not use TypeScript abstract classes. Instead, the abstract methods call unreachable, which throws. The contract is therefore not a convention written in documentation but a failure point that fires at runtime: when a subclass forgets a method, it does not silently return undefined and push the error downstream.

Module Overview

base_stream.js is only 168 lines and splits into eight blocks by responsibility:

Only peekByte, peekBytes, getUint16, getInt32, getString, skip, getImageData and a few getters have default implementations; everything else is abstract. All the defaults rest on two smaller primitives: getByte, getBytes, and a writable pos field.

Construction and the Abstract Contract

BaseStream never declares pos, yet the default implementations of peekByte and skip use this.pos directly. This is an implicit contract: a subclass must provide an integer cursor that starts at 0 and is both readable and writable. JavaScript needs no field declarations, but a reader of the source has to keep this in mind.

constructor

  constructor() {
    if (
      (typeof PDFJSDev === "undefined" || PDFJSDev.test("TESTING")) &&
      this.constructor === BaseStream
    ) {
      unreachable("Cannot initialize BaseStream.");
    }
  }

PDFJSDev is a constant injected at build time, so this condition only holds in a testing context; this.constructor === BaseStream then distinguishes "newing the base class directly" from "a subclass calling super()". Direct instantiation throws immediately, while subclass construction is unaffected.

The implementation of unreachable is a single line: throw new Error(msg). It serves two purposes at once: it marks "this should never run" for anyone reading the code, and it leaves a clear stack trace in tests or on abnormal paths. In production this construction check is stripped by preprocessing, so no extra check remains on every stream creation.

Design note: What an abstract base class fears most is not "not implemented" but "not implemented and yet returned a value that looks usable". undefined turns into NaN in arithmetic and quietly compares as false, so the error often surfaces hundreds of lines later. Throwing at the call site keeps the distance between contract violation and detection as short as possible.

length and isEmpty

  get length() {
    unreachable("Abstract getter `length` accessed");
  }
 
  get isEmpty() {
    unreachable("Abstract getter `isEmpty` accessed");
  }

length is the stream's total valid byte count, not "how many are left". Stream implements it as end - start; DecodeStream does not override the getter, because a decode stream makes no promise to know its final length up front, so callers can only read on demand.

isEmpty means "is this stream empty as a whole", independent of the current cursor position. Stream simply checks whether length is zero; DecodeStream is more proactive: as long as the end of the file has not been reached and the buffer is empty, it keeps calling readBlock until data arrives or the end is confirmed.

Both getters carry // eslint-disable-next-line getter-return, because the body only throws and never returns a value. That is deliberate, not an oversight: it keeps "not implemented" distinct from "returns empty".

Raw Reading Interface

getByte and getBytes

  getByte() {
    unreachable("Abstract method `getByte` called");
  }
 
  getBytes(length) {
    unreachable("Abstract method `getBytes` called");
  }

The contract of getByte is: return a byte from 0 to 255, or -1 at the end of the stream. It does not throw, and it does not use null to signal the end; parser loops can therefore be written as "stop at -1" without any extra check.

getBytes(length) returns a Uint8Array and advances the cursor. length is optional and its exact meaning is up to the subclass, but the implementations in this repository agree: omitted or 0 means read through end, a positive number means read at most that many, and near the end the result is truncated automatically. Callers must look at the returned array's length and must not assume it equals the requested value.

Another contract that never made it into the signature is that the return value may be a view. Stream returns bytes.subarray(...) directly, with zero copy; DecodeStream returns a subarray of its internal decode buffer. Callers therefore should not modify the returned array casually, or they will write into the stream's underlying data.

getImageData and asyncGetBytes

  async getImageData(length, decoderOptions) {
    return this.getBytes(length, decoderOptions);
  }
 
  async asyncGetBytes() {
    unreachable("Abstract method `asyncGetBytes` called");
  }

getImageData is an async facade whose default merely wraps the synchronous getBytes in a Promise. The comment is explicit: it may only be used for image data that is guaranteed to be fully loaded, since otherwise a half-read network stream can produce intermittent errors; ObjectLoader exists precisely as the preloader for that case.

decoderOptions is passed on to getBytes. The getBytes signature on BaseStream only mentions length, and extra arguments are not an error in JavaScript; DecodeStream takes advantage of that in its override, handing the options to the concrete image decoder.

asyncGetBytes is abstract by default. Only streams that can use the platform's async decoding override it, such as FlateStream and BrotliStream with the native DecompressionStream. Returning null means "the async path failed, fall back to synchronous decoding"; callers must not treat null as empty data.

getByteRange

  getByteRange(begin, end) {
    unreachable("Abstract method `getByteRange` called");
  }

It reads a run of bytes over [begin, end) without moving the cursor. Unlike getBytes, the range is absolute positions in the underlying buffer, not positions relative to the current pos.

The Stream implementation clamps a negative begin to 0, clamps an end beyond this.end to the end of the stream, and then returns a subarray. ChunkedStream also calls ensureRange first to confirm that the range has been downloaded, and otherwise throws MissingDataException so the caller requests the data.

The method mainly serves two cases: the document fingerprint reads a fixed number of bytes at the start of the file, and signature validation reads the signed content in pieces according to the ByteRange in the signature dictionary. Neither should disturb the current parse cursor.

Comparing the Reading Semantics

BaseStream's default implementations chain several methods together, so knowing whether each one moves the cursor matters more than remembering how it is written:

getString(length) is a one-liner: bytesToString(this.getBytes(length)). Internally, bytesToString calls String.fromCharCode in batches of 8192 arguments, so that a very long byte array does not hit the argument-count limit. It is a low-byte-to-character mapping, not a general Unicode decoder.

Lookahead and Multi-byte Reads

peekByte

  peekByte() {
    const peekedByte = this.getByte();
    if (peekedByte !== -1) {
      this.pos--;
    }
    return peekedByte;
  }

Lookahead does not need a separate reading path; it is "read one, then step back one". Every subclass that implements getByte and a writable pos therefore gets lookahead for free.

The step back only happens when a real byte was read. If the cursor is already at end, getByte returns -1 and pos never advanced, so subtracting one would move before the end; the -1 check is therefore required.

This implementation also explains why pos is part of the implicit contract: lookahead, skip and reset all operate on it directly, instead of going through separate tell and seek methods.

peekBytes

  peekBytes(length) {
    const bytes = this.getBytes(length);
    this.pos -= bytes.length;
    return bytes;
  }

Here the rewind uses the number of bytes actually returned, not the requested length. Near the end of the file getBytes truncates the result, and bytes.length is exactly the distance the cursor moved, so the position stays accurate after lookahead.

It relies on the same contract: what getBytes returns must match how far the cursor advanced. Stream, DecodeStream and ChunkedStream in this repository all satisfy it; a subclass that returned fewer bytes than the cursor advanced would misalign lookahead.

Parser uses it for two typical jobs: reading the bytes after an inline image's terminator to check whether the following bytes are still visible ASCII, and wrapping a small run of bytes in a temporary Stream for a fresh Lexer to try. Both only look and never consume, which is exactly what lookahead is for.

getUint16

  getUint16() {
    const b0 = this.getByte();
    const b1 = this.getByte();
    if (b0 === -1 || b1 === -1) {
      return -1;
    }
    return (b0 << 8) + b1;
  }

It assembles a 16-bit unsigned integer in big-endian order: the first byte is the high 8 bits, the second the low 8. PDF, JPEG and TrueType use this byte order for a great many length fields, which is why no little-endian variant is provided.

A failure returns -1, matching the end sentinel of single-byte reads. Note that if the first byte was read but the second hits the end, the cursor has already advanced by one and the method does not roll back. Callers usually stop parsing as soon as they see -1, so this side effect rarely matters, but it is real.

Parser uses stream.getUint16() while scanning an inline JPEG image to read the length of a marker segment, then skip to the next marker; that is the typical use of the method.

getInt32

  getInt32() {
    const b0 = this.getByte();
    const b1 = this.getByte();
    const b2 = this.getByte();
    const b3 = this.getByte();
    return (b0 << 24) + (b1 << 16) + (b2 << 8) + b3;
  }

It assembles a 32-bit signed integer in big-endian order. JavaScript's bitwise operators convert their operands to 32-bit integers first, so shifting the highest byte left by 24 gives the result its sign naturally; four bytes of 0xff produce -1.

Unlike getUint16, it has no EOF check. The reason is practical: -1 is itself a legal 32-bit result, so it cannot double as an end sentinel. Callers must guarantee that four bytes are readable, through getByteRange or chunked loading, or convert the result to an unsigned offset immediately, as fonts.js does when reading a table directory.

Cursor Control and Derived Streams

skip and reset

  skip(n) {
    this.pos += n || 1;
  }
 
  reset() {
    unreachable("Abstract method `reset` called");
  }

skip advances one byte by default, and a negative n moves backwards. It performs no bounds check and does not care whether it is skipping whitespace, a delimiter or binary data; callers must guarantee a legal landing point. Another detail is that n equal to 0 still advances by 1, because 0 || 1 evaluates to 1. Nothing in the repository uses skip(0) to mean "stay put", so do not read it as a safe no-op.

reset puts the cursor back at the logical origin. Stream goes back to start; DecodeStream goes back to 0. A reset only moves the cursor — it does not clear the decoded buffer and does not reset eof — so it suits "parse the same decoded data again" rather than "decode again".

moveStart

  moveStart() {
    unreachable("Abstract method `moveStart` called");
  }

moveStart is the rarest abstract method in BaseStream; only Stream implements it: this.start = this.pos. It pushes the start of the logical window to the current cursor, leaves pos alone, and length shrinks accordingly.

Its real use is in PDFDocument.checkHeader: the start of a file may contain whitespace, a BOM or junk bytes; after find locates %PDF-, moveStart makes every later offset relative to the real header. Only then does skip(PDF_HEADER_SIGNATURE.length) advance inside the header.

DecodeStream does not implement the method, because a decode stream has no start semantics; it only has a decode buffer that begins at 0.

makeSubStream and clone

  makeSubStream(start, length, dict = null) {
    unreachable("Abstract method `makeSubStream` called");
  }
 
  clone() {
    unreachable("Abstract method `clone` called");
  }

Both methods return a new BaseStream, but for different purposes. makeSubStream opens a child window over the same byte source; start is an absolute offset in the underlying buffer, not an offset relative to the current stream's origin; length may be omitted to mean "all the way to the end of the parent stream"; dict defaults to null.

After Parser.makeStream parses the stream keyword it computes the data start and length, then calls makeSubStream to cut out the raw bytes; XRef.fetchUncompressed is even more direct and passes xrefEntry.offset + this.stream.start, because offsets are relative to the file header while makeSubStream wants an offset in the underlying buffer.

clone, by contrast, copies the same logical window: same start, same length, an independent cursor and a cloned dictionary. It is used when the original stream must be kept and the cursor moved separately — for example when writing out a PDF: reset, read the raw bytes, then reuse the dictionary information.

How the Subclasses Differ

Abstract methods are only names; the subclasses decide the actual behaviour. Putting the three main implementations side by side makes the boundaries of the BaseStream contract clearer:

The most notable part of this table is the three strategies for makeSubStream: Stream opens a window with zero copy, DecodeStream decodes the data completely before opening the window, and ChunkedStream copies state along the prototype chain and then verifies missing chunks on demand. They expose the same set of methods upward, at completely different costs.

Design note: BaseStream specifies only "readable, resettable, derivable", never "where the data lives". Stream keeps its bytes in an ArrayBuffer, DecodeStream keeps them in a decode buffer, and ChunkedStream keeps them in a preallocated array filled chunk by chunk. This layer of abstraction is exactly what lets Lexer remain completely unaware of network chunks.

Loading State, Async, and the Stream Chain

isDataLoaded

  get isDataLoaded() {
    return shadow(this, "isDataLoaded", true);
  }

The default assumes "the data was already complete when this stream was handed over". shadow defines a non-writable, configurable own property on the instance, so true is pinned after the first read; test builds additionally assert that a property of that name exists on the prototype, which catches typos.

ChunkedStream overrides this getter and compares the number of loaded chunks with the total. ObjectLoader uses it to decide whether walking the object graph is worthwhile: if the main stream's isDataLoaded is already true, the whole resource tree needs no early requests.

Async and Image Capability Getters

  get isAsync() {
    return false;
  }
 
  get isAsyncDecoder() {
    return false;
  }
 
  get isImageStream() {
    return false;
  }
 
  get canAsyncDecodeImageFromBuffer() {
    return false;
  }
 
  async getTransferableImage(width, height) {
    return null;
  }

These five members do not read data; they declare capabilities. All of them are false by default and subclasses override them as needed, so the layer above can pick a path with a single property read — no instanceof test and no trial call required.

• isAsync: whether the whole stream can hand over its bytes in one go through asyncGetBytes. FlateStream and BrotliStream start on the native DecompressionStream async path by default, fall back to synchronous decoding on failure and set the flag to false; Page.getContentStream waits for the async bytes first when it sees true, then wraps them in a plain Stream.
• isAsyncDecoder: whether the decoder itself is asynchronous. JpxStream, Jbig2Stream and CCITTFaxStream are true; DecodeStream.getImageData uses this to call each decoder's decodeImage.
• isImageStream: this is an image stream, not a page content stream. StreamsSequenceStream filters such streams out when it is constructed, so image bytes are never concatenated into a content sequence; Page.getContentStream uses it to exclude images as well.
• canAsyncDecodeImageFromBuffer: whether the underlying stream can supply complete bytes asynchronously so an image decoder can consume the buffer directly. The JPEG, JPX and JBIG2 implementations forward this.stream.isAsync.
• getTransferableImage: attempts to return an image object that can go straight to the rendering layer, null by default. JpegStream overrides it to decode with the browser's ImageDecoder, and still returns null when that fails, so image.js falls back to ordinary pixel decoding.

Design note: capability queries separate "is it supported" from "how is it used". The layer above asks only isAsync, without knowing whether the peer is Flate or Brotli; it asks only isAsyncDecoder, without knowing whether the peer is JPX or JBIG2. Adding a new decoder means overriding the corresponding boolean getter, and no caller branch has to change.

getBaseStreams

  getBaseStreams() {
    return null;
  }

The return value has two meanings: null says "this stream has no underlying stream that needs extra downloading", while an array says "these underlying streams may still be missing data; request ranges according to their start and end".

DecodeStream delegates the call to this.stream; ChunkedStream returns [this]; StreamsSequenceStream merges the results of all its child streams. When ObjectLoader walks the object graph, this is the interface it uses to find the streams that genuinely need network data, instead of blindly requesting the whole file.

getOriginalStream

  getOriginalStream() {
    return this.stream?.getOriginalStream() || this;
  }

The filter chain is nested one layer at a time: PredictorStream wraps FlateStream, FlateStream wraps Stream, and an encrypted document adds another DecryptStream. This method recurses down through .stream until it reaches a stream with no stream field, returning the object at the bottom that holds the original bytes.

The default also holds for itself: a plain Stream has no stream field, so it returns itself directly. DecryptStream deliberately overrides it to return this, because what lies below it is still-encrypted ciphertext, while writing a PDF back needs the decrypted bytes.

Both Writer.writeStream and the editor's #rawStreamBytes call getOriginalStream() first, then reset() and getBytes(), in order to bypass every decoder and obtain the stream's original encoded data, so it can be written back out together with its dictionary.

Conclusion

BaseStream is PDF.js's byte interface layer: downward it hides the three sources — local memory, network chunks and compressed decoding — and upward it gives Lexer, Parser, XRef and PartialEvaluator a unified set of reading, lookahead, reset and derivation capabilities.

Where It Sits in the Overall Chain

In one sentence: PDFManager supplies the bytes, BaseStream wraps those bytes in a readable cursor, Parser turns what is on the cursor into objects, and XRef together with Evaluator organises those objects into pages and drawing instructions.

Five Design Principles

1. Abstraction must fail loudly. Abstract methods do not return undefined; they call unreachable and throw. When a subclass forgets one, the error happens on the first call and never masquerades as a legal value that keeps propagating.
2. Promise reading, not storage shape. getBytes may return a view or a copy; it may return everything at once or truncate at the end. The layer above looks only at the returned array's length and never assumes the storage is an ArrayBuffer, a decode buffer or a chunked array.
3. Lookahead is reading plus rewinding. peekByte and peekBytes have no independent implementation; they reuse getByte and getBytes and then adjust pos. A subclass only has to get sequential reading right, and lookahead follows automatically.
4. Substreams share bytes, cursors stay independent. makeSubStream and clone both return new objects that share the underlying storage; each new object has its own pos, so several objects can be parsed in parallel over the same data.
5. Capability queries beat type checks. Boolean getters such as isAsync, isAsyncDecoder and isImageStream let the layer above branch on capability instead of concrete class. Adding a stream type requires no change in callers.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.