PDF.js Source Walkthrough: stream

Introduction

This is the second article in the "bytes and streams" layer of the PDF.js source walkthrough series. It covers the three classes in src/core/stream.js (97 lines): Stream, StringStream and NullStream.

Looking back, article 10 on XRef and article 11 on Parser have already covered object retrieval and syntax parsing; both assume there is a stable byte source underneath, and the contract of that source is defined by the previous article on base_stream.js: what getByte, getBytes, peekByte, reset, makeSubStream, clone and the rest should and should not do. This article gives the most basic implementation of that contract: a cursor-bearing window over a run of bytes that already lives in memory.

Stream is the most frequently occurring stream type in the repository. LocalPdfManager wraps the whole file in one; Parser.makeStream uses makeSubStream to cut the stream data of each 12 0 obj out of its parent; XRef.fetchUncompressed uses it to locate objects as well. StringStream and NullStream are two specialised subclasses only a few lines long, solving the two concrete problems of "parse a string as a stream" and "use an empty stream as a placeholder".

In one sentence: Stream does no downloading, no decompression and no decryption; it maintains only three pointers, start, pos and end, and turns every read into an index into a Uint8Array and a subarray view.

Module Overview

stream.js consists of three classes; the file is short, yet the responsibilities are cleanly separated:

Stream overrides most of the abstract members of BaseStream, but not peekByte, peekBytes, getUint16, getInt32, getString or skip. All of those are built on getByte, getBytes and a writable pos, so the parent class defaults are sufficient.

The Three-Pointer Model

Stream has no extra state such as a "buffer offset". The whole read process is described by three numbers: start is the origin of the logical window, pos is the current cursor, and end is the end of the logical window. All three are absolute indices into the underlying bytes array, and the window is the half-open interval [start, end).

constructor

  constructor(arrayBuffer, start, length, dict) {
    super();
 
    this.bytes =
      arrayBuffer instanceof Uint8Array
        ? arrayBuffer
        : new Uint8Array(arrayBuffer);
    this.start = start || 0;
    this.pos = this.start;
    this.end = start + length || this.bytes.length;
    this.dict = dict;
  }

The first part handles the input shape. A Uint8Array is used as is, without copying; an ArrayBuffer is wrapped in a view with new Uint8Array(arrayBuffer). Both cases share the same memory, and the only difference is whether the caller already holds a typed array.

The second part establishes the three pointers. start is normalised with start || 0, so undefined, NaN and 0 all land on 0; pos initially equals start; end is computed as start + length || this.bytes.length. As long as length is undefined or NaN, the addition yields NaN and the expression falls back to the length of the whole byte array.

There is an easy-to-miss edge here: when length is 0 and start is also 0, start + length is 0, which is falsy, so it likewise falls back to bytes.length. In other words, explicitly creating a zero-length stream at offset 0 actually yields the whole data. In this repository, the empty stream is NullStream's job, and StringStream uses NaN for an unknown length, so the normal parsing paths never hit that combination; still, a reader of the source should know that the constructor does arithmetic only, with no semantic validation.

dict is stored by reference. A filter chain, an encrypted stream and its substreams may share the same dictionary object; when an independent dictionary is needed, clone makes the copy.

end is not clamped to bytes.length. The constructor trusts the length supplied by the caller; if that length exceeds the underlying array, getByte may read out undefined. On real paths, Parser corrects a wrong length by scanning for endstream, and the start and length given to makeSubStream come from offsets that have already been validated, so this trust is under control.

A Stream usually passes through only three stages from creation to disposal:

1. Establish the window. PDFManager builds the top-level stream from the whole file; Parser computes the startPos and length of a stream object and then builds a child window with makeSubStream. This stage writes only start, pos and end; no data is copied.
2. Advance the cursor. Lexer and Parser move forward through getByte, getBytes and skip; pos grows monotonically while end stays fixed. Only getBytes creates a lightweight view object, and the bytes themselves never move.
3. Reset or derive. Parsing the dictionary and the data of the same stream object may require several reads, so reset pulls pos back to start; writing or editing uses clone to copy the window description; keeping a position for later uses makeSubStream to open a new window.

These three stages map to three groups of methods: the constructor handles the first, reading and skip the second, and reset, moveStart, makeSubStream and clone the third. No hidden state is passed between them — at any moment three numbers are enough to reconstruct the stream's full position.

Design note: Describing the window with absolute indices rather than "relative offset plus a base address" reduces the read logic to one comparison and one increment. getByte need not compute start + pos first, getBytes need not maintain an extra read offset, and every derived stream needs to copy only three numbers. The less state there is, the less likely nested streams are to slip out of alignment.

length and isEmpty

  get length() {
    return this.end - this.start;
  }
 
  get isEmpty() {
    return this.length === 0;
  }

length is the total length of the window, not the remaining length. After the whole stream has been read, pos equals end and length is still the original value; whether the stream is exhausted must be judged by pos, not by length.

isEmpty likewise looks only at the window, not at the cursor. A stream of length 10 that has been read to the end is not an empty stream, just an exhausted one; a truly empty stream is one where start and end are equal. The repeated isEmpty checks in BaseStream and DecodeStream all rest on this distinction.

moveStart changes start, so length shrinks with it. This getter is not a cached read-only snapshot but a live computation over the current window.

Sequential Reading and Zero Copy

getByte

  getByte() {
    return this.pos >= this.end ? -1 : this.bytes[this.pos++];
  }

This is the hottest path in the whole stream. It compares pos with end and returns -1 once the end is reached; otherwise it reads bytes[pos] and increments pos. Because end is excluded from the window, the last readable byte has index end - 1.

-1 carries both meanings at once, "end of file" and "no more data". PDF bytes range from 0 to 255, so a negative number can never be a legal byte and the sentinel cannot collide with real data. Lexer loops can therefore test only for -1, without an extra length query.

The method does not check whether pos has gone past bytes.length. It trusts end alone, which is also why Stream is faster than DecodeStream: no buffer growth, no decoding state, no missing-chunk check — just one array index.

getBytes

  getBytes(length) {
    const pos = this.pos;
    const endPos = !length ? this.end : Math.min(pos + length, this.end);
 
    this.pos = endPos;
    return this.bytes.subarray(pos, endPos);
  }

It saves the old cursor first, then computes the new end point. When !length is true it takes the end of the window, so both "no argument" and "0" mean read everything that remains; reading a fixed length requires a positive integer. Math.min does the truncation: when the requested length exceeds what remains, only the part up to end is returned.

subarray is the key to zero copy. It returns a new Uint8Array view over exactly the same underlying ArrayBuffer as the parent array, differing only in byteOffset and length. Reading a content stream of several hundred kilobytes therefore costs no extra allocation and no copy; Lexer, Parser and bytesToString all consume the view directly.

Zero copy brings two constraints that must be remembered:

1. The returned array's buffer is the whole underlying buffer, not a buffer dedicated to this slice. After taking a view, do not assume buffer.byteLength equals bytes.length, and do not index into that data through buffer directly.
2. The view shares memory with the stream. Modifying the returned array writes into the stream's original bytes; when an independent copy is required, call slice explicitly.

pos has already advanced to endPos before the method returns, so a caller that immediately calls getByte again continues from the new position. If the requested length was truncated at the end, the cursor advances by the actual length, and that is exactly what peekBytes relies on to keep its position accurate.

The reading styles side by side:

Design note: subarray removes copying from the read path entirely. During PDF parsing the same content is often read several times: Parser tries it first, Lexer reads it properly, and Evaluator consumes it operator by operator. If every read called slice, large files would generate a flood of short-lived arrays; returning a view costs only one object header.

getByteRange

  getByteRange(begin, end) {
    if (begin < 0) {
      begin = 0;
    }
    if (end > this.end) {
      end = this.end;
    }
    return this.bytes.subarray(begin, end);
  }

The difference from getBytes is not in complexity but in semantics: the range consists of absolute indices into the underlying bytes, independent of the current pos, and the cursor does not move after the call.

The two clamps handle only the most common directions. A negative begin is pulled to 0; an end past the end of the logical window is clamped to this.end. It does not force begin to come before end, nor end to be non-negative; subarray handles those cases itself, producing an empty or a shorter view.

Note that the upper clamp is this.end, not bytes.length. That shows it still respects the current logical window: even when more bytes exist underneath, nothing past end is returned. Conversely, it does not raise begin to this.start; passing an absolute index earlier than the window origin still reads bytes from before the window. That is the natural consequence of the "absolute range" semantics.

ChunkedStream overrides this method: it performs the same clamps, then calls ensureRange on the part that has not been downloaded, throwing MissingDataException when data is missing. The interface seen from above is unchanged, but the read may turn into a network request.

Cursor and Window

reset and moveStart

  reset() {
    this.pos = this.start;
  }
 
  moveStart() {
    this.start = this.pos;
  }

reset returns the cursor to the logical origin, not to array index 0. A stream that has been moveStarted to the file header resets back to the header, not to the very start of the file. It changes only pos; it does not clear bytes and does not modify dict, so it can be called any number of times.

moveStart pushes the logical origin to the current cursor. pos stays put, start grows, end is unchanged, and so length shrinks. The most typical use is in PDFDocument.checkHeader: call reset, use find to locate %PDF-, then moveStart to exclude any whitespace, BOM or junk bytes at the start of the file from the logical window; after that, skip past the header signature and read the version number.

Judging by the public methods, moveStart can only narrow the window; it cannot pull a discarded prefix back in. A later reset returns the cursor to the new start, that is, to the header, not to the original start of the file.

skip is not overridden in Stream; it uses the BaseStream default, this.pos += n || 1. It does not check bounds, and a negative number moves backwards; Parser uses skip(-2) while scanning an inline JPEG image to step back from a bad length to the start of the marker. Because skip writes pos directly, using it together with moveStart requires the caller to know the cursor's absolute position.

Boundary Behaviour and Common Misuse

Stream is short, but its boundary semantics are numerous. To a caller all of the following look like "the same method", yet the actual outcomes differ substantially:

The first few rows all describe "silent truncation": the method neither throws nor returns an error code, only a shorter result. Streaming parsers must work this way — data boundaries are often decided by corrupt files or wrong lengths, and throwing would make the whole document impossible to open; returning the usable part at least lets the scan continue.

One distinction worth emphasising is "read to the end" versus "empty stream". pos reaching end only says the current window has been read; length is still non-zero and isEmpty is false. Only when start and end are equal is the stream empty. Many parsing loops terminate on the -1 from getByte rather than testing isEmpty, precisely because the two mean different things.

The last two rows concern ownership. Zero-copy views make reading free of allocation, but they also make modification contagious; before handing bytes to something that will modify them, slice out an independent copy. The same holds for the shallow-cloned dictionary from clone: changing a top-level key is safe, but changing a nested object still affects the original document.

Derived Streams: makeSubStream and clone

makeSubStream

  makeSubStream(start, length, dict = null) {
    return new Stream(this.bytes.buffer, start, length, dict);
  }

This is the single most important optimisation in Stream, and the key to understanding PDF.js's memory model. It copies no bytes, calls no slice, and does not even reuse this.bytes; it hands this.bytes.buffer to the constructor as a whole, and the new Stream wraps it in a fresh view through new Uint8Array(arrayBuffer). The new stream shares the same ArrayBuffer with its parent and owns only its own start, pos and end.

start is an absolute index into the underlying buffer, not an offset relative to the parent stream's start. XRef.fetchUncompressed makes this plain:

const stream = this.stream.makeSubStream(xrefEntry.offset + this.stream.start);

The offset in an xref entry is relative to the file header, while makeSubStream wants an offset into the underlying buffer, hence the addition of this.stream.start. The startPos passed by Parser.makeStream is likewise the absolute position of the Lexer cursor, not a difference from the parent window.

dict is passed by reference and not cloned. After cutting out the stream data, the parser sets stream.dict = dict, so parent and child share one dictionary object; that matches PDF semantics, where a substream is simply the data window of that stream object itself.

When length is omitted, the new stream's end falls back to the total length of the underlying buffer, not to the parent's end. For the top-level file stream the two values are usually the same; if the parent is itself a child window, they may differ. Callers should either pass the length explicitly or know that they intend to read to the end of the buffer.

There is one implicit premise as well: this.bytes is usually a full view starting at buffer index 0, in which case the coordinate systems of bytes.buffer and bytes coincide. If this.bytes is itself a subarray with a non-zero byteOffset, then pairing bytes.buffer with absolute indices shifts the coordinate system by the sub-view's origin. The repository's main paths satisfy the premise; the temporary Stream built from peekBytes is used only for small trial reads and never takes this derivation path.

makeSubStream is O(1). A PDF of tens of megabytes may contain thousands of stream objects, and copying the data of each one would crush memory immediately; sharing the underlying buffer and building only an object header is what keeps "parse one stream object" a constant cost.

Design note: What substreams share is bytes, not state. bytes.buffer keeps a single copy in memory, while start, pos and end give every stream its own window and cursor. The same content can be held at the same time by Parser, Evaluator and a font parser, each resetting and reading independently without disturbing the others.

clone

  clone() {
    return new Stream(
      this.bytes.buffer,
      this.start,
      this.length,
      this.dict?.clone()
    );
  }

clone follows the same optimisation route as makeSubStream: no bytes are copied, only a new object is created. The difference is that it copies the current window, preserving start and length as they are instead of specifying a new origin and length. The new stream's pos is initialised to start by the constructor, so a half-read original does not affect the copy.

The dictionary is where the two differ most. makeSubStream reuses the dict it receives, while clone calls this.dict?.clone(). Dict.clone in primitives.js performs a shallow clone: it creates a new Dict, keeps the same xref, and copies the original's key-value pairs one by one. The dictionary object itself is new, so top-level additions and removals do not affect the original; the values still point at the original objects, arrays and references.

This design serves writing and editing. When writing a PDF, entries such as Length, Filter and DecodeParms may need to change, or a stream may need a different dictionary; editing the original directly would break the semantics of the document already loaded. Cloning one layer of the dictionary isolates the mutable shell while avoiding a deep copy of the whole object graph.

The optional chain this.dict?.clone() handles streams without a dictionary. NullStream and temporary Stream instances have no dictionary, so cloning them yields undefined directly, without throwing.

Like makeSubStream, clone uses this.bytes.buffer and therefore carries the same implicit premise that bytes starts at buffer index 0. One more thing to know: clone always returns a Stream and does not preserve the subclass. ChunkedStream inherits this implementation, so cloning it produces a plain Stream with no chunk manager and no missing-range checks; that suits data that is already complete and only needs an independent cursor.

Compared With Other Stream Implementations

Putting Stream, DecodeStream and ChunkedStream side by side reveals three ways of achieving "shared bytes, independent cursor" over different data sources:

Together these three implementations prove one point: BaseStream's abstraction is not "different wrappers around one kind of storage" but "different data sources behind one reading semantics". Lexer cares only whether getByte returns the next byte, and not at all whether that byte came from a local array, a decompression buffer, or a chunk that was just downloaded.

Two Special Streams

StringStream

class StringStream extends Stream {
  constructor(str, dict = null) {
    super(stringToBytes(str), NaN, NaN, dict);
  }
}

StringStream turns a JavaScript string into a parseable stream. stringToBytes in shared/util.js calls charCodeAt for each character, masks it with 0xff to keep the low byte, and places the result in a new Uint8Array. It assumes the string is already the single-byte representation that PDF syntax requires, which suits content stream fragments, appearance streams and Latin-1 text; genuine Unicode text must be encoded by the caller first and cannot expect this to do it automatically.

Passing two NaNs is a small trick. NaN || 0 on start becomes 0, and NaN + NaN || bytes.length on end takes the full byte array length. The two "unknowns" are absorbed by the constructor's two fallback branches, ending with a window that starts at 0 and covers every byte of the string.

dict is optional and attaches a dictionary to the synthesised stream. annotation.js builds annotation appearance streams, default_appearance.js parses the /DA string, writer.js writes XFA data, and pdf_editor.js assembles edited content; all of them construct a StringStream and then hand it to the same Lexer, Parser and Evaluator. Callers need no special parsing path for "a string in memory".

The use in default_appearance.js is the most direct: an evaluator responsible for parsing default appearances only needs to wrap the string in a StringStream in its constructor:

  constructor(str) {
    super(new StringStream(str));
  }

The evaluator itself does not know the concrete type of BaseStream; it relies only on the reading and reset interfaces provided by the base class. StringStream disguises "a string" as "a stream", so a /DA string and a page content stream run exactly the same parsing code. That is precisely why an abstract base class exists: the data source can be swapped without branches in the parser.

NullStream

class NullStream extends Stream {
  constructor() {
    super(new Uint8Array(0));
  }
}

NullStream is a zero-length stream. The constructor passes an empty array; start lands on 0, and end falls back to bytes.length because start + undefined is NaN, which is also 0. The result: length is 0, isEmpty is true, getByte always returns -1, and getBytes returns an empty view.

Its value lies not in reading but in standing in. When Parser.makeFilter meets a stream whose Length is 0, or one whose filter name is unsupported or whose decoding throws, it warns and then returns a NullStream; Page.getContentStream also returns a NullStream when a page has no content stream at all. This way Evaluator, OperatorList and the layers above need no null check at every entry point, and an empty stream naturally produces zero operators.

The two most direct entry points in the source are:

if (maybeLength === 0) {
  warn(`Empty "${name}" stream.`);
  return new NullStream();
}

and the exception fallback of the filter chain:

    } catch (ex) {
      if (ex instanceof MissingDataException) {
        throw ex;
      }
      warn(`Invalid stream: "${ex}"`);
      return new NullStream();
    }

The first treats only "length is 0" as definitely empty; the comment points out that the Length in a stream dictionary is often unreliable, so any non-zero value must not be taken as an empty stream, which would misjudge non-empty data. The second degrades a decoding failure to an empty stream, but it must let MissingDataException through first: that is not bad data but chunks that have not arrived over the network, so the layer above must request them and the exception cannot be swallowed.

This is the concrete use of the null object pattern in the parser: representing "no data" with an object that satisfies the full BaseStream contract is more stable than scattering if (stream) checks everywhere; even if some caller forgets to check, nothing crashes on null.getByte.

Sharing and Copying at a Glance

Putting every operation in the file that creates a new object or new bytes into one table makes the memory relationships clear:

The only entry that truly allocates bytes is StringStream; NullStream allocates just an empty array; every other operation opens a view over an existing buffer. This distribution determines PDF.js's memory curve: when parsing a large file, memory is dominated by the original file and the decode buffers, not by thousands of stream objects each holding a copy of the data.

Design note: Copying happens only when "the shape of the data changes", for example a string becoming bytes or compressed data becoming a decode result. As long as the shape stays the same, only the description is copied — that is, the pointers. Separating "allocation" from "reading" is the premise that lets streaming PDFs keep memory low on large files.

Conclusion

Stream is PDF.js's in-memory byte window: it wraps a Uint8Array in a stream with start, pos and end, offering sequential reads, range reads, reset, derivation and cloning. StringStream and NullStream cover the two edges of "synthesised content" and "empty content" with the least code.

Where It Sits in the Overall Chain

In one sentence: PDFManager turns the file into a Stream, Parser uses makeSubStream to cut the file into object-level windows, DecodeStream then turns compressed bytes into readable bytes, and finally StringStream and NullStream fill in the boundaries of synthesised content and empty content.

Five Design Principles

1. The three pointers are the only truth. start, pos and end determine the window, the cursor and the length, with no hidden buffer offset. Every reading method derives from these three numbers, and derived streams copy only these three numbers.
2. Views first, copying as a fallback. getBytes, getByteRange and makeSubStream all return or share subarray views, and new bytes are allocated only when the shape of the data changes. What is saved is the allocation and copy on every read, not readability.
3. Substreams share memory, cursors stay independent. makeSubStream builds an O(1) child window from bytes.buffer; parent and child share the bytes but maintain their own pos. That lets one file be read by several parsers at the same time.
4. Cloning isolates only the mutable shell. clone shares the underlying buffer, copies the window description and shallow-clones the dictionary. Bytes are read-only and the dictionary is mutable; isolating the layer that will actually be changed avoids a deep copy of the whole object graph.
5. Special streams solve their problem with minimal code. StringStream and NullStream are only a few lines each, yet they let "string content" and "empty content" reuse the existing parsing chain completely. Callers need no branches for these edge cases.

StreamPDF design principle: anything that can be handled ahead of time is absolutely not deferred to parse time; the server solves all performance problems once. The browser only fetches and renders the data it must.