Node.js Streams and Backpressure

Incremental Data Flow

Streams move data incrementally. Their central contract is backpressure: a fast producer must slow down when the consumer cannot safely accept more data.

When a Stream Is the Right Boundary

Use streams for large files, HTTP bodies, compression, sockets, database exports, and any source that arrives over time. Streaming reduces peak memory and can lower time to first byte because processing starts before the complete payload exists.

A stream does not guarantee low memory by itself. Unbounded transforms, ignored backpressure, collecting every chunk, or oversized highWaterMark values can still exhaust memory. For small configuration files, a promise that returns one Buffer or string may be simpler.

Stream large or continuous data.
Use one-shot reads for small bounded values.
Define byte and time limits at input boundaries.

Readable, Writable, Duplex, and Transform

Readable streams produce chunks; writable streams consume them. Duplex streams do both with independent read and write sides. Transform streams are duplex streams whose output derives from input, such as compression or line conversion.

Binary mode usually emits Buffer chunks. Object mode treats each JavaScript value as one chunk and uses an object-count high-water mark. Do not switch modes accidentally; object mode can hide large objects behind a small chunk count.

Backpressure Contract

writable.write returns false when the internal queue reached its high-water mark. Stop writing and wait for drain before resuming. readable.pipe and pipeline coordinate this behavior automatically between compatible streams.

highWaterMark is a threshold, not a hard memory cap. Several pipeline stages, concurrent requests, transformed chunk sizes, and native buffers all contribute to process memory. Tune only after measuring throughput, latency, and memory together.

Respect write() Backpressure

import { once } from 'node:events';

async function writeAll(destination, chunks) {
  for (const chunk of chunks) {
    if (!destination.write(chunk)) await once(destination, 'drain');
  }
  destination.end();
}

The producer pauses whenever the writable queue asks for backpressure.

pipeline() Owns Completion

stream.pipeline connects stages, forwards errors, and destroys the pipeline when a stage fails. The promise API from node:stream/promises gives one awaitable completion point. Prefer it over manual chains of pipe calls when correctness depends on knowing whether the whole transfer succeeded.

Completion and close are different concerns. A writable finish means all data was handed to its destination; close means the underlying resource closed. Use finished when observing one stream and pipeline when owning a multi-stage transfer.

Async Iteration and Transform Design

A readable stream is async iterable, so for await...of provides sequential chunk handling with natural error propagation. It is useful for parsers that need state across chunks. If the loop exits early, decide whether the source should be destroyed or reused.

A custom Transform must call its callback exactly once for each chunk, push only valid output, and report errors through the callback. Never assume a chunk is a complete line, JSON document, or protocol frame; retain incomplete trailing data for the next chunk.

Compress a File with pipeline

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('report.csv'),
  createGzip(),
  createWriteStream('report.csv.gz')
);
console.log('archive complete');

Output

archive complete

pipeline applies backpressure and rejects if reading, compression, or writing fails.

Web Streams and Interoperability

Node supports WHATWG Web Streams as well as classic Node streams. Use Readable.fromWeb and Readable.toWeb, with corresponding writable adapters, when integrating fetch or web-compatible libraries. Do not mix APIs without an explicit adapter and ownership decision.

Conversion does not remove backpressure or cancellation responsibilities. Propagate AbortSignal where supported and verify whether cancelling the consumer closes the original socket, file, or response body.

Failure, Abort, and Verification

Handle source, transform, destination, and premature-close failures as one operation. Abort on client disconnect or shutdown, remove temporary files after failed uploads, and never publish a partial artifact under its final name. Use a temporary path plus atomic rename where the file-system contract allows it.

Test empty input, one-byte chunks, split multibyte characters, slow consumers, transform errors, destination full, source truncation, cancellation, and maximum allowed size. Observe bytes processed, duration, abort reason, and stage without logging private content.

Before you move on

Stream Pipeline Check

4 checks

I can choose the four stream types.
I stop writing when write() returns false.
I use pipeline for owned multi-stage transfers.
I test partial chunks, cancellation, and cleanup.

Node JS Questions Learners Ask

pipeline provides one completion contract, forwards errors across stages, and cleans up connected streams when a stage fails.

No. It is a buffering threshold for one stream side; total memory includes all stages, requests, chunks, and native allocations.

Previous Next