Node.js Buffers and Character Encodings

Byte-Level Data

Buffer represents a fixed-length sequence of bytes. Use it where files, sockets, compression, cryptography, or binary protocols expose bytes, and decode text only with the correct character encoding.

Bytes, Text, and Buffer

JavaScript strings represent text; Buffer stores bytes. The same bytes can mean text, an image header, an integer, encrypted material, or part of a network frame depending on the protocol. Buffer extends Uint8Array and works with Node streams and binary APIs.

Encoding is the mapping between text and bytes. UTF-8 uses a variable number of bytes per character, so string length and byte length can differ. Never truncate encoded text by character count when a protocol specifies a byte limit.

Know whether a boundary expects text or bytes.
Declare the encoding when converting.
Validate protocol length in bytes.

Safe Allocation and Creation

Buffer.from creates bytes from a string, array, ArrayBuffer, or another buffer. Buffer.alloc returns zero-filled memory. Buffer.allocUnsafe may reuse uninitialized memory and must be completely overwritten before any read or exposure; default to alloc unless profiling proves initialization cost matters.

A Buffer has fixed length. To build variable output, collect bounded chunks and concatenate once, or use a stream. Repeated concatenation inside a loop copies old data and can turn linear work into quadratic work.

Encode and Inspect UTF-8 Bytes

const text = 'Node ✓';
const bytes = Buffer.from(text, 'utf8');

console.log(bytes.length);
console.log(bytes.toString('hex'));
console.log(bytes.toString('utf8'));

Output

8
4e6f646520e29c93
Node ✓

The check mark occupies three UTF-8 bytes, so byte length is greater than JavaScript character count.

Encoding and Decoding

Common encodings include utf8 for text, hex for byte display, and base64 or base64url for transporting binary data through text fields. Base64 is encoding, not encryption. Validate decoded size before allocating or processing attacker-controlled input.

Decoding incomplete multibyte text one chunk at a time can insert replacement characters. Set an encoding on a readable stream or use StringDecoder when character boundaries may span chunks. For JSON and HTTP text, agree on UTF-8 end to end.

Reading and Writing Numbers

Buffer methods read and write signed or unsigned integers and floating-point values in big-endian or little-endian order. The protocol determines field width, offset, endianness, valid range, and whether a checksum is required.

Check the complete frame length before each read. A bounds error is preferable to silently accepting a partial message, but a parser should translate it into a controlled protocol error rather than crash the service. BigInt methods are required for integer fields beyond Number safe precision.

Encode a Small Binary Header

const header = Buffer.alloc(6);
header.writeUInt16BE(7, 0);
header.writeUInt32BE(1024, 2);

console.log(header.toString('hex'));
console.log(header.readUInt16BE(0), header.readUInt32BE(2));

Output

000700000400
7 1024

Both writer and reader use the same documented byte order and field offsets.

Views, Copies, and Mutation

Buffer.subarray creates a view over the same memory; mutating either view changes the shared bytes. Buffer.from(existingBuffer) creates a copy. Choose deliberately at ownership boundaries so one component cannot unexpectedly rewrite another component data.

ArrayBuffer conversion may also share memory depending on the constructor arguments. Treat buffers containing secrets as sensitive: minimize copies, avoid logs, overwrite only when the threat model benefits, and remember that garbage collection timing is not a secure erase guarantee.

Comparison and Constant Time

Buffer.equals and Buffer.compare perform ordinary byte comparison. Use them for identifiers, sorting, and deterministic tests. For authentication tags or secret-derived values, use crypto.timingSafeEqual with equal-length buffers after validating lengths.

Do not invent password storage or encryption with Buffer operations. Use established crypto APIs and algorithms, protect keys outside source, and authenticate ciphertext where the protocol requires it.

Buffer Boundaries and Limits

Network and file input can arrive in arbitrary chunk sizes. A data event is not a complete message unless the protocol guarantees it. Accumulate only until a validated frame length, or parse incrementally. Bound body, upload, decompression, and decoded base64 size to prevent memory exhaustion.

Test empty buffers, non-ASCII text, split characters, maximum field values, malformed hex or base64, truncated frames, and shared-view mutation. Record byte length and content type in diagnostics without dumping private payloads.

Before you move on

Buffer Safety Check

4 checks

I distinguish byte length from string length.
I use Buffer.alloc or fully overwrite allocUnsafe memory.
I know when subarray shares memory.
I enforce size and frame boundaries before decoding.

Node JS Questions Learners Ask

No. A string models text, while Buffer stores bytes. Converting between them requires a character encoding such as UTF-8.

Only after measurement shows a meaningful need and the program overwrites every byte before any read or exposure. Buffer.alloc is the safer default.

Previous Next