Streams & Buffers

🌿 Node.js Lesson 5 Intermediate

Streams let you process data piece by piece (in chunks) rather than loading the entire dataset into memory at once. Buffers represent raw binary data. Together they enable efficient handling of large files, network data, and video streaming.

1 Four Types of Streams

Readable: Source of data to read from (e.g. file read, HTTP request body).
Writable: Destination to write data to (e.g. file write, HTTP response).
Duplex: Both readable and writable (e.g. TCP socket).
Transform: Duplex stream that modifies data as it passes through (e.g. gzip compression, encryption).

2 Piping Streams

The pipe() method connects a readable stream to a writable stream, handling backpressure automatically:

JavaScript — Streaming a large file

const fs = require('fs');
const zlib = require('zlib');

// Read a large file, compress it, write to output — all streamed
const readStream  = fs.createReadStream('large-file.txt');
const writeStream = fs.createWriteStream('large-file.txt.gz');
const gzip        = zlib.createGzip();

// Chain: read -> gzip -> write
readStream
  .pipe(gzip)
  .pipe(writeStream)
  .on('finish', () => console.log('File compressed successfully!'))
  .on('error', (err) => console.error('Stream error:', err));

3 Working with Buffers

JavaScript — Buffer operations

// Create a buffer from a string
const buf = Buffer.from('Hello, Node.js!', 'utf8');
console.log(buf);                        // <Buffer 48 65 6c 6c 6f...>
console.log(buf.toString('utf8'));       // Hello, Node.js!
console.log(buf.toString('hex'));        // 48656c6c6f...
console.log(buf.length);                // 15 bytes

// Allocate a fixed-size buffer (zeroed)
const fixed = Buffer.alloc(8);
fixed.writeUInt32BE(12345, 0);           // write integer at offset 0
console.log(fixed.readUInt32BE(0));      // 12345

// Concatenate buffers
const combined = Buffer.concat([
  Buffer.from('Hello '),
  Buffer.from('World')
]);
console.log(combined.toString());        // Hello World

4 Code Challenge

Challenge: Write a script using streams that reads a CSV file line-by-line using the readline module, parses each row, and counts the total number of records without loading the whole file into memory.