Compress and Decompress Files with Gzip and Brotli

Overview

Network bandwidth is often the slowest component in web application delivery. A 500KB JavaScript bundle may take 2 seconds to download on a 3G connection, but only 50 milliseconds for the server to generate. Compression bridges this gap by reducing payload sizes before transmission. Modern algorithms like Gzip and Brotli can shrink text-based assets — HTML, CSS, JavaScript, JSON, and SVG — by 60-85% without any loss of data.

The key insight is that compression should happen transparently at the right layer. Web servers (Nginx, Apache) can compress responses on the fly. Build pipelines (Webpack, Vite) can pre-compress static assets during deployment. APIs can stream compressed JSON directly to clients that advertise support via the Accept-Encoding header. The following demonstrates how to Gzip, Brotli, and streaming compression across Python, Node.js, and web server configurations.

When to use it

Use this recipe when:

• Serving large JavaScript bundles, CSS stylesheets, or HTML documents. See SPA Code Splitting for reducing bundle sizes.
• Reducing API response sizes for mobile clients on metered connections. See Call REST API for efficient API design.
• Compressing log files before archiving them to cold storage. See Stream Processing for log pipeline processing.
• Uploading large payloads to object storage or transferring files between services. See Image Optimization for media compression.
• Complying with performance budgets that mandate maximum transfer sizes. See Lazy Loading Images for transfer reduction.

Solution

Gzip Streaming Compression (Node.js / zlib)

const zlib = require('zlib');
const fs = require('fs');
const { pipeline } = require('stream');

function compressFile(inputPath, outputPath) {
  const gzip = zlib.createGzip({ level: 6 });
  const source = fs.createReadStream(inputPath);
  const destination = fs.createWriteStream(outputPath);

  pipeline(source, gzip, destination, (err) => {
    if (err) console.error('Compression failed:', err);
    else console.log('File compressed successfully');
  });
}

compressFile('data.json', 'data.json.gz');

Brotli Compression (Python)

import brotli
import gzip

def compress_with_brotli(data: bytes) -> bytes:
    return brotli.compress(data, quality=4)

def compress_with_gzip(data: bytes) -> bytes:
    return gzip.compress(data, compresslevel=6)

# Example: compress a JSON response
json_data = b'{"users": [...]}' * 1000
brotli_compressed = compress_with_brotli(json_data)
gzip_compressed = compress_with_gzip(json_data)

print(f"Original: {len(json_data)} bytes")
print(f"Brotli: {len(brotli_compressed)} bytes ({len(brotli_compressed)/len(json_data)*100:.1f}%)")
print(f"Gzip: {len(gzip_compressed)} bytes ({len(gzip_compressed)/len(json_data)*100:.1f}%)")

Express Middleware with Content Negotiation

const express = require('express');
const compression = require('compression');

const app = express();

app.use(compression({
  filter: (req, res) => {
    if (req.headers['x-no-compression']) return false;
    return compression.filter(req, res);
  },
  level: 6,
  threshold: 1024, // Only compress responses > 1KB
}));

app.get('/api/data', (req, res) => {
  res.json(largeDataset);
});

Nginx Static Asset Pre-compression

gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;

# Serve pre-compressed Brotli files
brotli on;
brotli_static on;
brotli_comp_level 4;
brotli_types text/plain text/css application/javascript application/json image/svg+xml;

Explanation

• Gzip (DEFLATE): the universal compression standard supported by every browser and HTTP client since 1998. It uses LZ77 and Huffman coding to eliminate redundancy. Compression level 6 provides the best balance between CPU cost and size reduction.
• Brotli: developed by Google, Brotli achieves 15-25% better compression than Gzip for text assets. It uses a predefined dictionary of common web terms (HTML tags, CSS properties, JavaScript keywords) to improve ratios. Supported in all modern browsers.
• Streaming compression: instead of loading an entire file into memory, streaming reads chunks from disk, compresses them, and writes to the output. This handles multi-gigabyte files without exhausting RAM.
• Content negotiation: browsers send Accept-Encoding: gzip, deflate, br to indicate supported algorithms. Servers respond with Content-Encoding: br and the compressed payload. If the client does not support compression, the server returns uncompressed data.

Variants

Algorithm	Compression ratio	Speed	Browser support	Best for
Gzip	Good	Fast	Universal	Live responses, legacy support
Brotli	Excellent	Medium	Modern browsers	Static assets, pre-compressed files
Zstandard	Very good	Very fast	Limited	Internal APIs, microservices
LZ4	Low	Extremely fast	Tools	Real-time logs, speed-critical paths

What Works

• Pre-compress static assets during build: instead of compressing on every request, run brotli -q 11 and gzip -k during your CI/CD pipeline. Store .br and .gz variants alongside originals. Nginx can serve these directly with brotli_static on.
• Do not compress already-compressed formats: images (JPEG, PNG, WebP), videos (MP4), and archives (ZIP) are already compressed. Running Gzip on them wastes CPU and may increase file size. Skip compression for these MIME types.
• Use threshold filters: compressing a 200-byte JSON response adds more overhead (headers, framing) than it saves. Set a minimum size threshold of 1KB and only compress text/*, application/json, and image/svg+xml.
• Enable Vary: Accept-Encoding: caches and CDNs must store separate variants for compressed and uncompressed responses. The Vary header tells intermediaries to key the cache on the Accept-Encoding header, preventing serving gzip to clients that cannot decompress.
• Monitor CPU overhead: compression is CPU-intensive. On high-traffic APIs, pre-compression or dedicated compression appliances (CDNs) offload work from application servers. Profile your application to ensure compression does not starve request handling.

Common mistakes

• Double compression: applying Gzip to a response that is already Brotli-compressed, or vice versa, corrupts the data. Ensure your middleware stack does not apply multiple compression layers.
• Compressing on every request: live compression for static assets is wasteful. Pre-compress once at build time and serve the pre-compressed file directly. Live compression should only apply to truly live responses.
• Forgetting to decompress on the client: API clients must explicitly decompress responses or use libraries that handle Content-Encoding transparently. Raw Gzip bytes passed to a JSON parser will throw syntax errors.
• Ignoring memory limits: decompressing untrusted user input can trigger zip bomb attacks (a small compressed file that expands to terabytes). Limit decompression buffer sizes and use streaming APIs that process chunks incrementally.

Advanced Solutions

Python: Streaming Gzip with configurable levels and integrity checks

import gzip
import hashlib
from pathlib import Path

def gzip_streaming(src: str, dest: str, level: int = 6) -> str:
    """GZIP a file with streaming. Returns SHA256 of original for integrity."""
    hasher = hashlib.sha256()
    with open(src, 'rb') as f_in, gzip.open(dest, 'wb', compresslevel=level) as f_out:
        while True:
            chunk = f_in.read(65536)
            if not chunk:
                break
            hasher.update(chunk)
            f_out.write(chunk)
    return hasher.hexdigest()

def gunzip_streaming(src: str, dest: str, expected_sha256: str = None) -> bool:
    """Decompress GZIP with optional integrity verification. Returns True if valid."""
    hasher = hashlib.sha256()
    with gzip.open(src, 'rb') as f_in, open(dest, 'wb') as f_out:
        while True:
            chunk = f_in.read(65536)
            if not chunk:
                break
            hasher.update(chunk)
            f_out.write(chunk)
    if expected_sha256:
        return hasher.hexdigest() == expected_sha256
    return True

def gzip_batch(files: list[str], level: int = 6) -> dict[str, str]:
    """GZIP multiple files. Returns mapping of original path to SHA256."""
    results = {}
    for file_path in files:
        dest = f"{file_path}.gz"
        sha = gzip_streaming(file_path, dest, level)
        results[file_path] = sha
    return results

# Usage
# sha = gzip_streaming('large.log', 'large.log.gz', level=6)
# print(f"Original SHA256: {sha}")
# valid = gunzip_streaming('large.log.gz', 'large_restored.log', expected_sha256=sha)
# print(f"Integrity check: {'PASS' if valid else 'FAIL'}")

Node.js: Brotli streaming with content negotiation middleware

const zlib = require('zlib');
const fs = require('fs');
const { pipeline } = require('stream');
const { promisify } = require('util');
const pipe = promisify(pipeline);

async function compressBrotli(srcPath, destPath, quality = 11) {
    const src = fs.createReadStream(srcPath);
    const brotli = zlib.createBrotliCompress({
        params: {
            [zlib.constants.BROTLI_PARAM_QUALITY]: quality,
            [zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
        },
    });
    const dest = fs.createWriteStream(destPath);
    await pipe(src, brotli, dest);
}

async function compressGzip(srcPath, destPath, level = 6) {
    const src = fs.createReadStream(srcPath);
    const gzip = zlib.createGzip({ level });
    const dest = fs.createWriteStream(destPath);
    await pipe(src, gzip, dest);
}

// Express middleware: negotiate best encoding
function smartCompression() {
    return (req, res, next) => {
        const acceptEncoding = req.headers['accept-encoding'] || '';
        const originalSend = res.send.bind(res);

        res.send = function (body) {
            if (typeof body === 'string' && body.length > 1024) {
                if (acceptEncoding.includes('br')) {
                    res.setHeader('Content-Encoding', 'br');
                    res.setHeader('Vary', 'Accept-Encoding');
                    body = zlib.brotliCompressSync(body, {
                        params: { [zlib.constants.BROTLI_PARAM_QUALITY]: 4 },
                    });
                } else if (acceptEncoding.includes('gzip')) {
                    res.setHeader('Content-Encoding', 'gzip');
                    res.setHeader('Vary', 'Accept-Encoding');
                    body = zlib.gzipSync(body, { level: 6 });
                }
            }
            originalSend(body);
        };
        next();
    };
}

// Usage
// await compressBrotli('app.js', 'app.js.br', 11);
// await compressGzip('app.js', 'app.js.gz', 6);
// app.use(smartCompression());

Java: GZIP and Brotli compression with configurable buffers

import java.io.*;
import java.nio.file.*;
import java.util.zip.*;

public class CompressionUtils {

    // GZIP a file with streaming
    public static long gzipFile(Path src, Path dest, int bufferSize) throws IOException {
        long bytesWritten = 0;
        try (InputStream fis = Files.newInputStream(src);
             OutputStream fos = Files.newOutputStream(dest);
             GZIPOutputStream gzos = new GZIPOutputStream(fos, bufferSize) {{
                 def.setLevel(6); // Set compression level
             }}) {
            byte[] buffer = new byte[bufferSize];
            int read;
            while ((read = fis.read(buffer)) != -1) {
                gzos.write(buffer, 0, read);
                bytesWritten += read;
            }
        }
        return bytesWritten;
    }

    // Decompress GZIP with streaming
    public static long gunzipFile(Path src, Path dest, int bufferSize) throws IOException {
        long bytesWritten = 0;
        try (InputStream fis = Files.newInputStream(src);
             GZIPInputStream gzis = new GZIPInputStream(fis, bufferSize);
             OutputStream fos = Files.newOutputStream(dest)) {
            byte[] buffer = new byte[bufferSize];
            int read;
            while ((read = gzis.read(buffer)) != -1) {
                fos.write(buffer, 0, read);
                bytesWritten += read;
            }
        }
        return bytesWritten;
    }

    // Verify GZIP integrity without extracting
    public static boolean verifyGzip(Path gzPath) {
        try (InputStream fis = Files.newInputStream(gzPath);
             GZIPInputStream gzis = new GZIPInputStream(fis)) {
            byte[] buffer = new byte[8192];
            while (gzis.read(buffer) != -1) {
                // Read through entire file to verify integrity
            }
            return true;
        } catch (IOException e) {
            return false;
        }
    }

    // Batch compress files in a directory
    public static int gzipDirectory(Path srcDir, Path destDir, int bufferSize) throws IOException {
        Files.createDirectories(destDir);
        int count = 0;
        try (var files = Files.walk(srcDir)) {
            for (Path file : files.filter(Files::isRegularFile).toList()) {
                Path relative = srcDir.relativize(file);
                Path dest = destDir.resolve(relative.toString() + ".gz");
                Files.createDirectories(dest.getParent());
                gzipFile(file, dest, bufferSize);
                count++;
            }
        }
        return count;
    }
}

// Usage
// long bytes = CompressionUtils.gzipFile(Path.of("data.json"), Path.of("data.json.gz"), 8192);
// boolean valid = CompressionUtils.verifyGzip(Path.of("data.json.gz"));
// int count = CompressionUtils.gzipDirectory(Path.of("logs/"), Path.of("logs-gz/"), 16384);

Bash: Pre-compression pipeline for static assets

#!/usr/bin/env bash
set -euo pipefail

# Pre-compress static assets with both Gzip and Brotli
precompress_assets() {
    local dir="$1"
    local count=0

    for file in "$dir""/"*.{js,css,html,json,svg,xml,txt}; do
        [[ -f "$file" ]] || continue

        # Skip if already compressed
        [[ "$file" == *.gz || "$file" == *.br ]] && continue

        # Gzip compression (level 9 for static assets)
        if [[ ! -f "${file}.gz" || "$file" -nt "${file}.gz" ]]; then
            gzip -9 -k -f "$file"
            ((count++))
        fi

        # Brotli compression (quality 11 for static assets)
        if command -v brotli &>/dev/null; then
            if [[ ! -f "${file}.br" || "$file" -nt "${file}.br" ]]; then
                brotli -q 11 -k -f "$file"
                ((count++))
            fi
        fi
    done

    echo "Pre-compressed $count files in $dir"
}

# Compare compression ratios
compare_ratios() {
    local file="$1"
    local original_size
    local gzip_size
    local brotli_size

    original_size=$(stat -c%s "$file" 2>/dev/null || stat -f%z "$file")
    gzip -c -9 "$file" > /tmp/compare.gz
    brotli -c -q 11 "$file" > /tmp/compare.br 2>/dev/null || true

    gzip_size=$(stat -c%s /tmp/compare.gz 2>/dev/null || stat -f%z /tmp/compare.gz)
    brotli_size=$(stat -c%s /tmp/compare.br 2>/dev/null || stat -f%z /tmp/compare.br)

    echo "File: $file"
    echo "  Original: $original_size bytes"
    echo "  Gzip:     $gzip_size bytes ($(awk "BEGIN{printf \"%.1f\", ($gzip_size/$original_size)*100}")%)"
    echo "  Brotli:   $brotli_size bytes ($(awk "BEGIN{printf \"%.1f\", ($brotli_size/$original_size)*100}")%)"

    rm -f /tmp/compare.gz /tmp/compare.br
}

# Verify gzip integrity
verify_gzip() {
    local file="$1"
    if gzip -t "$file" 2>/dev/null; then
        echo "OK: $file is valid"
    else
        echo "FAIL: $file is corrupted"
        return 1
    fi
}

# Usage
# precompress_assets /var/www/static
# compare_ratios /var/www/static/app.js
# verify_gzip /var/www/static/app.js.gz