Compress and Decompress Files

Overview

Archiving and compressing files reduces storage and transfer costs. Programmatically handling ZIP, GZIP, and TAR is essential for backup scripts, data exports, and artifact packaging. Below is a practical approach to all three formats in Python, JavaScript, and Java.

When to Use

Use this resource when:

• Packaging log bundles or report exports for download
• Compressing HTTP responses to reduce bandwidth
• Extracting uploaded archives in web applications

Solution

Python

import zipfile
import gzip
import tarfile

# ZIP archive
with zipfile.ZipFile('archive.zip', 'w', zipfile.ZIP_DEFLATED) as z:
    z.write('file.txt')

# GZIP single file
with open('file.txt', 'rb') as f_in:
    with gzip.open('file.txt.gz', 'wb') as f_out:
        f_out.writelines(f_in)

# TAR archive
with tarfile.open('archive.tar.gz', 'w:gz') as tar:
    tar.add('data/')

JavaScript

const fs = require('fs');
const zlib = require('zlib');
const archiver = require('archiver');

// GZIP compress
const input = fs.createReadStream('file.txt');
const output = fs.createWriteStream('file.txt.gz');
input.pipe(zlib.createGzip()).pipe(output);

// ZIP archive
const archive = archiver('zip', { zlib: { level: 9 } });
archive.pipe(fs.createWriteStream('archive.zip'));
archive.file('file.txt', { name: 'file.txt' });
archive.finalize();

Java

import java.io.*;
import java.util.zip.*;

public class Compressor {
    // GZIP compress
    public void gzip(String src, String dest) throws IOException {
        try (FileInputStream fis = new FileInputStream(src);
             FileOutputStream fos = new FileOutputStream(dest);
             GZIPOutputStream gzos = new GZIPOutputStream(fos)) {
            fis.transferTo(gzos);
        }
    }

    // ZIP archive
    public void zip(String src, String dest) throws IOException {
        try (FileOutputStream fos = new FileOutputStream(dest);
             ZipOutputStream zos = new ZipOutputStream(fos);
             FileInputStream fis = new FileInputStream(src)) {
            zos.putNextEntry(new ZipEntry(new File(src).getName()));
            fis.transferTo(zos);
            zos.closeEntry();
        }
    }
}

Explanation

ZIP stores multiple files with optional per-file compression, preserving directory structure. GZIP compresses a single file or stream, commonly used for HTTP content encoding and log rotation. TAR archives multiple files without compression; paired with GZIP it becomes a .tar.gz (or .tgz). All three use DEFLATE internally, offering excellent compression for text data.

Variants

Technology	Approach	Notes
Python	shutil.make_archive()	One-liner for ZIP/TAR creation
JavaScript	adm-zip	In-memory ZIP manipulation, no streams
Java	Apache Commons Compress	Supports BZIP2, LZMA, and 7Z formats

What Works

1. Stream large files rather than buffering entire archives in memory
2. Use compression level 6 as a balanced default; level 9 is slower with diminishing returns
3. Validate extracted paths to prevent zip-slip directory traversal attacks
4. Prefer GZIP for single-file compression; ZIP/TAR for multi-file bundles
5. Close streams in try-with-resources / with blocks to avoid file descriptor leaks

Common Mistakes

1. Loading entire archives into memory instead of streaming
2. Not validating extracted entry paths, allowing directory traversal exploits
3. Forgetting to finalize() or closeEntry(), producing corrupt archives
4. Applying compression to already-compressed formats (e.g., JPEG, MP4)
5. Ignoring encoding when compressing text files across platforms

Advanced Solutions

Python: Streaming compression with progress and zip-slip protection

import zipfile
import gzip
import tarfile
import os
from pathlib import Path

def compress_directory_streaming(src_dir: str, dest_zip: str,
                                 compression: int = zipfile.ZIP_DEFLATED,
                                 level: int = 6) -> int:
    """Compress a directory to ZIP with streaming. Returns file count."""
    src_path = Path(src_dir)
    file_count = 0
    with zipfile.ZipFile(dest_zip, 'w', compression, compresslevel=level) as zf:
        for file_path in sorted(src_path.rglob('*')):
            if file_path.is_file():
                arcname = file_path.relative_to(src_path)
                zf.write(file_path, arcname)
                file_count += 1
    return file_count

def decompress_zip_safe(zip_path: str, dest_dir: str) -> int:
    """Extract ZIP with zip-slip protection. Returns file count."""
    dest_path = Path(dest_dir).resolve()
    dest_path.mkdir(parents=True, exist_ok=True)
    file_count = 0

    with zipfile.ZipFile(zip_path, 'r') as zf:
        for member in zf.namelist():
            member_path = (dest_path / member).resolve()
            # Prevent zip-slip: ensure resolved path is under dest
            if not str(member_path).startswith(str(dest_path)):
                raise ValueError(f"Unsafe path detected: {member}")
            zf.extract(member, dest_path)
            file_count += 1
    return file_count

def gzip_file_streaming(src: str, dest: str, level: int = 6) -> None:
    """GZIP a single file with streaming and configurable level."""
    with open(src, 'rb') as f_in, gzip.open(dest, 'wb', compresslevel=level) as f_out:
        while True:
            chunk = f_in.read(65536)
            if not chunk:
                break
            f_out.write(chunk)

def tar_directory_streaming(src_dir: str, dest: str,
                            mode: str = 'w:gz', level: int = 6) -> None:
    """Create a TAR.GZ archive with streaming."""
    with tarfile.open(dest, mode, compresslevel=level) as tar:
        tar.add(src_dir, arcname=Path(src_dir).name)

def list_archive_contents(archive_path: str) -> list[str]:
    """List contents of ZIP or TAR archive."""
    if archive_path.endswith('.zip'):
        with zipfile.ZipFile(archive_path, 'r') as zf:
            return zf.namelist()
    elif archive_path.endswith(('.tar.gz', '.tgz', '.tar')):
        with tarfile.open(archive_path, 'r:*') as tar:
            return tar.getnames()
    raise ValueError(f"Unsupported archive format: {archive_path}")

# Usage
# count = compress_directory_streaming('logs/', 'logs.zip', level=6)
# print(f"Compressed {count} files")
# extracted = decompress_zip_safe('upload.zip', 'extracted/')
# print(f"Extracted {extracted} files safely")

Node.js: Streaming compression pipeline with zlib

const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream');
const { promisify } = require('util');
const pipe = promisify(pipeline);

async function gzipFile(srcPath, destPath, level = 6) {
    const src = fs.createReadStream(srcPath);
    const gzip = zlib.createGzip({ level });
    const dest = fs.createWriteStream(destPath);
    await pipe(src, gzip, dest);
}

async function gunzipFile(srcPath, destPath) {
    const src = fs.createReadStream(srcPath);
    const gunzip = zlib.createGunzip();
    const dest = fs.createWriteStream(destPath);
    await pipe(src, gunzip, dest);
}

async function gzipDirectory(srcDir, destZip) {
    const archiver = require('archiver');
    const output = fs.createWriteStream(destZip);
    const archive = archiver('zip', { zlib: { level: 6 } });

    const done = new Promise((resolve, reject) => {
        output.on('close', () => resolve(archive.pointer()));
        output.on('error', reject);
        archive.on('error', reject);
    });

    archive.pipe(output);
    archive.directory(srcDir, false);
    archive.finalize();

    const bytes = await done;
    return bytes;
}

async function extractZipSafe(zipPath, destDir) {
    const extract = require('extract-zip');
    const path = require('path');

    await extract(zipPath, {
        dir: path.resolve(destDir),
        onEntry: (entry, zipfile) => {
            // Prevent zip-slip: reject paths escaping destDir
            const dest = path.resolve(destDir, entry.fileName);
            if (!dest.startsWith(path.resolve(destDir))) {
                throw new Error(`Unsafe path in archive: ${entry.fileName}`);
            }
        },
    });
}

// Usage
// gzipFile('large.log', 'large.log.gz', 9);
// const bytes = await gzipDirectory('logs/', 'logs.zip');
// console.log(`Archive size: ${bytes} bytes`);
// await extractZipSafe('upload.zip', 'extracted/');

Java: Batch compression with try-with-resources

import java.io.*;
import java.nio.file.*;
import java.util.zip.*;
import java.util.List;
import java.util.ArrayList;
import java.util.stream.Stream;

public class BatchCompressor {

    // GZIP a single file
    public static void gzipFile(Path src, Path dest, int bufferSize) throws IOException {
        try (InputStream fis = Files.newInputStream(src);
             OutputStream fos = Files.newOutputStream(dest);
             GZIPOutputStream gzos = new GZIPOutputStream(fos, bufferSize)) {
            fis.transferTo(gzos);
        }
    }

    // ZIP multiple files with streaming
    public static int zipFiles(List<Path> sources, Path destZip) throws IOException {
        int count = 0;
        try (OutputStream fos = Files.newOutputStream(destZip);
             ZipOutputStream zos = new ZipOutputStream(fos)) {
            for (Path src : sources) {
                ZipEntry entry = new ZipEntry(src.getFileName().toString());
                zos.putNextEntry(entry);
                try (InputStream fis = Files.newInputStream(src)) {
                    fis.transferTo(zos);
                }
                zos.closeEntry();
                count++;
            }
        }
        return count;
    }

    // Extract ZIP with zip-slip protection
    public static int extractZipSafe(Path zipPath, Path destDir) throws IOException {
        Files.createDirectories(destDir);
        int count = 0;
        try (InputStream fis = Files.newInputStream(zipPath);
             ZipInputStream zis = new ZipInputStream(fis)) {
            ZipEntry entry;
            while ((entry = zis.getNextEntry()) != null) {
                Path destFile = destDir.resolve(entry.getName()).normalize();
                // Prevent zip-slip
                if (!destFile.startsWith(destDir)) {
                    throw new IOException("Unsafe zip entry: " + entry.getName());
                }
                if (entry.isDirectory()) {
                    Files.createDirectories(destFile);
                } else {
                    Files.createDirectories(destFile.getParent());
                    Files.copy(zis, destFile, StandardCopyOption.REPLACE_EXISTING);
                }
                count++;
            }
        }
        return count;
    }

    // Compress all files in a directory
    public static int compressDirectory(Path srcDir, Path destZip) throws IOException {
        List<Path> files = new ArrayList<>();
        try (Stream<Path> stream = Files.walk(srcDir)) {
            stream.filter(Files::isRegularFile).forEach(files::add);
        }
        return zipFiles(files, destZip);
    }
}

// Usage
// BatchCompressor.gzipFile(Path.of("large.log"), Path.of("large.log.gz"), 8192);
// int count = BatchCompressor.compressDirectory(Path.of("logs/"), Path.of("logs.zip"));
// System.out.println("Compressed " + count + " files");
// int extracted = BatchCompressor.extractZipSafe(Path.of("upload.zip"), Path.of("extracted/"));

Bash: tar/gzip with progress and parallel compression

#!/usr/bin/env bash
set -euo pipefail

# Compress directory to tar.gz with progress
compress_tarball() {
    local src="$1"
    local dest="$2"
    local level="${3:-6}"
    tar -c -C "$(dirname "$src")" "$(basename "$src")" \
        | gzip -"$level" -c > "$dest"
    echo "Created $dest ($(du -h "$dest" | cut -f1))"
}

# Extract tarball safely (prevent path traversal)
extract_tarball_safe() {
    local archive="$1"
    local dest="$2"
    mkdir -p "$dest"
    # List contents first, verify no absolute paths or ../ escapes
    if tar -tf "$archive" | grep -E '^/|\.\.'; then
        echo "Error: unsafe paths detected in $archive" >&2
        return 1
    fi
    tar -xzf "$archive" -C "$dest"
    echo "Extracted to $dest"
}

# Parallel gzip compression using pigz (3-5x faster than gzip)
compress_parallel() {
    local src="$1"
    local dest="$2"
    if command -v pigz &>/dev/null; then
        tar -c -C "$(dirname "$src")" "$(basename "$src")" | pigz -p 4 > "$dest"
    else
        tar -czf "$dest" -C "$(dirname "$src")" "$(basename "$src")"
    fi
    echo "Created $dest"
}

# Batch compress individual files
batch_gzip() {
    local dir="$1"
    local count=0
    for file in "$dir"/*; do
        [[ -f "$file" ]] || continue
        gzip -c "$file" > "${file}.gz"
        ((count++))
    done
    echo "Compressed $count files in $dir"
}

# Usage
# compress_tarball logs/ logs.tar.gz 6
# extract_tarball_safe archive.tar.gz extracted/
# compress_parallel data/ data.tar.gz
# batch_gzip /var/log/app/