Read and Write Files in Python, Node.js & Bash

Overview

Few I/O jobs come up as often as reading and writing files: config loading, log processing, report exports, state persistence. Doing it safely means handling encoding correctly and always closing the file handle.

Files are the common interface between a program and persistent storage. Saving user prefs, scanning server logs, exporting a CSV — the rules don’t change: open the file, do the work, release the handle even when something fails. Today’s runtimes take care of buffering, encoding, and cleanup for you, but knowing what happens underneath pays off when you’re chasing a slow read or a corrupted file.

Below you’ll find the idiomatic read/write for text files in Python, Node.js, and Bash, plus the streaming pattern for files too big for memory.

When to Use

Reach for this recipe when you’re:

• Loading configuration or data files at startup. See Parse JSON for structured config files.
• Generating reports, exports, or logs for audit and analytics
• Processing text line by line (CSV, logs, fixtures)
• Persisting small amounts of state without a database
• Reading and writing JSON or YAML configuration files
• Streaming large log files without loading them entirely into memory
• Creating temporary files for intermediate processing in data pipelines. See Call REST API for downloading remote data.

Skip this recipe when the file is too large to buffer or you need a continuous transform pipeline — Stream Processing covers that case.

Solution

Python

In Python, with wraps the file in a context manager — the handle closes itself even if the block throws. Always pass encoding="utf-8" to avoid platform-dependent defaults.

# Write
with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("Hello, file!\n")

# Read
with open("notes.txt", "r", encoding="utf-8") as f:
    content = f.read()
print(content)

JavaScript

Under node:fs/promises, Node.js ships a promise-based API that never blocks the event loop — non-negotiable for a server juggling concurrent requests.

import { readFile, writeFile } from "node:fs/promises";

await writeFile("notes.txt", "Hello, file!\n", "utf-8");

const content = await readFile("notes.txt", "utf-8");
console.log(content);

Bash

In Bash, file work runs through shell redirection. The > operator overwrites the target file and >> appends to it — the quickest option for small writes inside scripts.

# Write (overwrite) and append
echo "Hello, file!" > notes.txt
echo "Another line" >> notes.txt

# Read
cat notes.txt

Explanation

• Python leans on the with statement (a context manager) so the file closes itself even when something fails. open() takes a mode string — "r" reads, "w" writes and truncates, "a" appends, "x" creates only if the file isn’t there. Always pass encoding="utf-8".
• JavaScript — fs/promises does the heavy lifting. The synchronous readFileSync/writeFileSync freeze the event loop, so keep them out of request handlers. For large files, createReadStream() processes the data in chunks.
• Bash works through redirection: > overwrites, >> appends, and cat prints the contents. For structured parsing, pipe cat into jq for JSON or awk for CSV.

Atomic writes follow a fixed sequence that keeps readers safe from half-written files:

To turn file contents into structured data, see Parse JSON.

Errors you’ll actually hit

Error	Where it comes from	Fix
FileNotFoundError / ENOENT	missing path or a typo	check pathlib.Path.exists() / fs.access() first
PermissionError / EACCES	read-only file or missing execute bit on a parent dir	fix permissions (chmod); retrying won’t help
IsADirectoryError / EISDIR	you opened a directory as a file	verify path.is_file() before opening
UnicodeDecodeError	the file isn’t UTF-8	detect the real encoding (file -i, chardet) or pass it explicitly
ENOSPC	disk full mid-write	atomic write keeps the original intact; surface the error and alert

Variants

Language	Read	Write	Append
Python	open(p).read()	open(p, "w")	open(p, "a")
JavaScript	readFile(p)	writeFile(p, data)	appendFile(p, data)
Bash	cat p	> p	>> p

What Works

• Always set encoding: an explicit utf-8 avoids platform-dependent defaults that corrupt non-ASCII characters on Windows or macOS.
• Use context managers / async APIs: with in Python, fs/promises in Node, to avoid descriptor leaks and event-loop blocking. Cleanup still runs if an exception hits mid-block.
• Check the path exists: a missing file should degrade gracefully, not crash the process. In Python, pathlib.Path.exists() tells you upfront; in Node, fs.access() or fs.stat() do the job.
• Stream large files: don’t pull gigabytes into memory when a line at a time will do. Python gives you for line in f, Node has readline or createReadStream, and Bash has while read line.
• Write atomically: put the bytes in a temp file, then rename — a crash mid-write leaves the original untouched.
• Use absolute paths in scripts: relative paths break the moment the working directory changes. Resolve them to absolute form — pathlib in Python, path.resolve() in Node — before the first open().
• Set restrictive permissions on sensitive files: secrets in a config file? Lock it down to the owner only (chmod 600).

Common Mistakes

• Forgetting to close the handle: leaks file descriptors and eventually exhausts the process limit; always use with or try/finally.
• Blocking the event loop in Node: don’t put readFileSync inside request handlers — one synchronous read freezes the whole server for every concurrent user.
• Wrong encoding: read UTF-8 as ASCII and non-English text turns to mojibake in your logs and UI.
• Overwriting with >: using > instead of >> in Bash silently erases the file — no undo, no confirmation.
• Ignoring errors: a missing file or a permission failure deserves a real handler — not an empty catch. Log it and fail in a controlled way.
• Reading entire files into memory: loading a 10 GB log file into a string will kill your process. Check the file size first, or stream anything over a few megabytes.
• Writing to the same file you’re reading: overwrite your input in place and you can truncate it mid-read — data gone.

Advanced Solutions

Python: Atomic writes with pathlib and error handling

import os
import tempfile
from pathlib import Path
from typing import Any

def safe_write(path: str | Path, data: str, encoding: str = 'utf-8') -> None:
    """Write text atomically: temp file + rename. Safe against crashes."""
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)

    fd, tmp_path = tempfile.mkstemp(
        dir=path.parent, suffix='.tmp', prefix=path.name
    )
    try:
        with os.fdopen(fd, 'w', encoding=encoding) as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp_path, path)
    except Exception:
        Path(tmp_path).unlink(missing_ok=True)
        raise

def safe_read(path: str | Path, encoding: str = 'utf-8',
              default: str | None = None) -> str | None:
    """Read text with graceful fallback for missing files."""
    path = Path(path)
    if not path.exists():
        return default
    try:
        return path.read_text(encoding=encoding)
    except PermissionError:
        raise PermissionError(f"Cannot read {path}: permission denied")
    except UnicodeDecodeError as e:
        raise UnicodeDecodeError(
            e.encoding, e.object, e.start, e.end,
            f"File {path} is not valid {encoding}"
        )

def read_lines_lazy(path: str | Path, encoding: str = 'utf-8') -> list[str]:
    """Read file lines, dropping blanks and trailing newlines."""
    path = Path(path)
    with path.open('r', encoding=encoding) as f:
        return [line.rstrip('\n\r') for line in f if line.strip()]

def write_json_atomic(path: str | Path, data: Any, indent: int = 2) -> None:
    """Serialize JSON and write atomically."""
    import json
    text = json.dumps(data, indent=indent, ensure_ascii=False, default=str)
    safe_write(path, text)

# Usage
# safe_write('/etc/app/config.yaml', 'key: value\n')
# content = safe_read('/etc/app/config.yaml', default='key: default\n')
# write_json_atomic('/data/state.json', {'users': 42, 'active': 10})

Node.js: Streaming read/write with error recovery

const fs = require('fs');
const { pipeline } = require('stream');
const { promisify } = require('util');
const { createReadStream, createWriteStream } = fs;
const pipe = promisify(pipeline);

async function streamFile(srcPath, destPath, transformFn) {
    const tmpPath = destPath + '.tmp';
    const readStream = createReadStream(srcPath, { encoding: 'utf-8' });
    const writeStream = createWriteStream(tmpPath, { encoding: 'utf-8' });

    let lineBuffer = '';
    const lineTransform = new (require('stream').Transform)({
        transform(chunk, encoding, callback) {
            lineBuffer += chunk;
            const lines = lineBuffer.split('\n');
            lineBuffer = lines.pop();
            for (const line of lines) {
                const result = transformFn(line);
                if (result !== null) this.push(result + '\n');
            }
            callback();
        },
        flush(callback) {
            if (lineBuffer) {
                const result = transformFn(lineBuffer);
                if (result !== null) this.push(result + '\n');
            }
            callback();
        },
    });

    try {
        await pipe(readStream, lineTransform, writeStream);
        await fs.promises.rename(tmpPath, destPath);
    } catch (err) {
        try { await fs.promises.unlink(tmpPath); } catch {}
        throw err;
    }
}

async function readLines(path) {
    const content = await fs.promises.readFile(path, 'utf-8');
    return content.split('\n').filter(l => l.trim());
}

async function appendLine(path, line) {
    await fs.promises.appendFile(path, line + '\n', 'utf-8');
}

// Usage
// streamFile('input.log', 'output.log', line => line.toUpperCase());
// const lines = await readLines('config.txt');
// await appendLine('app.log', `[${new Date().toISOString()}] Started`);

Java: NIO file operations with atomic writes

import java.io.*;
import java.nio.file.*;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.ArrayList;

public class FileOps {

    // Atomic write: temp file + Files.move with ATOMIC_MOVE
    public static void atomicWrite(Path path, String content) throws IOException {
        Path parent = path.getParent();
        if (parent != null) Files.createDirectories(parent);
        Path tmp = Files.createTempFile(parent, path.getFileName().toString(), ".tmp");
        try {
            Files.writeString(tmp, content, StandardCharsets.UTF_8);
            Files.move(tmp, path, StandardCopyOption.ATOMIC_MOVE,
                       StandardCopyOption.REPLACE_EXISTING);
        } catch (Exception e) {
            Files.deleteIfExists(tmp);
            throw e;
        }
    }

    // Safe read with fallback
    public static String safeRead(Path path, String defaultValue) {
        if (!Files.exists(path)) return defaultValue;
        try {
            return Files.readString(path, StandardCharsets.UTF_8);
        } catch (IOException e) {
            throw new UncheckedIOException("Failed to read " + path, e);
        }
    }

    // Read all lines lazily
    public static List<String> readLines(Path path) throws IOException {
        return Files.readAllLines(path, StandardCharsets.UTF_8);
    }

    // Append a line
    public static void appendLine(Path path, String line) throws IOException {
        String entry = line + System.lineSeparator();
        Files.writeString(path, entry,
            StandardCharsets.UTF_8,
            StandardOpenOption.CREATE,
            StandardOpenOption.APPEND);
    }

    // Stream lines with try-with-resources
    public static void processLines(Path path, LineHandler handler) throws IOException {
        try (BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8)) {
            String line;
            while ((line = reader.readLine()) != null) {
                handler.handle(line);
            }
        }
    }

    @FunctionalInterface
    public interface LineHandler {
        void handle(String line) throws IOException;
    }
}

// Usage
// FileOps.atomicWrite(Path.of("/etc/app/config.yaml"), "key: value\n");
// String config = FileOps.safeRead(Path.of("config.yaml"), "key: default\n");
// FileOps.processLines(Path.of("large.log"), line -> {
//     if (line.contains("ERROR")) System.err.println(line);
// });

Bash: Safe file operations with error checking

#!/usr/bin/env bash
set -euo pipefail

# Safe write: write to temp file, then atomically rename
safe_write() {
    local file="$1"
    local content="$2"
    local tmp="${file}.tmp.$$"
    local dir
    dir="$(dirname "$file")"
    mkdir -p "$dir"
    printf '%s' "$content" > "$tmp"
    mv "$tmp" "$file"
}

# Safe read: check existence first, provide default
safe_read() {
    local file="$1"
    local default="${2:-}"
    if [[ -f "$file" && -r "$file" ]]; then
        cat "$file"
    else
        printf '%s' "$default"
    fi
}

# Append with timestamp (for logging)
log_append() {
    local file="$1"
    local message="$2"
    local timestamp
    timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
    printf '[%s] %s\n' "$timestamp" "$message" >> "$file"
}

# Read file line by line with error handling
read_lines() {
    local file="$1"
    if [[ ! -f "$file" ]]; then
        echo "Error: $file not found" >&2
        return 1
    fi
    while IFS= read -r line || [[ -n "$line" ]]; do
        echo "$line"
    done < "$file"
}

# Create file with restrictive permissions (for secrets)
create_secret_file() {
    local file="$1"
    local content="$2"
    # Create with 600 permissions directly
    (umask 077; printf '%s' "$content" > "$file")
    echo "Created $file with 600 permissions"
}

# Usage
# safe_write /etc/app/config.txt "key=value"
# content=$(safe_read /etc/app/config.txt "key=default")
# log_append /var/log/app.log "Application started"
# create_secret_file /etc/app/secret.key "my-secret-key-123"

See Also

• Python open() and the io module: modes, buffering, and text vs binary.
• The fs/promises API docs — every async file op Node.js ships, event-loop friendly.
• Bash redirection: the full reference for >, >>, and here-docs.
• The NIO Files API: Files, Paths, and the ATOMIC_MOVE option used in the Java example.

Companion code: runnable safe-read/write utilities for Python, Node.js, Java and Bash live in the companion repo.