Watch File Changes

Overview

File system watchers react to create, modify, delete, and rename events in real time. They power hot-reload dev servers, log tailers, and sync tools. Below is the idiomatic way to cross-platform implementations in Python, JavaScript, and Java.

When to Use

Use this resource when:

• Building development servers that reload on code changes
• Monitoring log directories for new files to process
• Triggering pipelines when upload folders receive files

Solution

Python

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class Handler(FileSystemEventHandler):
    def on_modified(self, event):
        if not event.is_directory:
            print(f"Modified: {event.src_path}")

observer = Observer()
observer.schedule(Handler(), path='./watched', recursive=True)
observer.start()

try:
    while True:
        pass
except KeyboardInterrupt:
    observer.stop()
observer.join()

JavaScript

const fs = require('fs');

// Watch a file or directory
const watcher = fs.watch('./watched', { recursive: true }, (eventType, filename) => {
    console.log(`${eventType}: ${filename}`);
});

// Cleanup
process.on('SIGINT', () => watcher.close());

Java

import java.nio.file.*;

public class FileWatcher {
    public static void watch(Path path) throws Exception {
        WatchService watchService = FileSystems.getDefault().newWatchService();
        path.register(watchService,
                StandardWatchEventKinds.ENTRY_CREATE,
                StandardWatchEventKinds.ENTRY_MODIFY,
                StandardWatchEventKinds.ENTRY_DELETE);

        while (true) {
            WatchKey key = watchService.take();
            for (WatchEvent<?> event : key.pollEvents()) {
                System.out.println(event.kind() + ": " + event.context());
            }
            key.reset();
        }
    }
}

Explanation

Watchers register with the OS kernel, which then pushes events to your process rather than requiring expensive polling. Python watchdog abstracts inotify (Linux), FSEvents (macOS), and ReadDirectoryChangesW (Windows). Node.js fs.watch delegates to the most efficient native API per platform. Java NIO WatchService uses the same underlying OS mechanisms through a standardized API.

Variants

Technology	Approach	Notes
Python	watchdog library	Cross-platform, handles edge cases like rapid renames
JavaScript	chokidar npm package	More reliable than fs.watch on macOS and Windows
Java	Apache Commons IO FileAlterationMonitor	Polling fallback for older JDKs

What Works

1. Debounce rapid events (editors often trigger multiple writes)
2. Always handle the error event / WatchService exceptions
3. Use recursive watches sparingly; they consume OS resources
4. Filter by file extension to ignore temp files (e.g., .tmp, .swp)
5. Run watchers in a dedicated thread or process to avoid blocking

Common Mistakes

1. Assuming modify events fire only once per save (editors may trigger many)
2. Not cleaning up watcher resources on shutdown, causing resource leaks
3. Watching network drives with native APIs that don’t support them
4. Ignoring rename events, which appear as separate create + delete on some OSes
5. Processing files immediately on create before the writer has closed them

FAQ

Can I watch remote or network paths?

Native watchers generally do not support network shares. Use polling libraries like chokidar with usePolling: true or FileAlterationMonitor as fallbacks.

Why do I get duplicate events?

Many text editors write files atomically (create temp, rename), triggering multiple events. Debounce with a small delay (e.g., 100 ms) before acting.

How many files can I watch at once?

OS limits vary. Linux inotify has a per-user max_user_watches limit (default ~8K). macOS FSEvents scales to millions. Avoid recursive watches on huge trees.

Advanced Solutions

Python: Debounced watcher with event coalescing

import time
import threading
from pathlib import Path
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
from collections import defaultdict
from typing import Callable

class DebouncedEventHandler(FileSystemEventHandler):
    """Coalesces rapid file events into a single callback after a quiet period."""

    def __init__(self, callback: Callable[[str, str], None],
                 debounce_seconds: float = 0.3,
                 extensions: list[str] | None = None):
        self.callback = callback
        self.debounce = debounce_seconds
        self.extensions = extensions or []
        self._pending: dict[str, dict] = {}
        self._lock = threading.Lock()
        self._timer: threading.Timer | None = None

    def _should_process(self, path: str) -> bool:
        if not self.extensions:
            return True
        return any(path.endswith(ext) for ext in self.extensions)

    def _on_event(self, event_type: str, src_path: str):
        if not self._should_process(src_path):
            return
        with self._lock:
            self._pending[src_path] = {
                "type": event_type,
                "time": time.time(),
            }
            if self._timer:
                self._timer.cancel()
            self._timer = threading.Timer(self.debounce, self._flush)
            self._timer.start()

    def _flush(self):
        with self._lock:
            for path, info in self._pending.items():
                self.callback(info["type"], path)
            self._pending.clear()

    def on_created(self, event):
        if not event.is_directory:
            self._on_event("created", event.src_path)

    def on_modified(self, event):
        if not event.is_directory:
            self._on_event("modified", event.src_path)

    def on_deleted(self, event):
        if not event.is_directory:
            self._on_event("deleted", event.src_path)

    def on_moved(self, event):
        if not event.is_directory:
            self._on_event("moved", event.dest_path)

def handle_change(event_type: str, path: str):
    print(f"[{event_type}] {path}")

observer = Observer()
handler = DebouncedEventHandler(
    callback=handle_change,
    debounce_seconds=0.3,
    extensions=[".py", ".js", ".json", ".yaml"],
)
observer.schedule(handler, path="./src", recursive=True)
observer.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()

JavaScript: Chokidar with glob filtering and async handlers

const chokidar = require('chokidar');
const path = require('path');

const watcher = chokidar.watch('./src', {
    ignored: /(^|[\/\\])\./,  // Ignore dotfiles
    persistent: true,
    ignoreInitial: true,
    followSymlinks: false,
    usePolling: false,
    interval: 100,
    binaryInterval: 300,
    awaitWriteFinish: {
        stabilityThreshold: 500,
        pollInterval: 100,
    },
});

const debounce = new Map();
function debouncedRun(file, fn, delay = 300) {
    if (debounce.has(file)) clearTimeout(debounce.get(file));
    debounce.set(file, setTimeout(() => {
        fn(file);
        debounce.delete(file);
    }, delay));
}

watcher
    .on('add', file => debouncedRun(file, f => {
        if (f.endsWith('.csv')) processCSV(f);
    }))
    .on('change', file => debouncedRun(file, f => {
        if (f.endsWith('.js')) rebuildBundle(f);
        if (f.endsWith('.css')) recompileStyles(f);
    }))
    .on('unlink', file => {
        console.log(`Deleted: ${file}`);
        cleanupCache(file);
    })
    .on('error', err => console.error('Watcher error:', err))
    .on('ready', () => console.log('Initial scan complete. Watching for changes...'));

function processCSV(file) { console.log(`Processing CSV: ${file}`); }
function rebuildBundle(file) { console.log(`Rebuilding: ${file}`); }
function recompileStyles(file) { console.log(`Recompiling CSS: ${file}`); }
function cleanupCache(file) { console.log(`Cleaning cache for: ${file}`); }

// Cleanup on exit
process.on('SIGINT', () => watcher.close().then(() => process.exit(0)));

Java: Recursive watch with thread pool

import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;

public class RecursiveWatcher {

    private final WatchService watchService;
    private final ExecutorService executor;
    private final AtomicBoolean running = new AtomicBoolean(true);
    private final ConcurrentHashMap<WatchKey, Path> keys = new ConcurrentHashMap<>();

    public RecursiveWatcher() throws Exception {
        this.watchService = FileSystems.getDefault().newWatchService();
        this.executor = Executors.newSingleThreadExecutor();
    }

    public void registerAll(Path start) throws Exception {
        Files.walkFileTree(start, new SimpleFileVisitor<>() {
            @Override
            public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws Exception {
                WatchKey key = dir.register(watchService,
                    StandardWatchEventKinds.ENTRY_CREATE,
                    StandardWatchEventKinds.ENTRY_MODIFY,
                    StandardWatchEventKinds.ENTRY_DELETE);
                keys.put(key, dir);
                return FileVisitResult.CONTINUE;
            }
        });
    }

    public void start() {
        executor.submit(() -> {
            while (running.get()) {
                try {
                    WatchKey key = watchService.poll(1, TimeUnit.SECONDS);
                    if (key == null) continue;

                    Path dir = keys.get(key);
                    for (WatchEvent<?> event : key.pollEvents()) {
                        Path fullPath = dir.resolve((Path) event.context());
                        System.out.println(event.kind() + ": " + fullPath);

                        // Auto-register new subdirectories
                        if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
                            if (Files.isDirectory(fullPath)) {
                                registerAll(fullPath);
                            }
                        }
                    }
                    key.reset();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    break;
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });
    }

    public void stop() throws Exception {
        running.set(false);
        executor.shutdown();
        executor.awaitTermination(5, TimeUnit.SECONDS);
        watchService.close();
    }
}

// Usage
// RecursiveWatcher watcher = new RecursiveWatcher();
// watcher.registerAll(Path.of("./src"));
// watcher.start();
// Runtime.getRuntime().addShutdownHook(new Thread(watcher::stop));

Bash: inotifywait for simple file watching

#!/usr/bin/env bash
set -euo pipefail

# Watch a directory with inotifywait (Linux only)
# Requires: apt install inotify-tools

WATCH_DIR="${1:-./watched}"
DEBOUNCE_SECONDS=0.3

echo "Watching: $WATCH_DIR"

inotifywait -m -r --format '%w%f|%e' \
    -e create,modify,delete,move \
    --exclude '\.(swp|tmp|log)' \
    "$WATCH_DIR" | while IFS='|' read -r file event; do
        # Debounce: skip if same file+event seen recently
        CACHE_KEY="${event}:${file}"
        if [[ -f /tmp/.watch_cache ]] && grep -q "^${CACHE_KEY}$" /tmp/.watch_cache 2>/dev/null; then
            continue
        fi
        echo "${CACHE_KEY}" >> /tmp/.watch_cache
        sleep "$DEBOUNCE_SECONDS"
        sed -i "/^${CACHE_KEY//\//\\/}$/d" /tmp/.watch_cache 2>/dev/null || true

        echo "[$(date +%H:%M:%S)] $event: $file"

        # Trigger action based on extension
        case "$file" in
            *.py) echo "  -> Python file changed, running lint..." ;;
            *.js) echo "  -> JS file changed, rebuilding bundle..." ;;
            *.csv) echo "  -> CSV file added, processing..." ;;
        esac
    done

Additional Best Practices

• For a deeper guide, see Copy and Move Files.

1. Use awaitWriteFinish to detect file completion. When watching for uploads or exports, the file may still be written when the create event fires. Wait for the writer to finish:

const watcher = chokidar.watch('./uploads', {
    awaitWriteFinish: {
        stabilityThreshold: 1000,  // File must be stable for 1s
        pollInterval: 200,         // Check every 200ms
    },
});

2. Increase inotify limits on Linux for large projects. The default max_user_watches of 8192 is insufficient for large codebases. Increase it system-wide:

# Check current limit
cat /proc/sys/fs/inotify/max_user_watches

# Increase to 524288 (512K)
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# For Docker containers, set this on the host, not inside the container

3. Filter events by file extension early. Watching all files wastes resources on temp files, swap files, and editor backups. Filter at the watcher level:

IGNORE_EXTENSIONS = {".tmp", ".swp", ".swo", ".bak", ".log", ".pid"}

class FilteredHandler(FileSystemEventHandler):
    def on_modified(self, event):
        if event.is_directory:
            return
        ext = Path(event.src_path).suffix
        if ext in IGNORE_EXTENSIONS:
            return
        print(f"Modified: {event.src_path}")

Additional Common Mistakes

1. Processing files before the write is complete. A create or modify event fires when the file is opened or written to, not when the writer closes the file. Read the file too early and you get partial data:

# Wrong: process immediately on create
def on_created(self, event):
    data = Path(event.src_path).read_text()  # May be incomplete!

# Right: wait for file to stabilize
def on_created(self, event):
    path = Path(event.src_path)
    while True:
        size1 = path.stat().st_size
        time.sleep(0.2)
        size2 = path.stat().st_size
        if size1 == size2:
            break
    data = path.read_text()

2. Not handling watcher overflow. OS event queues have finite size. Under heavy I/O, events may be dropped. Check for OVERFLOW events in Java and handle them:

for (WatchEvent<?> event : key.pollEvents()) {
    if (event.kind() == StandardWatchEventKinds.OVERFLOW) {
        System.err.println("WARNING: Watcher overflow, events may have been lost");
        rescanDirectory(keys.get(key));
        continue;
    }
    // ... handle normal events
}

3. Using fs.watch on macOS without chokidar. Node.js fs.watch uses FSEvents on macOS, which has known issues with recursive watches and atomic saves. Always prefer chokidar for production code:

// Avoid: fs.watch is unreliable on macOS
fs.watch('./src', { recursive: true }, cb);

// Prefer: chokidar handles platform quirks
const chokidar = require('chokidar');
chokidar.watch('./src').on('all', (event, path) => cb(event, path));

Additional FAQ

How do I watch for changes across multiple directories?

Register multiple paths with the same observer or watcher instance. In Python, call observer.schedule() for each directory. In Java, register each directory with the same WatchService. In Node.js, pass an array to chokidar.watch():

const watcher = chokidar.watch([
    './src/components',
    './src/layouts',
    './src/pages',
], { ignoreInitial: true });

Is this solution production-ready?

Yes. watchdog is used by Ansible, Sphinx, and Jupyter for file monitoring. chokidar is used by Webpack, Vite, Nodemon, and VS Code’s file watcher. Java NIO WatchService is used by Gradle’s incremental build, IntelliJ’s file indexing, and Elasticsearch’s config reload. inotifywait is used in production by rsync-based sync tools, log shippers, and CI/CD pipelines. The debounce pattern (coalescing rapid events) is standard in every hot-reload dev server from Next.js to Rails.

What are the performance characteristics?

inotify on Linux uses ~1KB of kernel memory per watched file. FSEvents on macOS uses a single system-wide daemon with near-zero per-file overhead. ReadDirectoryChangesW on Windows uses I/O completion ports with minimal overhead. watchdog adds 2-5ms latency per event for Python callback dispatch. chokidar adds 1-3ms for event normalization. Java WatchService has sub-millisecond event delivery. Debouncing adds delay seconds of latency but reduces event count by 80-95% for editor saves. Polling mode (fallback for network drives) uses 1-5% CPU per watched directory at 100ms intervals. A recursive watch on 10,000 files consumes ~10MB of kernel memory on Linux.

How do I debug issues with this approach?

Enable debug logging in watchdog by setting logging.basicConfig(level=logging.DEBUG). In chokidar, listen to the raw event to see unprocessed OS events. In Java, print WatchKey.isValid() to detect dropped watchers. For inotify issues on Linux, check cat /proc/sys/fs/inotify/max_user_watches and cat /proc/sys/fs/inotify/max_user_instances. For missing events, verify the watched path exists and the process has read permissions with ls -la. For duplicate events, add timestamps to log output to measure event spacing and tune debounce delay. For network drive issues, switch to polling mode and verify with stat -f /mounted/path. For memory leaks, monitor open file descriptors with lsof -p <pid> | grep inotify.