StackPractices
intermediate By Mathias Paulenko

Centralize Container Logs with Fluentd and Docker

Collect, filter, and forward Docker container logs to Elasticsearch, S3, or stdout using Fluentd as a logging driver or sidecar.

Overview

Docker containers write their logs to stdout and stderr, and by default the daemon dumps them into JSON files on the host — files that are hard to query, easy to lose, and scattered across every machine you run. Fluentd is an open-source log collector that sits between your containers and your storage: it receives each log event, enriches it with metadata like container name and image, optionally parses and filters it, then forwards it to Elasticsearch, S3, Kafka, or just stdout.

This recipe covers the two ways of getting container logs into Fluentd — the built-in fluentd logging driver, which captures stdout/stderr at the daemon level with zero app changes, and the sidecar pattern, which tails log files through a shared volume for apps that don’t log to stdout. By the end you’ll have a working Fluentd → Elasticsearch → Kibana stack you can query in Kibana’s Discover tab. Every config on this page is in the companion repo, ready to run.

When to Use

  • You need centralized log aggregation across your containers and hosts
  • You want to enrich logs with container metadata (name, image, labels)
  • You need to filter or transform logs before storing them
  • You forward logs to Elasticsearch, S3, Kafka, or other destinations
  • Your team queries logs in Kibana or needs long-term retention outside the host

Skip Fluentd when a simpler option fits: docker logs + a JSON driver for single-host debugging, or the Grafana/Loki stack if you’re already invested in that ecosystem — see the Grafana observability guide for the comparison.

Solution

Fluentd as Docker logging driver

# Start Fluentd first — containers using the driver fail if it's not up
docker run -d \
    --name fluentd \
    -p 24224:24224 \
    -p 24224:24224/udp \
    -v $(pwd)/fluentd.conf:/fluentd/etc/fluent.conf \
    fluent/fluentd:v1.16

# Run containers with the fluentd log driver
docker run -d \
    --name api \
    --log-driver fluentd \
    --log-opt fluentd-address=localhost:24224 \
    --log-opt tag="docker.{{.Name}}" \
    --log-opt mode=non-blocking \
    --log-opt max-buffer-size=4m \
    my-api:latest

Two of those options do real work in production. With mode=non-blocking the container keeps running even when Fluentd is slow or down — leave the default blocking mode and a hiccup in Fluentd freezes your app’s stdout writes. max-buffer-size sets the ceiling on the per-container buffer the daemon holds while Fluentd is unreachable; once it’s full, log lines start getting dropped — which beats a frozen app. Set both, always.

Fluentd configuration file

# fluentd.conf

# Source: receive logs from the Docker fluentd driver
<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>

# Filter: add container metadata
<filter docker.**>
    @type record_transformer
    <record>
        hostname ${hostname}
        source "docker"
    </record>
</filter>

# Filter: parse JSON logs emitted by the application
<filter docker.**>
    @type parser
    key_name log
    reserve_data true
    <parse>
        @type json
    </parse>
</filter>

# Match: fan out to Elasticsearch AND stdout
# Fluentd routes each event to the FIRST matching <match> —
# two separate <match docker.**> blocks don't both fire.
# @type copy fans each event out to every <store> child.
<match docker.**>
    @type copy
    <store>
        @type elasticsearch
        host elasticsearch
        port 9200
        index_name docker-logs-%Y%m%d
        type_name _doc
        flush_interval 5s
    </store>
    <store>
        @type stdout
    </store>
</match>

A subtlety that bites everyone once: <match> blocks are evaluated in order and each event goes to the first one that fits — writing two <match docker.**> blocks doesn’t mean “send to both”. That’s what @type copy with <store> children is for. The daily index name (docker-logs-%Y%m%d) keeps Elasticsearch indices manageable; a single index per container is the pattern that breaks at scale.

Docker Compose with Fluentd + Elasticsearch + Kibana

# docker-compose.yml
services:
    fluentd:
        image: fluent/fluentd:v1.16
        ports:
            - "24224:24224"
            - "24224:24224/udp"
        volumes:
            - ./fluentd.conf:/fluentd/etc/fluent.conf
        depends_on:
            - elasticsearch
        networks:
            - logging
        deploy:
            resources:
                limits:
                    memory: 512M

    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
        environment:
            - discovery.type=single-node
            - xpack.security.enabled=false
        ports:
            - "9200:9200"
        networks:
            - logging

    kibana:
        image: docker.elastic.co/kibana/kibana:8.13.0
        ports:
            - "5601:5601"
        environment:
            - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
        depends_on:
            - elasticsearch
        networks:
            - logging

    api:
        build: .
        logging:
            driver: fluentd
            options:
                fluentd-address: localhost:24224
                tag: docker.api
                mode: non-blocking
                max-buffer-size: 4m
        depends_on:
            - fluentd
        networks:
            - logging

networks:
    logging:
        driver: bridge

Notice the dedicated logging network: if you already split services into frontend and backend bridges, give the logging plane its own the same way you’d wall off any internal dependency (see Docker network isolation). The memory limit on Fluentd isn’t decoration either — under a log burst the buffer balloons, and an unbounded Fluentd ends up OOM-killed in the middle of a flush.

Sidecar pattern for log file collection

# docker-compose.yml — sidecar reads log files from shared volume
services:
    api:
        build: .
        volumes:
            - api-logs:/var/log/app
        # App writes logs to /var/log/app/app.log

    fluentd-sidecar:
        image: fluent/fluentd:v1.16
        volumes:
            - api-logs:/var/log/app
            - ./fluentd-sidecar.conf:/fluentd/etc/fluent.conf
        # Fluentd tails the log file

volumes:
    api-logs:
# fluentd-sidecar.conf — tail log files
<source>
    @type tail
    path /var/log/app/app.log
    pos_file /var/log/app/app.log.pos
    tag app.api
    <parse>
        @type json
        time_key timestamp
        time_format %Y-%m-%dT%H:%M:%S.%LZ
    </parse>
</source>

<match app.**>
    @type elasticsearch
    host elasticsearch
    port 9200
    index_name app-logs-%Y%m%d
    flush_interval 5s
</match>

That pos_file is what lets the sidecar survive restarts: Fluentd keeps its read offset there, so a restarted container resumes where it stopped instead of re-ingesting the whole file — or skipping entries. One thing not shown above: mount that volume read-only on the sidecar side. Fluentd never writes to it, so :ro is free hardening.

Forward to AWS S3

# fluentd.conf — S3 output for long-term archive
<match docker.**>
    @type s3
    aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
    aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
    s3_bucket my-log-bucket
    s3_region us-east-1
    path logs/
    buffer_path /var/log/fluent/s3
    time_slice_format %Y%m%d
    time_slice_wait 10m
    flush_interval 30s
</match>

S3 is the cold-storage destination — cheap, durable, queryable later with Athena or a lifecycle rule into Glacier. Pull credentials from environment variables (or an IAM role on EC2/ECS) rather than hardcoding them, and combine this with the ES store via @type copy if you want hot search plus archive.

Structured JSON logging from your app

// Node.js — structured logging for Fluentd
const logger = {
    info(message, meta = {}) {
        console.log(JSON.stringify({
            level: "info",
            message,
            timestamp: new Date().toISOString(),
            service: "api",
            ...meta
        }));
    },
    error(message, meta = {}) {
        console.error(JSON.stringify({
            level: "error",
            message,
            timestamp: new Date().toISOString(),
            service: "api",
            ...meta
        }));
    }
};

logger.info("Request received", { method: "GET", path: "/users", duration: 12 });
import json
import logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "level": record.levelname.lower(),
            "message": record.getMessage(),
            "timestamp": self.formatTime(record),
            "service": "api",
        }
        return json.dumps(log_data)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

That JSON output is exactly what the <filter> parser block unpacks — Fluentd turns each line into real fields, so Kibana queries like level:error AND service:api just work instead of you grepping message text. There’s a full treatment of this pattern in the structured logging guide.

Explanation

With the fluentd driver, container stdout/stderr travels to Fluentd over the Forward protocol — compact, binary, TCP-based, and native to Fluentd’s own plugins. Every log line shows up as an event, its tag coming straight from --log-opt tag= — docker.{{.Name}} yields docker.api, docker.worker, and so on. From there the pipeline never changes shape: source (ingest) → filter (transform/enrich) → match (output).

Mermaid flowchart LR diagram

For sources you can pick forward (the Docker driver), tail for following files, http for an ingest endpoint, or syslog. For filters you get record_transformer to add fields, parser to unpack formats, and grep to drop lines you don’t want. Outputs cover elasticsearch, s3, kafka, stdout, and file.

Key concepts:

  • tag: labels log streams for routing — every filter and match keys off it
  • buffer: Fluentd buffers events before flushing to the destination, so an Elasticsearch outage queues logs on disk instead of dropping them
  • flush_interval: how often buffered logs ship out — lower values mean near-real-time delivery but more requests and index churn
  • blocking vs non-blocking: a Docker-side concern, not Fluentd’s — the driver buffers in the daemon and mode=non-blocking trades possible log loss for app availability

Driver vs sidecar: pick the driver when your app writes to stdout (99% of containerized apps should). Pick the sidecar when the app insists on writing to log files — legacy apps, JVMs with file appenders — or when you need per-app Fluentd config without touching the daemon. On Kubernetes, neither pattern is typical: Fluent Bit as a DaemonSet forwarding to a Fluentd aggregator is the standard topology, since it watches pod metadata natively.

Verifying the pipeline works

# 1. Fluentd is listening
docker logs fluentd | tail -5

# 2. Send a test event through the Forward protocol
echo '{"key":"value"}' | fluent-cat docker.test -h localhost -p 24224

# 3. The index exists and is growing
curl -s "localhost:9200/_cat/indices?v" | grep docker-logs

# 4. Query the docs that landed
curl -s "localhost:9200/docker-logs-*/_search?q=service:api&size=3" | jq .hits.hits

If events never reach ES, the usual suspects are the tag (does docker.** actually match your --log-opt tag?), Docker’s default docker.{{.ID}} tag producing unmatchable tags, or a parse failure that sent records to Fluentd’s @ERROR label — docker logs fluentd shows all three.

Variants

PatternLog SourceComplexityUse When
Docker driverstdout/stderrLowApps log to stdout
SidecarLog filesMediumApps write to files
HTTP inputHTTP POSTMediumApps send logs via HTTP
Syslog inputsyslogMediumLegacy syslog apps
Fluent Bit + Fluentdstdout/stderrMediumKubernetes / large fleets

Best Practices

  • Use the Fluentd logging driver for apps that log to stdout/stderr, the sidecar for apps that write to files.
  • Always parse logs as JSON — unstructured text is nearly unqueryable in Elasticsearch.
  • Add metadata (hostname, service name, container name) with record_transformer; raw container IDs are useless for humans.
  • Buffer every output; use @type copy to fan out to several destinations instead of duplicate <match> blocks.
  • Keep flush_interval between 5 and 30 seconds depending on how fresh you need the data, and switch to daily indices (docker-logs-%Y%m%d) from day one.
  • Set --log-opt mode=non-blocking with max-buffer-size so a slow Fluentd can’t freeze your containers.
  • Put Fluentd on its own network and cap its memory — it’s infrastructure, and when it dies it’s also your only window into why.
  • Monitor Fluentd itself (log volume, buffer queue length, output errors); it ships its own metrics via the monitor_agent endpoint.

Common Mistakes

  • Booting app containers ahead of Fluentd — the fluentd driver needs it alive at start, so they simply don’t come up; lock the order with depends_on or build in a retry.
  • Two <match> blocks with the same pattern expecting both to fire — only the first does; use @type copy for fan-out.
  • Leaving mode=blocking (the default): if Fluentd is slow or down, your containers stall on stdout writes.
  • Unparsed logs reaching Elasticsearch — “works” technically but leaves you grepping log fields forever.
  • A single index for everything: docker-logs grows unbounded; docker-logs-%Y%m%d lets you enforce retention by deleting old indices.
  • No memory limit on Fluentd: a log burst grows the buffer until the kernel OOM-kills Fluentd mid-flush.
  • Unrotated Fluentd buffer files filling the disk over weeks — set total_limit_size or watch buffer growth.
  • Hardcoding fluentd-address per environment — manage it with overrides like any other env-specific setting (see Docker Compose overrides).

Further Reading

See Also

Frequently Asked Questions

What happens if Fluentd is down when using the logging driver?

In blocking mode (the default), Docker stalls the container's process on stdout writes until Fluentd recovers. Under mode=non-blocking the daemon holds logs in a per-container buffer up to max-buffer-size; past that they're gone. Use non-blocking in production — losing some logs beats freezing the app — and size max-buffer-size to cover your expected Fluentd restart window.

Fluentd vs Fluent Bit — which should I use?

Fluent Bit is the lighter collector (C-based, ~2MB) built for edge roles like a Kubernetes DaemonSet. Fluentd is richer (Ruby-based) with ~1000 plugins but heavier (~40MB+). The common production topology is Fluent Bit on nodes collecting and forwarding to a Fluentd aggregator that does the heavy filtering and routing. For a single-host Docker setup, Fluentd alone is fine.

How do I test my Fluentd configuration?
# Send a test log through the Forward protocol
echo '{"key":"value"}' | fluent-cat docker.test -h localhost -p 24224

# Check Fluentd's own logs for parse/route errors
docker logs fluentd

# Dry-run the config before reloading
fluentd --dry-run -c /fluentd/etc/fluent.conf
Can I use Fluentd with Docker Swarm?

Yes. Deploy Fluentd as a global service (one replica per node) and point each service's --log-driver fluentd at the node's Fluentd instance. Put the aggregator tier on an overlay network so cross-node forwarding works. Note the daemon-level driver doesn't know about Swarm services — tag by service name via {{.Name}} variants or labels.