Centralize Container Logs with Fluentd and Docker
Collect, filter, and forward Docker container logs to Elasticsearch, S3, or stdout using Fluentd as a logging driver or sidecar.
Overview
Docker containers write their logs to stdout and stderr, and by default the daemon dumps them into JSON files on the host — files that are hard to query, easy to lose, and scattered across every machine you run. Fluentd is an open-source log collector that sits between your containers and your storage: it receives each log event, enriches it with metadata like container name and image, optionally parses and filters it, then forwards it to Elasticsearch, S3, Kafka, or just stdout.
This recipe covers the two ways of getting container logs into Fluentd — the built-in fluentd logging driver, which captures stdout/stderr at the daemon level with zero app changes, and the sidecar pattern, which tails log files through a shared volume for apps that don’t log to stdout. By the end you’ll have a working Fluentd → Elasticsearch → Kibana stack you can query in Kibana’s Discover tab. Every config on this page is in the companion repo, ready to run.
When to Use
- You need centralized log aggregation across your containers and hosts
- You want to enrich logs with container metadata (name, image, labels)
- You need to filter or transform logs before storing them
- You forward logs to Elasticsearch, S3, Kafka, or other destinations
- Your team queries logs in Kibana or needs long-term retention outside the host
Skip Fluentd when a simpler option fits: docker logs + a JSON driver for single-host debugging, or the Grafana/Loki stack if you’re already invested in that ecosystem — see the Grafana observability guide for the comparison.
Solution
Fluentd as Docker logging driver
# Start Fluentd first — containers using the driver fail if it's not up
docker run -d \
--name fluentd \
-p 24224:24224 \
-p 24224:24224/udp \
-v $(pwd)/fluentd.conf:/fluentd/etc/fluent.conf \
fluent/fluentd:v1.16
# Run containers with the fluentd log driver
docker run -d \
--name api \
--log-driver fluentd \
--log-opt fluentd-address=localhost:24224 \
--log-opt tag="docker.{{.Name}}" \
--log-opt mode=non-blocking \
--log-opt max-buffer-size=4m \
my-api:latest
Two of those options do real work in production. With mode=non-blocking the container keeps running even when Fluentd is slow or down — leave the default blocking mode and a hiccup in Fluentd freezes your app’s stdout writes. max-buffer-size sets the ceiling on the per-container buffer the daemon holds while Fluentd is unreachable; once it’s full, log lines start getting dropped — which beats a frozen app. Set both, always.
Fluentd configuration file
# fluentd.conf
# Source: receive logs from the Docker fluentd driver
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
# Filter: add container metadata
<filter docker.**>
@type record_transformer
<record>
hostname ${hostname}
source "docker"
</record>
</filter>
# Filter: parse JSON logs emitted by the application
<filter docker.**>
@type parser
key_name log
reserve_data true
<parse>
@type json
</parse>
</filter>
# Match: fan out to Elasticsearch AND stdout
# Fluentd routes each event to the FIRST matching <match> —
# two separate <match docker.**> blocks don't both fire.
# @type copy fans each event out to every <store> child.
<match docker.**>
@type copy
<store>
@type elasticsearch
host elasticsearch
port 9200
index_name docker-logs-%Y%m%d
type_name _doc
flush_interval 5s
</store>
<store>
@type stdout
</store>
</match>
A subtlety that bites everyone once: <match> blocks are evaluated in order and each event goes to the first one that fits — writing two <match docker.**> blocks doesn’t mean “send to both”. That’s what @type copy with <store> children is for. The daily index name (docker-logs-%Y%m%d) keeps Elasticsearch indices manageable; a single index per container is the pattern that breaks at scale.
Docker Compose with Fluentd + Elasticsearch + Kibana
# docker-compose.yml
services:
fluentd:
image: fluent/fluentd:v1.16
ports:
- "24224:24224"
- "24224:24224/udp"
volumes:
- ./fluentd.conf:/fluentd/etc/fluent.conf
depends_on:
- elasticsearch
networks:
- logging
deploy:
resources:
limits:
memory: 512M
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
environment:
- discovery.type=single-node
- xpack.security.enabled=false
ports:
- "9200:9200"
networks:
- logging
kibana:
image: docker.elastic.co/kibana/kibana:8.13.0
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
depends_on:
- elasticsearch
networks:
- logging
api:
build: .
logging:
driver: fluentd
options:
fluentd-address: localhost:24224
tag: docker.api
mode: non-blocking
max-buffer-size: 4m
depends_on:
- fluentd
networks:
- logging
networks:
logging:
driver: bridge
Notice the dedicated logging network: if you already split services into frontend and backend bridges, give the logging plane its own the same way you’d wall off any internal dependency (see Docker network isolation). The memory limit on Fluentd isn’t decoration either — under a log burst the buffer balloons, and an unbounded Fluentd ends up OOM-killed in the middle of a flush.
Sidecar pattern for log file collection
# docker-compose.yml — sidecar reads log files from shared volume
services:
api:
build: .
volumes:
- api-logs:/var/log/app
# App writes logs to /var/log/app/app.log
fluentd-sidecar:
image: fluent/fluentd:v1.16
volumes:
- api-logs:/var/log/app
- ./fluentd-sidecar.conf:/fluentd/etc/fluent.conf
# Fluentd tails the log file
volumes:
api-logs:
# fluentd-sidecar.conf — tail log files
<source>
@type tail
path /var/log/app/app.log
pos_file /var/log/app/app.log.pos
tag app.api
<parse>
@type json
time_key timestamp
time_format %Y-%m-%dT%H:%M:%S.%LZ
</parse>
</source>
<match app.**>
@type elasticsearch
host elasticsearch
port 9200
index_name app-logs-%Y%m%d
flush_interval 5s
</match>
That pos_file is what lets the sidecar survive restarts: Fluentd keeps its read offset there, so a restarted container resumes where it stopped instead of re-ingesting the whole file — or skipping entries. One thing not shown above: mount that volume read-only on the sidecar side. Fluentd never writes to it, so :ro is free hardening.
Forward to AWS S3
# fluentd.conf — S3 output for long-term archive
<match docker.**>
@type s3
aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
s3_bucket my-log-bucket
s3_region us-east-1
path logs/
buffer_path /var/log/fluent/s3
time_slice_format %Y%m%d
time_slice_wait 10m
flush_interval 30s
</match>
S3 is the cold-storage destination — cheap, durable, queryable later with Athena or a lifecycle rule into Glacier. Pull credentials from environment variables (or an IAM role on EC2/ECS) rather than hardcoding them, and combine this with the ES store via @type copy if you want hot search plus archive.
Structured JSON logging from your app
// Node.js — structured logging for Fluentd
const logger = {
info(message, meta = {}) {
console.log(JSON.stringify({
level: "info",
message,
timestamp: new Date().toISOString(),
service: "api",
...meta
}));
},
error(message, meta = {}) {
console.error(JSON.stringify({
level: "error",
message,
timestamp: new Date().toISOString(),
service: "api",
...meta
}));
}
};
logger.info("Request received", { method: "GET", path: "/users", duration: 12 });
import json
import logging
class JsonFormatter(logging.Formatter):
def format(self, record):
log_data = {
"level": record.levelname.lower(),
"message": record.getMessage(),
"timestamp": self.formatTime(record),
"service": "api",
}
return json.dumps(log_data)
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler])
That JSON output is exactly what the <filter> parser block unpacks — Fluentd turns each line into real fields, so Kibana queries like level:error AND service:api just work instead of you grepping message text. There’s a full treatment of this pattern in the structured logging guide.
Explanation
With the fluentd driver, container stdout/stderr travels to Fluentd over the Forward protocol — compact, binary, TCP-based, and native to Fluentd’s own plugins. Every log line shows up as an event, its tag coming straight from --log-opt tag= — docker.{{.Name}} yields docker.api, docker.worker, and so on. From there the pipeline never changes shape: source (ingest) → filter (transform/enrich) → match (output).
For sources you can pick forward (the Docker driver), tail for following files, http for an ingest endpoint, or syslog. For filters you get record_transformer to add fields, parser to unpack formats, and grep to drop lines you don’t want. Outputs cover elasticsearch, s3, kafka, stdout, and file.
Key concepts:
- tag: labels log streams for routing — every filter and match keys off it
- buffer: Fluentd buffers events before flushing to the destination, so an Elasticsearch outage queues logs on disk instead of dropping them
- flush_interval: how often buffered logs ship out — lower values mean near-real-time delivery but more requests and index churn
- blocking vs non-blocking: a Docker-side concern, not Fluentd’s — the driver buffers in the daemon and
mode=non-blockingtrades possible log loss for app availability
Driver vs sidecar: pick the driver when your app writes to stdout (99% of containerized apps should). Pick the sidecar when the app insists on writing to log files — legacy apps, JVMs with file appenders — or when you need per-app Fluentd config without touching the daemon. On Kubernetes, neither pattern is typical: Fluent Bit as a DaemonSet forwarding to a Fluentd aggregator is the standard topology, since it watches pod metadata natively.
Verifying the pipeline works
# 1. Fluentd is listening
docker logs fluentd | tail -5
# 2. Send a test event through the Forward protocol
echo '{"key":"value"}' | fluent-cat docker.test -h localhost -p 24224
# 3. The index exists and is growing
curl -s "localhost:9200/_cat/indices?v" | grep docker-logs
# 4. Query the docs that landed
curl -s "localhost:9200/docker-logs-*/_search?q=service:api&size=3" | jq .hits.hits
If events never reach ES, the usual suspects are the tag (does docker.** actually match your --log-opt tag?), Docker’s default docker.{{.ID}} tag producing unmatchable tags, or a parse failure that sent records to Fluentd’s @ERROR label — docker logs fluentd shows all three.
Variants
| Pattern | Log Source | Complexity | Use When |
|---|---|---|---|
| Docker driver | stdout/stderr | Low | Apps log to stdout |
| Sidecar | Log files | Medium | Apps write to files |
| HTTP input | HTTP POST | Medium | Apps send logs via HTTP |
| Syslog input | syslog | Medium | Legacy syslog apps |
| Fluent Bit + Fluentd | stdout/stderr | Medium | Kubernetes / large fleets |
Best Practices
- Use the Fluentd logging driver for apps that log to stdout/stderr, the sidecar for apps that write to files.
- Always parse logs as JSON — unstructured text is nearly unqueryable in Elasticsearch.
- Add metadata (hostname, service name, container name) with
record_transformer; raw container IDs are useless for humans. - Buffer every output; use
@type copyto fan out to several destinations instead of duplicate<match>blocks. - Keep
flush_intervalbetween 5 and 30 seconds depending on how fresh you need the data, and switch to daily indices (docker-logs-%Y%m%d) from day one. - Set
--log-opt mode=non-blockingwithmax-buffer-sizeso a slow Fluentd can’t freeze your containers. - Put Fluentd on its own network and cap its memory — it’s infrastructure, and when it dies it’s also your only window into why.
- Monitor Fluentd itself (log volume, buffer queue length, output errors); it ships its own metrics via the
monitor_agentendpoint.
Common Mistakes
- Booting app containers ahead of Fluentd — the fluentd driver needs it alive at start, so they simply don’t come up; lock the order with
depends_onor build in a retry. - Two
<match>blocks with the same pattern expecting both to fire — only the first does; use@type copyfor fan-out. - Leaving
mode=blocking(the default): if Fluentd is slow or down, your containers stall on stdout writes. - Unparsed logs reaching Elasticsearch — “works” technically but leaves you grepping
logfields forever. - A single index for everything:
docker-logsgrows unbounded;docker-logs-%Y%m%dlets you enforce retention by deleting old indices. - No memory limit on Fluentd: a log burst grows the buffer until the kernel OOM-kills Fluentd mid-flush.
- Unrotated Fluentd buffer files filling the disk over weeks — set
total_limit_sizeor watch buffer growth. - Hardcoding
fluentd-addressper environment — manage it with overrides like any other env-specific setting (see Docker Compose overrides).
Further Reading
- Fluentd documentation — forward input, outputs, buffering
- Docker Docs — Fluentd logging driver — driver options reference
- Docker Docs — Configure logging drivers — blocking modes and buffer sizing
See Also
Frequently Asked Questions
What happens if Fluentd is down when using the logging driver?
In blocking mode (the default), Docker stalls the container's process on stdout writes until Fluentd recovers. Under mode=non-blocking the daemon holds logs in a per-container buffer up to max-buffer-size; past that they're gone. Use non-blocking in production — losing some logs beats freezing the app — and size max-buffer-size to cover your expected Fluentd restart window.
Fluentd vs Fluent Bit — which should I use?
Fluent Bit is the lighter collector (C-based, ~2MB) built for edge roles like a Kubernetes DaemonSet. Fluentd is richer (Ruby-based) with ~1000 plugins but heavier (~40MB+). The common production topology is Fluent Bit on nodes collecting and forwarding to a Fluentd aggregator that does the heavy filtering and routing. For a single-host Docker setup, Fluentd alone is fine.
How do I test my Fluentd configuration?
# Send a test log through the Forward protocol
echo '{"key":"value"}' | fluent-cat docker.test -h localhost -p 24224
# Check Fluentd's own logs for parse/route errors
docker logs fluentd
# Dry-run the config before reloading
fluentd --dry-run -c /fluentd/etc/fluent.conf
Can I use Fluentd with Docker Swarm?
Yes. Deploy Fluentd as a global service (one replica per node) and point each service's --log-driver fluentd at the node's Fluentd instance. Put the aggregator tier on an overlay network so cross-node forwarding works. Note the daemon-level driver doesn't know about Swarm services — tag by service name via {{.Name}} variants or labels.
Related Resources
Docker Network Isolation and Inter-Container Security
Secure inter-container communication with custom Docker networks, network segmentation, and access control policies.
RecipeDocker Health Check Configuration for Container Reliability
Configure Docker HEALTHCHECK with curl, wget, or custom scripts so Docker can detect containers that stop responding — not just ones that crash.
RecipeOverride Docker Compose Configs per Environment
How to use Docker Compose override files for environment-specific configurations, covering dev, test, staging, production, profiles, and secrets.
GuideComplete Guide to Observability with the Grafana Stack
Set up the full Grafana observability stack: Prometheus, Loki, Tempo, Alloy and Grafana with a single docker-compose, correlation between signals, and SLO alerting.
GuideStructured Logging: JSON Logs, Correlation IDs, Aggregation
Structured logging with JSON output, correlation IDs, and log levels. Python structlog, Node.js pino, Java SLF4J, plus ELK and Loki aggregation stacks.
RecipeExpose Custom Application Metrics with Python and Prometheus
Build a custom Prometheus metrics exporter in Python using prometheus_client. Covers counters, gauges, histograms, summaries, and Flask/FastAPI integration.