Complete Guide to Elasticsearch Cluster Setup
Deploy and scale Elasticsearch clusters in production. Covers node roles, sharding, index templates, mapping, ILM, snapshots, security, and tuning.
Introduction
Elasticsearch is a distributed search and analytics engine used for full-text search, structured filtering, analytics aggregations, and vector search. A single node on your laptop works fine for development, but production is a different problem: you need quorum so the cluster can’t split-brain, enough nodes for replicas to live somewhere, security enabled, and a plan for when indices outgrow the hardware.
This guide walks through a real 3-node production setup on Elasticsearch 8.x: node roles, sharding strategy, index templates, mapping, lifecycle management, snapshots, and the tuning that keeps a cluster healthy under load. Each section ends with how to verify the step worked — a config that “should be fine” is how clusters end up red at 2 a.m.
If you only need search on top of an existing cluster, jump straight to the Full-Text Search guide. This one is about running the cluster itself.
Prerequisites
Before touching elasticsearch.yml, get the host ready — most “mysterious” ES failures are OS settings, not Elasticsearch bugs:
vm.max_map_count >= 262144on Linux (sysctl -w vm.max_map_count=262144and make it permanent in/etc/sysctl.conf). Elasticsearch memory-maps its indices; with the default 65530 it crashes or fails bootstrap checks whenbootstrap.memory_lockis set.- File descriptors: raise the limit to 65535 for the elasticsearch user (each shard holds open files).
- RAM: heap should be 50% of RAM, capped at ~31GB (compressed oops threshold). A 16GB-RAM node gets
-Xms8g -Xmx8g. - No swap: either disable swap or set
bootstrap.memory_lock: true— a swapped-out JVM turns millisecond queries into second-long ones. - Java: none needed — Elasticsearch 8 bundles its own JDK. Don’t install a system JDK unless you’ve got a reason.
- Disk: SSDs for hot data. Plan for index size plus ~50% headroom for merges, plus space for the snapshot repository.
For local experiments, a single-node Docker container is enough. For a 3-node test cluster, run three containers on a dedicated Docker network — same configs as below. The companion repo ships a ready docker-compose.yml for exactly that, with TLS already wired: it’s the fastest way to see the bootstrap output, enrollment tokens, and _cluster/health go green before touching real hosts.
Cluster Architecture
Every node plays one or more roles. The roles you assign decide where data lives, who makes cluster-wide decisions, and who answers queries:
# elasticsearch.yml — role examples
# Master-eligible node — manages cluster state (who holds which shard)
node.roles: [master]
# Data node — stores shards, executes CRUD and search
node.roles: [data]
# Coordinating-only node — routes requests, aggregates results
node.roles: []
# Ingest node — pre-processes documents before indexing
node.roles: [data, ingest]
# Hot/warm/cold tiers for time-series data
node.roles: [data_hot] # recent data, fast SSD
node.roles: [data_warm] # older data, cheaper disks
node.roles: [data_cold] # rarely queried, cheapest storage
Two decisions matter most:
How many master-eligible nodes? Always an odd number — 3 in practice. Master election needs a quorum (2 of 3), so a 3-master cluster survives one master going down without risking two masters electing themselves (split-brain, which corrupts the cluster state). On small clusters, masters can double as data nodes.
Dedicated or shared roles? Below ~10 nodes or a few TB, master+data on the same node is fine. Beyond that, dedicate 3 small nodes to node.roles: [master] only — master nodes do almost no I/O, so 2 vCPU / 4GB RAM each is enough, and isolating them keeps heavy indexing from destabilizing elections.
# Node 1 — master + data (small clusters, all 3 nodes identical)
cluster.name: prod-es-cluster
node.name: node-1
node.roles: [master, data, ingest]
# Container/development only — in production bind the private interface
network.host: 0.0.0.0
discovery.seed_hosts: ["node-1", "node-2", "node-3"]
# BOOTSTRAP ONLY — used once when the cluster first forms.
# Remove or comment out after initial startup; a stale value on
# restart is a classic way to break an existing cluster.
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
xpack.security.enabled: true
The topology this produces:
Larger clusters look the same with 3 dedicated masters feeding N data nodes — the masters stay identical, you just add node.roles: [master] and point data nodes’ discovery.seed_hosts at them.
Security Bootstrap
Elasticsearch 8 turns security on by default — there’s no “enable it later” anymore. On the first node:
- Start Elasticsearch; it prints an autogenerated password for the
elasticuser and an enrollment token to stdout. Save both. - Verify:
curl -u elastic:<password> https://localhost:9200 -kshould return cluster info over HTTPS. - Add nodes with enrollment tokens —
GET /_security/enrollment/nodeon the first node prints a token; setxpack.security.enrollment.enabled: trueon new nodes and start them with--enrollment-token <token>. This configures transport TLS automatically. - Create application users in Kibana or via
_security/user— never let apps connect aselastic.
Skipping TLS between nodes in a multi-node cluster isn’t an option in ES 8: transport-layer TLS is mandatory once security is on, which is why the enrollment flow exists — doing it by hand means generating CAs and node certs yourself.
Sharding and Replicas
Every index splits into primary shards; each primary can have replica copies on other nodes. Two hard constraints drive the numbers:
- Primary shard count is set at index creation and can’t change without reindexing (or
shrink/splitAPIs with restrictions). Undersize it and you’ll reindex sooner than you’d like. - Replicas do two jobs: availability (a replica promotes when its primary’s node dies) and read throughput (searches hit primaries and replicas alike). They cost disk and indexing time — every write goes to every copy.
# Create an index with 3 primaries and 1 replica — 6 shard copies total
PUT /products
{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 1
}
}
}
Shard sizing is a balancing act. Small shards waste heap (each carries fixed overhead — the Elastic guidance is a few hundred shards per GB of heap at most); big shards slow recovery and keep queries serial. Aim for 10–50GB per shard:
| Index Size | Shards | Rationale |
|---|---|---|
| < 1GB | 1 | No benefit from sharding |
| 1-10GB | 1-2 | One shard handles this well |
| 10-50GB | 3-5 | Parallel search across shards |
| 50-200GB | 5-10 | Max 50GB per shard |
| > 200GB | 10-20 | Split by time or category |
For time-series data (logs, metrics) don’t pre-split huge static indices — let ILM roll over fresh indices when they hit a size threshold instead.
Replicas are the one setting you can change on a live index — no reindex needed:
PUT /products/_settings
{
"number_of_replicas": 2
}
Verify with GET /_cat/shards/products?v — you should see each primary’s replicas on different nodes. If a replica lands on the same node as its primary, cluster.routing.allocation rules are off or you’re short a data node. For deeper partition strategy beyond one cluster, see the Database Sharding guide.
Index Templates
Index-level defaults in elasticsearch.yml were removed in Elasticsearch 8 — the template is where index settings live now. A template applies settings, mappings, and aliases to every index matching a name pattern, so logs-2026.10.20 inherits the same shape as tomorrow’s logs-2026.10.21 with zero manual work:
PUT /_index_template/logs
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.refresh_interval": "5s",
"index.lifecycle.name": "logs-policy"
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"level": { "type": "keyword" },
"message": { "type": "text" },
"service": { "type": "keyword" },
"host": { "type": "keyword" }
}
}
},
"priority": 100
}
The priority field resolves conflicts when several templates match — highest wins. index.lifecycle.name ties new indices straight into the ILM policy below.
Mapping
Mapping is Elasticsearch’s schema: it decides which fields are searchable as full-text, which are exact-match keywords, and which are dates, numbers, or nested objects. Dynamic mapping guesses types from the first document it sees — convenient for prototyping, risky in production, where one mis-typed field ("price": "19.99" as text) breaks queries until you reindex.
Production rule: explicit mapping for fields you query, dynamic: strict or runtime for the rest:
PUT /products
{
"mappings": {
"properties": {
"name": {
"type": "text",
"analyzer": "standard",
"fields": {
"keyword": { "type": "keyword", "ignore_above": 256 }
}
},
"price": { "type": "scaled_float", "scaling_factor": 100 },
"in_stock": { "type": "boolean" },
"created_at": { "type": "date" },
"tags": { "type": "keyword" },
"description": {
"type": "text",
"analyzer": "english"
},
"metadata": { "type": "object", "enabled": false }
}
}
}
Two patterns worth stealing:
- Multi-fields (
name+name.keyword):textfor relevance-scored search,keywordfor exact matches, sorting, and aggregations — you get both from one field. scaled_floatfor money: stores19.99as the integer1999, avoiding float rounding artifacts in aggregations."enabled": falseonmetadata: indexes nothing, stores it in_source— the cheap way to accept arbitrary payloads without mapping explosion.
dynamic: strict rejects documents with unmapped fields — great for catching producer bugs early, painful if upstream schemas change without warning. "dynamic": "runtime" is the middle ground: unmapped fields become queryable at search time without being indexed.
Dynamic templates still earn their place for convention-driven mapping — every *_at field as date, every string as keyword:
PUT /products
{
"mappings": {
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": { "type": "keyword" }
}
},
{
"dates_default": {
"match": ".*_at",
"mapping": { "type": "date" }
}
}
]
}
}
Index Lifecycle Management (ILM)
Time-series indices grow forever unless something retires them. ILM automates the whole arc: hot (write-heavy) → warm (read-mostly, shrunk) → cold (cheap storage) → delete:
PUT /_ilm/policy/logs-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "7d",
"max_primary_shard_size": "50gb"
},
"set_priority": { "priority": 100 }
}
},
"warm": {
"min_age": "30d",
"actions": {
"shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 },
"set_priority": { "priority": 50 }
}
},
"cold": {
"min_age": "90d",
"actions": {
"set_priority": { "priority": 0 },
"allocate": { "require": { "data": "cold" } }
}
},
"delete": {
"min_age": "365d",
"actions": { "delete": {} }
}
}
}
}
How it plays out on real data:
- hot: rollover creates a new index when the current one hits 7 days or 50GB primary shards — so individual shards stay in the healthy 10–50GB band instead of one index growing to terabytes.
- warm:
shrinkconsolidates 3 primaries into 1 (indices are read-mostly by now, fewer shards = less overhead) andforcemergeto one segment makes searches fast. - cold:
allocatemoves shards todata_coldnodes if you’ve got them; skip the action otherwise. Note the oldfreezeaction is deprecated in ES 8 — searchable snapshots are the modern way to keep cold data queryable cheaply. - delete: hard cutoff at one year. Adjust to your compliance needs.
Wire it up: attach index.lifecycle.name: "logs-policy" in the index template, write to an alias (logs-write) that rollover manages, and point readers at logs-*. Check progress with GET /logs-*/_ilm/explain — indices stuck in a phase tell you the policy or cluster health needs attention.
Snapshots and Backup
Replicas protect against node failure — not against DELETE /*, corruption, or a bad mapping migration. Snapshots are the only real backup. They capture an index at a point in time into a repository: shared filesystem (fs), S3, GCS, or Azure storage.
# Register a repository — filesystem for self-hosted, S3 in the cloud
PUT /_snapshot/backup_repo
{
"type": "fs",
"settings": {
"location": "/mnt/backups/elasticsearch",
"compress": true
}
}
# Take a snapshot
PUT /_snapshot/backup_repo/snapshot_2026_01_01?wait_for_completion=true
{
"indices": "products,orders",
"ignore_unavailable": true
}
# Restore under a new name for verification or rollback
POST /_snapshot/backup_repo/snapshot_2026_01_01/_restore
{
"indices": "products",
"rename_pattern": "products",
"rename_replacement": "restored_products"
}
Two operational rules: snapshots are incremental and cheap, so run them on a schedule (SLM — _slm/policy) rather than ad hoc; and a backup you haven’t restored is a rumor — restore into a scratch cluster or renamed index periodically to prove the pipeline works.
Cluster Health and Monitoring
The daily-driver commands for cluster state:
GET /_cluster/health # green/yellow/red + node and shard counts
GET /_cluster/stats # cluster-wide CPU/heap/disk/docs
GET /_nodes/info # per-node roles, JVM, OS
GET /products/_stats # per-index docs, size, search/indexing rates
GET /_cat/shards/products?v # where every shard lives
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason # what failed and why
GET /_cluster/health is the first thing to run when something feels off:
- green — all shards assigned, normal.
- yellow — all primaries assigned but some replicas missing. Usually a node down or too few data nodes; not data loss, but availability is degraded.
- red — at least one primary shard unassigned: that data is unavailable. Find the reason in the
unassigned.reasoncolumn — disk watermark, missing replica, or allocation rules.
For continuous monitoring, ship these to whatever you already use — Prometheus exporter, Metricbeat/Elastic Agent, or a cron on _cluster/health that pages when status isn’t green. Alert on unassigned_shards > 0 and status != green before users notice slow queries.
The metrics worth a dashboard: heap pressure per node (GET /_nodes/stats/jvm), disk usage against the watermarks, search and indexing latency percentiles, rejected requests in thread pools, and index count growth. Heap hovering near 75% between GC cycles or steady growth in rejected searches means the cluster is telling you it’s undersized — listen before it turns red.
Production Tuning
Most of Elasticsearch’s defaults are sane; the settings that actually move the needle in production:
# elasticsearch.yml — system-level tuning
# Prevent the JVM from being swapped out
bootstrap.memory_lock: true
# (Optional) thread pools are auto-sized to your CPU in ES 8 —
# override only with measured evidence of rejection queues.
# thread_pool.search.size: 50
# thread_pool.search.queue_size: 1000
# Disk watermarks — leave a real buffer, don't accept defaults blindly
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
cluster.routing.allocation.disk.watermark.flood_stage: 95%
What to actually touch, and why:
- JVM heap (in
jvm.options, notelasticsearch.yml):-Xms16g -Xmx16g— same value for both, 50% of RAM, never above ~31GB. Crossing the compressed-oops threshold makes a 32GB heap perform worse than a 31GB one. - Disk watermarks: at 85% Elasticsearch stops allocating new shards to the node, at 90% it tries to relocate shards off it, at 95% (flood_stage) it sets every index on the node to read-only. A red cluster from
flood_stageis usually a monitoring gap, not an Elasticsearch bug. index.refresh_interval: set it per-index or in templates (e.g.,5sunder write load,30sduring bulk reindexing) — it’s an index setting, not a node setting. The default 1s burns CPU re-running searches per shard every second.- Circuit breakers (
indices.breaker.*): guardrails, not tuning. Leave the defaults; a cluster that needs looser breakers has a query problem.
Bulk indexing is where throughput lives. Single-doc indexing is 10–100x slower because each request pays full transport and commit overhead:
POST /_bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Widget", "price": 9.99 }
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Gadget", "price": 19.99 }
{ "update": { "_index": "products", "_id": "3" } }
{ "doc": { "price": 14.99 } }
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch("http://localhost:9200")
actions = [
{"_index": "products", "_id": i, "_source": {"name": f"Product {i}", "price": i * 1.99}}
for i in range(10000)
]
helpers.bulk(es, actions, chunk_size=1000, request_timeout=60)
Sweet spot is usually 5–15MB per bulk request — measure on your data; too big and requests queue, too small and you’re back to per-doc overhead. During bulk reindexing, set refresh_interval: -1 (or 30s) and number_of_replicas: 0, then restore both when the load finishes.
Rolling Upgrades and Restarts
Elasticsearch handles rolling upgrades natively — that’s the point of replicas. The safe sequence per node:
- Disable shard allocation:
PUT /_cluster/settingswith"persistent": {"cluster.routing.allocation.enable": "primaries"}. Stops the cluster relocating shards while you bounce nodes. - Stop the node, upgrade the package, start it.
- Wait for
GET /_cluster/healthto show the node rejoined. - Repeat per node, then re-enable allocation with
"enable": "all"and wait for green.
For a plain restart (no version change), the same sequence applies minus the upgrade. Skipping the allocation-disable step turns every restart into a minutes-long shard-relocation storm — the cluster “self-heals” by copying terabytes it didn’t need to move.
Best Practices
- Always use 3 master-eligible nodes — quorum survives a failure without split-brain
- Keep shard size in the 10–50GB band — small shards waste heap, big shards slow recovery
- Use explicit mapping — dynamic mapping guesses types once and you’re stuck with them until reindex
- Set
bootstrap.memory_lock: true— but only aftervm.max_map_countand swap are configured; memory_lock without the OS prerequisites fails bootstrap checks - Use ILM for time-series data — rollover beats planning shard counts for indices you’ll never size right
- Use index templates, not node-level index settings —
index.*defaults inelasticsearch.ymldon’t exist in ES 8 - Treat snapshots as the only backup — replication is availability, not recoverability
- Bulk for batch writes — chunk to 5–15MB, drop refresh_interval during mass reindexing
- Monitor
unassigned_shards— a silent yellow turns red on the next node failure - Use hot/warm/cold tiers when data ages — cold nodes can be a fraction of the cost of hot nodes
- Write against an alias, read against a pattern — lets ILM rollover and reindexing swap indices invisibly
Troubleshooting
The scenarios that page you, and how to work them:
- Cluster is red — primary shards unassigned: read
unassigned.reasonfromGET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason.NODE_LEFTmeans a node is missing (bring it back);ALLOCATION_FAILEDwith disk messages means watermarks — free space or raise them temporarily, thenPOST /_cluster/reroute?retry_failed=trueto clear the stuck allocations. - Yellow with
unassigned_shardsclimbing: too few data nodes for the replica count, or allocation rules exclude the nodes that are left. If the node isn’t coming back, dropnumber_of_replicastemporarily so the cluster goes green, then fix the node count. - “master not discovered yet” in logs: nodes can’t reach each other — check
discovery.seed_hostsresolves and port 9300 is open between them. If the cluster already formed, also check nobody leftcluster.initial_master_nodesset on a restarted node. - Indices locked read-only after disk full: that’s
flood_stagedoing its job. Free disk below the high watermark, then lift the read-only block manually:PUT /*/_settings {"index.blocks.read_only_allow_delete": null}. - OOM /
circuit_breaking_exception: a query or aggregation exceeded the request breaker — the cluster saved itself. The fix isn’t raising the breaker, it’s finding the query: check_tasksfor the offender, look forfielddataon text fields or aggregations over high-cardinality terms, and move the workload tokeywordfields.
Common Mistakes
- Over-sharding — hundreds of tiny shards burn heap on overhead; the “20 shards per GB heap” rule of thumb keeps you honest
- Leaving
cluster.initial_master_nodesset — it’s bootstrap-only; a stale value on restart can split the cluster - Dynamic mapping in production — one
pricefield mapped as text breaks aggregations until you reindex - Setting heap above 31GB — compressed oops disables and the bigger heap performs worse
- No ILM policy — indices grow until flood_stage locks everything read-only
- No snapshots — discovering replication isn’t a backup happens exactly once
- Single node in production — no replicas means no availability and no rolling restarts
- Restarts without disabling allocation — self-inflicted shard-relocation storms
network.host: 0.0.0.0outside containers — binds Elasticsearch to every interface including public ones; bind the private interface in production- Manually rerouting shards —
cluster/reroutemasks the root cause; the unassigned.reason column exists for a reason - Ignoring index explosion — thousands of tiny per-day indices each carry overhead; ILM rollover into size-based indices, not date-based ones, when volume allows
Further Reading
- Elasticsearch: Tune for indexing speed — official tuning reference behind the bulk and refresh-interval guidance
- Index Lifecycle Management — official ILM docs: phases, actions, and rollover mechanics
- Size your shards — Elastic’s own shard-sizing guidance behind the 10–50GB rule
- Companion resources — downloadable
elasticsearch.yml, 3-nodedocker-compose.yml, ILM policy, and index template
Frequently Asked Questions
How many shards should I use?
Aim for 10–50GB per shard. A 100GB index lands at 3–5 primaries; under 1GB use a single shard — extra shards only add overhead. Primaries can't change without reindexing, so plan ahead or let ILM rollover size indices for you on time-series data.
Can I run Elasticsearch on a single node?
For development and tiny workloads, yes — set discovery.type: single-node and accept that replicas can't be allocated (yellow health is normal). In production a single node means no replicas, no high availability, and every restart is downtime. Three nodes is the real minimum.
How much RAM does an Elasticsearch node need?
Minimum useful is 8GB total: 4GB heap, 4GB for Lucene's OS page cache — search speed lives in that cache, not the heap. Realistic production data nodes run 32–64GB RAM, and doubling RAM often beats doubling heap because it feeds the cache.
Should I use Elasticsearch or OpenSearch?
OpenSearch is the Apache 2.0 fork of Elasticsearch 7.10, maintained by AWS. Choose Elasticsearch for the latest capabilities and the bundled X-Pack (security, ILM, monitoring included free); choose OpenSearch if licensing purity matters more than recency. APIs stay compatible for common use, but the two diverge more every release — don't plan to migrate casually.
How do I reindex without downtime?
Write through an alias. Point products at products_v1, create products_v2 with the new mapping, POST /_reindex from old to new, then flip the alias atomically:
POST /_aliases
{
"actions": [
{ "remove": { "index": "products_v1", "alias": "products" } },
{ "add": { "index": "products_v2", "alias": "products" } }
]
}
Reads never see the switch. Keep the old index until the new one proves itself, then snapshot and delete it.
What version of Elasticsearch should I install?
The latest 8.x minor. Major-version upgrades go through the previous major's last minor (8 → 9 needs the last 8.x first), so staying current keeps upgrades cheap. If you're on 7.x still, upgrading to 8.x is the right time to adopt enrollment-based security and drop node-level index settings — they don't exist in 8.
Is Elasticsearch replication a backup?
No. Replicas guard against node failure — they copy deletes and corruption just as faithfully as data. If someone runs DELETE /products or a bad migration corrupts the index, every replica holds the same damage. Snapshots to a separate repository are the backup; replicas are availability.
How do I scale the cluster later?
Add data nodes — that's what horizontal scaling means here. New nodes join discovery.seed_hosts, shards rebalance automatically, and searches spread across the extra capacity. Add nodes when heap pressure stays high or shard counts approach the ~20-per-GB-heap ceiling; add a replica when you need read throughput, not more storage.
Related Resources
Complete Guide to Database Sharding
Master database sharding: shard keys, partitioning strategies, consistent hashing, cross-shard queries, resharding, Vitess and Citus.
GuideFull-Text Search: Implement Search That Actually Works
A practical guide to full-text search: PostgreSQL tsvector, Elasticsearch indexing, query design, relevance tuning, and building search that users trust with autocomplete, faceting, and typo tolerance.
RecipeHow to Use Elasticsearch Aggregations (With Examples)
Use Elasticsearch aggregations for faceted search, metrics, and time-series analytics. Step-by-step examples for terms, date_histogram, range, and composite.
GuideComplete Guide to Kafka Stream Processing
Build real-time event pipelines with Kafka: producers, consumers, Kafka Streams, Connect, schema registry, and processing patterns.
GuideComplete Guide to PostgreSQL Replication
Set up and operate PostgreSQL replication in production: streaming and logical replication, synchronous commit, Patroni failover, and lag monitoring.
GuideDatabase Denormalization: Storage for Read Speed
A practical guide to database denormalization: when to trade storage for read performance, common patterns, and how to keep derived data consistent.