StackPractices
advanced By Mathias Paulenko

Complete Guide to Elasticsearch Cluster Setup

Deploy and scale Elasticsearch clusters in production. Covers node roles, sharding, index templates, mapping, ILM, snapshots, security, and tuning.

Introduction

Elasticsearch is a distributed search and analytics engine used for full-text search, structured filtering, analytics aggregations, and vector search. A single node on your laptop works fine for development, but production is a different problem: you need quorum so the cluster can’t split-brain, enough nodes for replicas to live somewhere, security enabled, and a plan for when indices outgrow the hardware.

This guide walks through a real 3-node production setup on Elasticsearch 8.x: node roles, sharding strategy, index templates, mapping, lifecycle management, snapshots, and the tuning that keeps a cluster healthy under load. Each section ends with how to verify the step worked — a config that “should be fine” is how clusters end up red at 2 a.m.

If you only need search on top of an existing cluster, jump straight to the Full-Text Search guide. This one is about running the cluster itself.

Prerequisites

Before touching elasticsearch.yml, get the host ready — most “mysterious” ES failures are OS settings, not Elasticsearch bugs:

  • vm.max_map_count >= 262144 on Linux (sysctl -w vm.max_map_count=262144 and make it permanent in /etc/sysctl.conf). Elasticsearch memory-maps its indices; with the default 65530 it crashes or fails bootstrap checks when bootstrap.memory_lock is set.
  • File descriptors: raise the limit to 65535 for the elasticsearch user (each shard holds open files).
  • RAM: heap should be 50% of RAM, capped at ~31GB (compressed oops threshold). A 16GB-RAM node gets -Xms8g -Xmx8g.
  • No swap: either disable swap or set bootstrap.memory_lock: true — a swapped-out JVM turns millisecond queries into second-long ones.
  • Java: none needed — Elasticsearch 8 bundles its own JDK. Don’t install a system JDK unless you’ve got a reason.
  • Disk: SSDs for hot data. Plan for index size plus ~50% headroom for merges, plus space for the snapshot repository.

For local experiments, a single-node Docker container is enough. For a 3-node test cluster, run three containers on a dedicated Docker network — same configs as below. The companion repo ships a ready docker-compose.yml for exactly that, with TLS already wired: it’s the fastest way to see the bootstrap output, enrollment tokens, and _cluster/health go green before touching real hosts.

Cluster Architecture

Every node plays one or more roles. The roles you assign decide where data lives, who makes cluster-wide decisions, and who answers queries:

# elasticsearch.yml — role examples

# Master-eligible node — manages cluster state (who holds which shard)
node.roles: [master]

# Data node — stores shards, executes CRUD and search
node.roles: [data]

# Coordinating-only node — routes requests, aggregates results
node.roles: []

# Ingest node — pre-processes documents before indexing
node.roles: [data, ingest]

# Hot/warm/cold tiers for time-series data
node.roles: [data_hot]    # recent data, fast SSD
node.roles: [data_warm]   # older data, cheaper disks
node.roles: [data_cold]   # rarely queried, cheapest storage

Two decisions matter most:

How many master-eligible nodes? Always an odd number — 3 in practice. Master election needs a quorum (2 of 3), so a 3-master cluster survives one master going down without risking two masters electing themselves (split-brain, which corrupts the cluster state). On small clusters, masters can double as data nodes.

Dedicated or shared roles? Below ~10 nodes or a few TB, master+data on the same node is fine. Beyond that, dedicate 3 small nodes to node.roles: [master] only — master nodes do almost no I/O, so 2 vCPU / 4GB RAM each is enough, and isolating them keeps heavy indexing from destabilizing elections.

# Node 1 — master + data (small clusters, all 3 nodes identical)
cluster.name: prod-es-cluster
node.name: node-1
node.roles: [master, data, ingest]

# Container/development only — in production bind the private interface
network.host: 0.0.0.0
discovery.seed_hosts: ["node-1", "node-2", "node-3"]

# BOOTSTRAP ONLY — used once when the cluster first forms.
# Remove or comment out after initial startup; a stale value on
# restart is a classic way to break an existing cluster.
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]

xpack.security.enabled: true

The topology this produces:

Three-node Elasticsearch cluster: each node is master-eligible and holds data; replicas distribute across nodes so the cluster survives one node failure

Larger clusters look the same with 3 dedicated masters feeding N data nodes — the masters stay identical, you just add node.roles: [master] and point data nodes’ discovery.seed_hosts at them.

Security Bootstrap

Elasticsearch 8 turns security on by default — there’s no “enable it later” anymore. On the first node:

  1. Start Elasticsearch; it prints an autogenerated password for the elastic user and an enrollment token to stdout. Save both.
  2. Verify: curl -u elastic:<password> https://localhost:9200 -k should return cluster info over HTTPS.
  3. Add nodes with enrollment tokens — GET /_security/enrollment/node on the first node prints a token; set xpack.security.enrollment.enabled: true on new nodes and start them with --enrollment-token <token>. This configures transport TLS automatically.
  4. Create application users in Kibana or via _security/user — never let apps connect as elastic.

Skipping TLS between nodes in a multi-node cluster isn’t an option in ES 8: transport-layer TLS is mandatory once security is on, which is why the enrollment flow exists — doing it by hand means generating CAs and node certs yourself.

Sharding and Replicas

Every index splits into primary shards; each primary can have replica copies on other nodes. Two hard constraints drive the numbers:

  • Primary shard count is set at index creation and can’t change without reindexing (or shrink/split APIs with restrictions). Undersize it and you’ll reindex sooner than you’d like.
  • Replicas do two jobs: availability (a replica promotes when its primary’s node dies) and read throughput (searches hit primaries and replicas alike). They cost disk and indexing time — every write goes to every copy.
# Create an index with 3 primaries and 1 replica — 6 shard copies total
PUT /products
{
  "settings": {
    "index": {
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }
}

Shard sizing is a balancing act. Small shards waste heap (each carries fixed overhead — the Elastic guidance is a few hundred shards per GB of heap at most); big shards slow recovery and keep queries serial. Aim for 10–50GB per shard:

Index SizeShardsRationale
< 1GB1No benefit from sharding
1-10GB1-2One shard handles this well
10-50GB3-5Parallel search across shards
50-200GB5-10Max 50GB per shard
> 200GB10-20Split by time or category

For time-series data (logs, metrics) don’t pre-split huge static indices — let ILM roll over fresh indices when they hit a size threshold instead.

Replicas are the one setting you can change on a live index — no reindex needed:

PUT /products/_settings
{
  "number_of_replicas": 2
}

Verify with GET /_cat/shards/products?v — you should see each primary’s replicas on different nodes. If a replica lands on the same node as its primary, cluster.routing.allocation rules are off or you’re short a data node. For deeper partition strategy beyond one cluster, see the Database Sharding guide.

Index Templates

Index-level defaults in elasticsearch.yml were removed in Elasticsearch 8 — the template is where index settings live now. A template applies settings, mappings, and aliases to every index matching a name pattern, so logs-2026.10.20 inherits the same shape as tomorrow’s logs-2026.10.21 with zero manual work:

PUT /_index_template/logs
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.refresh_interval": "5s",
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "message": { "type": "text" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" }
      }
    }
  },
  "priority": 100
}

The priority field resolves conflicts when several templates match — highest wins. index.lifecycle.name ties new indices straight into the ILM policy below.

Mapping

Mapping is Elasticsearch’s schema: it decides which fields are searchable as full-text, which are exact-match keywords, and which are dates, numbers, or nested objects. Dynamic mapping guesses types from the first document it sees — convenient for prototyping, risky in production, where one mis-typed field ("price": "19.99" as text) breaks queries until you reindex.

Production rule: explicit mapping for fields you query, dynamic: strict or runtime for the rest:

PUT /products
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 }
        }
      },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "in_stock": { "type": "boolean" },
      "created_at": { "type": "date" },
      "tags": { "type": "keyword" },
      "description": {
        "type": "text",
        "analyzer": "english"
      },
      "metadata": { "type": "object", "enabled": false }
    }
  }
}

Two patterns worth stealing:

  • Multi-fields (name + name.keyword): text for relevance-scored search, keyword for exact matches, sorting, and aggregations — you get both from one field.
  • scaled_float for money: stores 19.99 as the integer 1999, avoiding float rounding artifacts in aggregations.
  • "enabled": false on metadata: indexes nothing, stores it in _source — the cheap way to accept arbitrary payloads without mapping explosion.

dynamic: strict rejects documents with unmapped fields — great for catching producer bugs early, painful if upstream schemas change without warning. "dynamic": "runtime" is the middle ground: unmapped fields become queryable at search time without being indexed.

Dynamic templates still earn their place for convention-driven mapping — every *_at field as date, every string as keyword:

PUT /products
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": { "type": "keyword" }
        }
      },
      {
        "dates_default": {
          "match": ".*_at",
          "mapping": { "type": "date" }
        }
      }
    ]
  }
}

Index Lifecycle Management (ILM)

Time-series indices grow forever unless something retires them. ILM automates the whole arc: hot (write-heavy) → warm (read-mostly, shrunk) → cold (cheap storage) → delete:

PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_primary_shard_size": "50gb"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "set_priority": { "priority": 0 },
          "allocate": { "require": { "data": "cold" } }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

How it plays out on real data:

  • hot: rollover creates a new index when the current one hits 7 days or 50GB primary shards — so individual shards stay in the healthy 10–50GB band instead of one index growing to terabytes.
  • warm: shrink consolidates 3 primaries into 1 (indices are read-mostly by now, fewer shards = less overhead) and forcemerge to one segment makes searches fast.
  • cold: allocate moves shards to data_cold nodes if you’ve got them; skip the action otherwise. Note the old freeze action is deprecated in ES 8 — searchable snapshots are the modern way to keep cold data queryable cheaply.
  • delete: hard cutoff at one year. Adjust to your compliance needs.
Index lifecycle: hot phase rolls over at 7 days or 50GB, warm shrinks and forcemerges at 30 days, cold allocates to cheap nodes at 90 days, delete at 365 days

Wire it up: attach index.lifecycle.name: "logs-policy" in the index template, write to an alias (logs-write) that rollover manages, and point readers at logs-*. Check progress with GET /logs-*/_ilm/explain — indices stuck in a phase tell you the policy or cluster health needs attention.

Snapshots and Backup

Replicas protect against node failure — not against DELETE /*, corruption, or a bad mapping migration. Snapshots are the only real backup. They capture an index at a point in time into a repository: shared filesystem (fs), S3, GCS, or Azure storage.

# Register a repository — filesystem for self-hosted, S3 in the cloud
PUT /_snapshot/backup_repo
{
  "type": "fs",
  "settings": {
    "location": "/mnt/backups/elasticsearch",
    "compress": true
  }
}

# Take a snapshot
PUT /_snapshot/backup_repo/snapshot_2026_01_01?wait_for_completion=true
{
  "indices": "products,orders",
  "ignore_unavailable": true
}

# Restore under a new name for verification or rollback
POST /_snapshot/backup_repo/snapshot_2026_01_01/_restore
{
  "indices": "products",
  "rename_pattern": "products",
  "rename_replacement": "restored_products"
}

Two operational rules: snapshots are incremental and cheap, so run them on a schedule (SLM — _slm/policy) rather than ad hoc; and a backup you haven’t restored is a rumor — restore into a scratch cluster or renamed index periodically to prove the pipeline works.

Cluster Health and Monitoring

The daily-driver commands for cluster state:

GET /_cluster/health                  # green/yellow/red + node and shard counts
GET /_cluster/stats                   # cluster-wide CPU/heap/disk/docs
GET /_nodes/info                      # per-node roles, JVM, OS
GET /products/_stats                  # per-index docs, size, search/indexing rates
GET /_cat/shards/products?v           # where every shard lives
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason   # what failed and why

GET /_cluster/health is the first thing to run when something feels off:

  • green — all shards assigned, normal.
  • yellow — all primaries assigned but some replicas missing. Usually a node down or too few data nodes; not data loss, but availability is degraded.
  • red — at least one primary shard unassigned: that data is unavailable. Find the reason in the unassigned.reason column — disk watermark, missing replica, or allocation rules.

For continuous monitoring, ship these to whatever you already use — Prometheus exporter, Metricbeat/Elastic Agent, or a cron on _cluster/health that pages when status isn’t green. Alert on unassigned_shards > 0 and status != green before users notice slow queries.

The metrics worth a dashboard: heap pressure per node (GET /_nodes/stats/jvm), disk usage against the watermarks, search and indexing latency percentiles, rejected requests in thread pools, and index count growth. Heap hovering near 75% between GC cycles or steady growth in rejected searches means the cluster is telling you it’s undersized — listen before it turns red.

Production Tuning

Most of Elasticsearch’s defaults are sane; the settings that actually move the needle in production:

# elasticsearch.yml — system-level tuning

# Prevent the JVM from being swapped out
bootstrap.memory_lock: true

# (Optional) thread pools are auto-sized to your CPU in ES 8 —
# override only with measured evidence of rejection queues.
# thread_pool.search.size: 50
# thread_pool.search.queue_size: 1000

# Disk watermarks — leave a real buffer, don't accept defaults blindly
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
cluster.routing.allocation.disk.watermark.flood_stage: 95%

What to actually touch, and why:

  • JVM heap (in jvm.options, not elasticsearch.yml): -Xms16g -Xmx16g — same value for both, 50% of RAM, never above ~31GB. Crossing the compressed-oops threshold makes a 32GB heap perform worse than a 31GB one.
  • Disk watermarks: at 85% Elasticsearch stops allocating new shards to the node, at 90% it tries to relocate shards off it, at 95% (flood_stage) it sets every index on the node to read-only. A red cluster from flood_stage is usually a monitoring gap, not an Elasticsearch bug.
  • index.refresh_interval: set it per-index or in templates (e.g., 5s under write load, 30s during bulk reindexing) — it’s an index setting, not a node setting. The default 1s burns CPU re-running searches per shard every second.
  • Circuit breakers (indices.breaker.*): guardrails, not tuning. Leave the defaults; a cluster that needs looser breakers has a query problem.

Bulk indexing is where throughput lives. Single-doc indexing is 10–100x slower because each request pays full transport and commit overhead:

POST /_bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Widget", "price": 9.99 }
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Gadget", "price": 19.99 }
{ "update": { "_index": "products", "_id": "3" } }
{ "doc": { "price": 14.99 } }
from elasticsearch import Elasticsearch, helpers

es = Elasticsearch("http://localhost:9200")

actions = [
    {"_index": "products", "_id": i, "_source": {"name": f"Product {i}", "price": i * 1.99}}
    for i in range(10000)
]

helpers.bulk(es, actions, chunk_size=1000, request_timeout=60)

Sweet spot is usually 5–15MB per bulk request — measure on your data; too big and requests queue, too small and you’re back to per-doc overhead. During bulk reindexing, set refresh_interval: -1 (or 30s) and number_of_replicas: 0, then restore both when the load finishes.

Rolling Upgrades and Restarts

Elasticsearch handles rolling upgrades natively — that’s the point of replicas. The safe sequence per node:

  1. Disable shard allocation: PUT /_cluster/settings with "persistent": {"cluster.routing.allocation.enable": "primaries"}. Stops the cluster relocating shards while you bounce nodes.
  2. Stop the node, upgrade the package, start it.
  3. Wait for GET /_cluster/health to show the node rejoined.
  4. Repeat per node, then re-enable allocation with "enable": "all" and wait for green.

For a plain restart (no version change), the same sequence applies minus the upgrade. Skipping the allocation-disable step turns every restart into a minutes-long shard-relocation storm — the cluster “self-heals” by copying terabytes it didn’t need to move.

Best Practices

  • Always use 3 master-eligible nodes — quorum survives a failure without split-brain
  • Keep shard size in the 10–50GB band — small shards waste heap, big shards slow recovery
  • Use explicit mapping — dynamic mapping guesses types once and you’re stuck with them until reindex
  • Set bootstrap.memory_lock: true — but only after vm.max_map_count and swap are configured; memory_lock without the OS prerequisites fails bootstrap checks
  • Use ILM for time-series data — rollover beats planning shard counts for indices you’ll never size right
  • Use index templates, not node-level index settings — index.* defaults in elasticsearch.yml don’t exist in ES 8
  • Treat snapshots as the only backup — replication is availability, not recoverability
  • Bulk for batch writes — chunk to 5–15MB, drop refresh_interval during mass reindexing
  • Monitor unassigned_shards — a silent yellow turns red on the next node failure
  • Use hot/warm/cold tiers when data ages — cold nodes can be a fraction of the cost of hot nodes
  • Write against an alias, read against a pattern — lets ILM rollover and reindexing swap indices invisibly

Troubleshooting

The scenarios that page you, and how to work them:

  • Cluster is red — primary shards unassigned: read unassigned.reason from GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason. NODE_LEFT means a node is missing (bring it back); ALLOCATION_FAILED with disk messages means watermarks — free space or raise them temporarily, then POST /_cluster/reroute?retry_failed=true to clear the stuck allocations.
  • Yellow with unassigned_shards climbing: too few data nodes for the replica count, or allocation rules exclude the nodes that are left. If the node isn’t coming back, drop number_of_replicas temporarily so the cluster goes green, then fix the node count.
  • “master not discovered yet” in logs: nodes can’t reach each other — check discovery.seed_hosts resolves and port 9300 is open between them. If the cluster already formed, also check nobody left cluster.initial_master_nodes set on a restarted node.
  • Indices locked read-only after disk full: that’s flood_stage doing its job. Free disk below the high watermark, then lift the read-only block manually: PUT /*/_settings {"index.blocks.read_only_allow_delete": null}.
  • OOM / circuit_breaking_exception: a query or aggregation exceeded the request breaker — the cluster saved itself. The fix isn’t raising the breaker, it’s finding the query: check _tasks for the offender, look for fielddata on text fields or aggregations over high-cardinality terms, and move the workload to keyword fields.

Common Mistakes

  • Over-sharding — hundreds of tiny shards burn heap on overhead; the “20 shards per GB heap” rule of thumb keeps you honest
  • Leaving cluster.initial_master_nodes set — it’s bootstrap-only; a stale value on restart can split the cluster
  • Dynamic mapping in production — one price field mapped as text breaks aggregations until you reindex
  • Setting heap above 31GB — compressed oops disables and the bigger heap performs worse
  • No ILM policy — indices grow until flood_stage locks everything read-only
  • No snapshots — discovering replication isn’t a backup happens exactly once
  • Single node in production — no replicas means no availability and no rolling restarts
  • Restarts without disabling allocation — self-inflicted shard-relocation storms
  • network.host: 0.0.0.0 outside containers — binds Elasticsearch to every interface including public ones; bind the private interface in production
  • Manually rerouting shards — cluster/reroute masks the root cause; the unassigned.reason column exists for a reason
  • Ignoring index explosion — thousands of tiny per-day indices each carry overhead; ILM rollover into size-based indices, not date-based ones, when volume allows

Further Reading

Frequently Asked Questions

How many shards should I use?

Aim for 10–50GB per shard. A 100GB index lands at 3–5 primaries; under 1GB use a single shard — extra shards only add overhead. Primaries can't change without reindexing, so plan ahead or let ILM rollover size indices for you on time-series data.

Can I run Elasticsearch on a single node?

For development and tiny workloads, yes — set discovery.type: single-node and accept that replicas can't be allocated (yellow health is normal). In production a single node means no replicas, no high availability, and every restart is downtime. Three nodes is the real minimum.

How much RAM does an Elasticsearch node need?

Minimum useful is 8GB total: 4GB heap, 4GB for Lucene's OS page cache — search speed lives in that cache, not the heap. Realistic production data nodes run 32–64GB RAM, and doubling RAM often beats doubling heap because it feeds the cache.

Should I use Elasticsearch or OpenSearch?

OpenSearch is the Apache 2.0 fork of Elasticsearch 7.10, maintained by AWS. Choose Elasticsearch for the latest capabilities and the bundled X-Pack (security, ILM, monitoring included free); choose OpenSearch if licensing purity matters more than recency. APIs stay compatible for common use, but the two diverge more every release — don't plan to migrate casually.

How do I reindex without downtime?

Write through an alias. Point products at products_v1, create products_v2 with the new mapping, POST /_reindex from old to new, then flip the alias atomically:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "products_v1", "alias": "products" } },
    { "add": { "index": "products_v2", "alias": "products" } }
  ]
}

Reads never see the switch. Keep the old index until the new one proves itself, then snapshot and delete it.

What version of Elasticsearch should I install?

The latest 8.x minor. Major-version upgrades go through the previous major's last minor (8 → 9 needs the last 8.x first), so staying current keeps upgrades cheap. If you're on 7.x still, upgrading to 8.x is the right time to adopt enrollment-based security and drop node-level index settings — they don't exist in 8.

Is Elasticsearch replication a backup?

No. Replicas guard against node failure — they copy deletes and corruption just as faithfully as data. If someone runs DELETE /products or a bad migration corrupts the index, every replica holds the same damage. Snapshots to a separate repository are the backup; replicas are availability.

How do I scale the cluster later?

Add data nodes — that's what horizontal scaling means here. New nodes join discovery.seed_hosts, shards rebalance automatically, and searches spread across the extra capacity. Add nodes when heap pressure stays high or shard counts approach the ~20-per-GB-heap ceiling; add a replica when you need read throughput, not more storage.