Service Discovery

Overview

Service discovery is the mechanism by which microservices locate and communicate with each other in live environments where IP addresses change constantly. Instead of hard-coding endpoints, services register themselves in a registry and clients query it to find healthy instances. Combined with health checks, it enables self-healing systems that route around failures automatically.

When to Use

Use this resource when:

• Running microservices on Kubernetes, ECS, or auto-scaling groups where IPs are ephemeral
• You need automatic failover when service instances crash or become unhealthy
• Load balancing across multiple instances without manual configuration updates
• Implementing blue-green deployments or canary releases that require live traffic routing

Solution

Consul Service Registration (Go)

import "github.com/hashicorp/consul/api"

func registerService(consulAddr, serviceID, name, host string, port int) error {
    config := api.DefaultConfig()
    config.Address = consulAddr
    client, err := api.NewClient(config)
    if err != nil {
        return err
    }

    registration := &api.AgentServiceRegistration{
        ID:      serviceID,
        Name:    name,
        Address: host,
        Port:    port,
        Check: &api.AgentServiceCheck{
            HTTP:     fmt.Sprintf("http://%s:%d/health", host, port),
            Interval: "10s",
            Timeout:  "5s",
        },
    }

    return client.Agent().ServiceRegister(registration)
}

DNS-Based Discovery (Kubernetes)

apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment
  ports:
    - port: 8080
      targetPort: 8080

# Services discover each other via DNS
PAYMENT_URL=http://payment-service:8080

Client-Side Load Balancing with Eureka (Java/Spring)

@SpringBootApplication
@EnableDiscoveryClient
public class OrderService {
    @Bean
    @LoadBalanced
    public WebClient.Builder webClientBuilder() {
        return WebClient.builder();
    }
}

@Service
public class OrderProcessor {
    @Autowired
    private WebClient.Builder webClientBuilder;

    public Mono<PaymentResult> processPayment(PaymentRequest request) {
        return webClientBuilder.build()
            .post()
            .uri("lb://payment-service/payments")
            .bodyValue(request)
            .retrieve()
            .bodyToMono(PaymentResult.class);
    }
}

Explanation

Three discovery patterns:

Pattern	Mechanism	Best For
Client-side	Client queries registry; picks instance	High performance; language-native
Server-side	Load balancer queries registry; client uses one URL	Simpler clients; central control
DNS-based	Service names resolve to IPs via DNS	Kubernetes; zero client changes

Health check integration:

• Services register with a health endpoint
• Registry polls health checks periodically
• Unhealthy instances are removed from the pool
• Clients cache registry data and refresh on failure

Variants

Tool	Model	Language	Notable capabilities
Consul	Client + server	Any	Multi-datacenter; KV store; ACLs
Eureka	Client-side	Java	Netflix OSS; Spring integration
etcd	Server-side	Any	Kubernetes default; Raft consensus
Zookeeper	Server-side	Any	Mature; strong consistency
AWS Cloud Map	Server-side	Any	AWS-native; ECS integration

What Works

• Heartbeat with TTL: Services must renew registration or be auto-deregistered
• Cache with fallback: Clients should cache instance lists and use stale data briefly if registry is unreachable
• Zone-aware routing: Prefer instances in the same AZ to reduce latency and data transfer costs
• Metadata for routing: Tag instances with versions to enable canary and A/B testing
• Secure with mTLS: Encrypt service-to-service communication; authenticate registered services. See API security checklist.

Common Mistakes

1. No health checks: Unregistered dead instances still receive traffic
2. Thundering herd: All clients querying the registry simultaneously under load
3. Ignoring deregistration: Crashed services remain in the pool until TTL expires
4. Hard-coding fallback IPs: Defeats the purpose of live discovery
5. Skipping retries: One failed instance should trigger a retry on another, not fail the request. Use retry with exponential backoff for resilient clients.

Additional Common Mistakes

1. Registry as single point of failure. If the registry goes down, all service discovery fails. Run registries in clusters (Consul: 3-5 nodes, Eureka: peer-to-peer) and cache instance lists client-side.

2. No connection pooling with discovered endpoints. Creating a new HTTP connection per request to a discovered instance is expensive. Pool connections per instance and recycle them:

import requests
from requests.adapters import HTTPAdapter

class PooledServiceClient:
    def __init__(self):
        self.sessions: dict[str, requests.Session] = {}

    def get_session(self, instance_url: str) -> requests.Session:
        if instance_url not in self.sessions:
            session = requests.Session()
            adapter = HTTPAdapter(pool_connections=10, pool_maxsize=100)
            session.mount('http://', adapter)
            self.sessions[instance_url] = session
        return self.sessions[instance_url]

3. Ignoring startup race conditions. A service that registers before it’s ready to accept traffic will receive requests it can’t handle. Register only after passing startup checks, and use readiness probes in Kubernetes to gate traffic.