StackPractices
advanced By Mathias Paulenko

Complete Guide to Kafka Stream Processing

Build real-time event pipelines with Kafka: producers, consumers, Kafka Streams, Connect, schema registry, and processing patterns.

Introduction

Moving data through Kafka is easy; processing it correctly at scale is where teams get burned. The producer/consumer APIs give you the pipe — but the moment you need to aggregate, join, window, or enrich events, you need stream processing, and that’s where most pipelines go sideways: state you didn’t plan for, ordering assumptions that break, schemas that silently change.

This guide covers the full processing stack: producers and consumers in three languages, the Kafka Streams DSL with state stores and windowing, Kafka Connect for moving data in and out, Schema Registry for contracts, and the patterns that show up in every real pipeline. Code is runnable — pick the language your service already uses.

For the broker-level side — partitions, replication, monitoring — the Kafka production operations guide pairs with this one. For the platform-agnostic version of these patterns, see Stream Processing: Event-Driven Data Pipelines.

Choosing Your Processing Model

Before writing code, pick the right tool. Kafka gives you three built-in ways to process events, and choosing wrong means either rebuilding later or running infrastructure you don’t need.

ToolModelBest forCost
Plain consumersPull-based, you manage everythingSimple transforms, sink-to-DB writesLowest — no extra infra
Kafka StreamsLibrary inside your app, local stateAggregations, joins, windowing, stateful logicMedium — embedded, no cluster
Kafka ConnectWorkers + connector configsMoving data between Kafka and external systemsMedium — needs a Connect cluster
Flink / SparkDedicated processing clustersComplex event-time logic, huge state, multi-sourceHighest — separate cluster

The rule that saves the most pain: start with a plain consumer if you just transform and write; go to Kafka Streams the moment you need state (counts, joins, windows); reach for Flink only when Kafka Streams’ event-time semantics or state size genuinely can’t keep up. Most pipelines never get past option two.

Core Concepts

  • Topic: A named stream of events (like a category).
  • Partition: A topic is split into partitions for parallelism; each partition is an ordered, append-only log.
  • Offset: The position of a message within a partition.
  • Consumer Group: A set of consumers that share partitions of a topic.
  • Broker: A Kafka server that stores and serves messages.
  • Producer: An application that publishes events to topics.
  • Consumer: An application that subscribes to topics and processes events.

A stream processing topology reads from source topics, transforms events (possibly with state), and writes to sink topics — everything in between is your code.

Kafka Streams topology: source topics feed into a filter/branch step, then a stateful aggregation backed by a local state store with a changelog topic, and results go to sink topics.

Producer

The producer contract in every language: set acks=all for durability and enable.idempotence=true so retries can’t produce duplicates. Both are cheap insurance against the two failures you’ll actually see — broker crashes mid-write and client timeouts.

Python (aiokafka)

from aiokafka import AIOKafkaProducer
import json
import asyncio

async def produce_events():
    producer = AIOKafkaProducer(
        bootstrap_servers="localhost:9092",
        value_serializer=lambda v: json.dumps(v).encode("utf-8"),
        key_serializer=lambda k: k.encode("utf-8") if k else None,
        acks="all",
        enable_idempotence=True,
    )
    await producer.start()
    try:
        for i in range(100):
            await producer.send_and_wait(
                "orders",
                key=f"order-{i}",
                value={"order_id": i, "amount": 99.99 * i, "status": "created"},
            )
    finally:
        await producer.stop()

asyncio.run(produce_events())

The key matters more than it looks: order-{i} becomes the partition key, so all events for the same order land on the same partition and stay ordered. Drop the key and Kafka round-robins — you lose per-entity ordering and any downstream join keyed on it breaks.

Java (Kafka Producer)

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class OrderProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all");
        props.put("enable.idempotence", "true");

        try (Producer<String, String> producer = new KafkaProducer<>(props)) {
            for (int i = 0; i < 100; i++) {
                ProducerRecord<String, String> record = new ProducerRecord<>(
                    "orders",
                    "order-" + i,
                    "{\"order_id\":" + i + ",\"amount\":" + (99.99 * i) + "}"
                );
                producer.send(record, (metadata, e) -> {
                    if (e != null) {
                        e.printStackTrace();
                    } else {
                        System.out.printf("Sent to partition %d offset %d%n",
                            metadata.partition(), metadata.offset());
                    }
                });
            }
        }
    }
}

The callback version is async: messages batch up and flush in the background. send().get() would serialize every write and destroy throughput — use the callback, handle errors there, and let the buffer work.

Consumer

Consumers read from partitions in the context of a consumer group. The one rule that matters: commit offsets after processing, never before — otherwise a crash loses messages silently.

Python (aiokafka)

from aiokafka import AIOKafkaConsumer
import json
import asyncio

async def consume_events():
    consumer = AIOKafkaConsumer(
        "orders",
        bootstrap_servers="localhost:9092",
        group_id="order-processor",
        value_deserializer=lambda v: json.loads(v.decode("utf-8")),
        key_deserializer=lambda k: k.decode("utf-8") if k else None,
        auto_offset_reset="earliest",
        enable_auto_commit=False,
    )
    await consumer.start()
    try:
        async for msg in consumer:
            print(f"Topic={msg.topic}, Partition={msg.partition}, Offset={msg.offset}")
            print(f"Key={msg.key}, Value={msg.value}")
            await consumer.commit()
    finally:
        await consumer.stop()

asyncio.run(consume_events())

auto_offset_reset="earliest" replays history for a new group — usually what you want for a processor. "latest" skips everything published before the consumer started, which silently drops backlog on first deploy.

JavaScript (KafkaJS)

const { Kafka } = require("kafkajs");

const kafka = new Kafka({
    clientId: "order-consumer",
    brokers: ["localhost:9092"],
});

const consumer = kafka.consumer({ groupId: "order-processor" });

async function consumeEvents() {
    await consumer.connect();
    await consumer.subscribe({ topic: "orders", fromBeginning: true });

    await consumer.run({
        eachMessage: async ({ topic, partition, message }) => {
            const event = JSON.parse(message.value.toString());
            console.log(`Processing order ${event.order_id}, amount: ${event.amount}`);

            // Process the order...
        },
    });
}

consumeEvents();

KafkaJS commits automatically after eachMessage resolves by default — the same at-least-once guarantee as manual commit, because a thrown error leaves the offset uncommitted and the message redelivers.

Kafka Streams (Java)

Kafka Streams is the right tool when processing needs state — aggregations, joins, windows — without standing up a separate processing cluster. It’s a library inside your Java app: topology, local state, exactly-once, and fault tolerance included.

A topology is a directed graph of stream processors: source nodes read topics, processor nodes transform, sink nodes write back. State lives in local RocksDB stores, backed by changelog topics so another instance can rebuild it if the first dies.

Filter and branch

import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;
import java.util.Properties;

public class OrderStreamProcessor {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("application.id", "order-processor");
        props.put("bootstrap.servers", "localhost:9092");
        props.put("default.key.serde", "org.apache.kafka.common.serialization.Serdes$StringSerde");
        props.put("default.value.serde", "org.apache.kafka.common.serialization.Serdes$StringSerde");

        StreamsBuilder builder = new StreamsBuilder();
        KStream<String, String> orders = builder.stream("orders");

        // Branch by amount
        KStream<String, String>[] branches = orders.branch(
            (key, value) -> value.contains("\"amount\":0"),
            (key, value) -> true
        );

        KStream<String, String> zeroOrders = branches[0];
        KStream<String, String> validOrders = branches[1];

        // Filter high-value orders
        KStream<String, String> highValue = validOrders.filter(
            (key, value) -> {
                // Parse and check amount > 1000
                return value.contains("\"amount\":1");
            }
        );

        highValue.to("high-value-orders");
        validOrders.to("processed-orders");

        KafkaStreams streams = new KafkaStreams(builder.build(), props);
        streams.start();
    }
}

application.id doubles as the consumer group name — every instance with the same ID shares partitions and state automatically. That’s how Streams scales: add instances, partitions reassign, work distributes.

Aggregation (count by key)

KTable<String, Long> orderCounts = orders
    .groupBy((key, value) -> key, Grouped.with(
        Serdes.String(),
        Serdes.String()
    ))
    .count();

orderCounts.toStream().to("order-counts");

KStream vs KTable: a stream is a sequence of events; a table is the current value per key — materialized state. Aggregation turns a stream into a table, and the table’s changelog flows to an internal *-changelog topic for fault recovery. GlobalKTable replicates the full table to every instance — use it for small lookup data, never for high-volume state.

Windowed aggregation (5-minute tumbling windows)

KTable<Windowed<String>, Long> windowedCounts = orders
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
    .count();

windowedCounts.toStream((key, value) -> key.key() + "@" + key.window().start())
    .to("windowed-order-counts");

Windowing is where semantics get subtle. Three window types: tumbling (fixed, non-overlapping — reports), hopping (overlapping, e.g. 5-min window every minute — trends), session (gap-driven — user activity bursts). The grace period controls how long a window accepts late events: ofSizeWithNoGrace drops anything arriving after the window closes — the default ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)) keeps a minute of buffer for stragglers.

Joining streams

KStream<String, String> orders = builder.stream("orders");
KStream<String, String> payments = builder.stream("payments");

KStream<String, String> enriched = orders.join(
    payments,
    (orderValue, paymentValue) -> orderValue + "|" + paymentValue,
    JoinWindows.of(Duration.ofMinutes(5)),
    Joined.with(Serdes.String(), Serdes.String(), Serdes.String())
);

enriched.to("enriched-orders");

Stream-stream joins are windowed: the join only matches events within the time window — a payment 10 minutes after the order won’t match. For reference data that doesn’t expire (customer lookups), join against a KTable or GlobalKTable instead — it never misses.

DSL vs Processor API

The DSL (stream, filter, groupBy, windowedBy) covers 90% of cases. Drop to the Processor API (process(), punctuators, manual state store access) only when you need: custom punctuation on wall-clock or stream-time intervals, direct state store reads mid-topology, or non-standard forwarding behavior. Mix them: build the DSL topology and attach one .process() node where you need the escape hatch.

Exactly-once with Kafka Streams

props.put("processing.guarantee", "exactly_once_v2");

With exactly_once_v2, Kafka Streams wraps read-process-write-commit in a transaction per task: duplicate events, state writes, and output records either all land or none do. The cost is a latency hit of a few hundred ms per commit interval — shrink commit.interval.ms (default 30s) for faster commits at higher overhead.

Dead Letter Topics and Error Handling

Every pipeline eventually meets a poison message: malformed JSON, a missing field, a payload bigger than the broker limit. Without a dead-letter path, that message blocks its partition forever — the consumer retries it until lag alerts fire and someone manually skips the offset.

The pattern is simple: catch per-message, route failures to a DLT with enough context to debug later, and continue.

async for msg in consumer:
    try:
        process_order(msg.value)
        await consumer.commit()
    except Exception as e:
        # Route the poison message to a dead-letter topic with context
        await producer.send_and_wait(
            "orders-dlt",
            key=msg.key,
            value={
                "original_value": msg.value,
                "error": str(e),
                "partition": msg.partition,
                "offset": msg.offset,
            },
        )
        await consumer.commit()  # Skip the poison message, keep processing

Monitor the DLT like a real topic — a growing DLT means upstream producers are shipping bad data, not that consumers are broken. Alert on DLT rate, sample it weekly, and replay it after fixes (consume it back into the pipeline with a dedicated group).

Scaling and Testing Stream Applications

Kafka Streams scales by adding instances of the same application.id: partitions rebalance across them automatically, and each instance gets its share of state. Two knobs matter for resilience: num.standby.replicas=1 keeps a passive copy of each state store on another instance (failover without a cold rebuild), and num.stream.threads controls parallelism within one instance — match it to cores.

For testing, TopologyTestDriver runs your full topology in-memory — no cluster needed:

TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props);
TestInputTopic<String, String> input =
    driver.createInputTopic("orders", Serdes.String().serializer(), Serdes.String().serializer());
TestOutputTopic<String, Long> output =
    driver.createOutputTopic("order-counts", Serdes.String().deserializer(), Serdes.Long().deserializer());

input.pipeInput("order-1", "{\"amount\": 99.99}");
assertThat(output.readValue()).isEqualTo(1L);

Test stateful logic this way — windowing, joins, aggregations all work in the driver. The gap it can’t cover is distributed behavior (rebalancing, standby failover), which needs a real cluster; the companion repo’s docker-compose gives you one.

Operationally, keep an eye on rebalance frequency: every instance restart triggers partition reassignment, and stateful tasks take longer to come back than stateless ones because they replay their changelog topics first. Long catch-up after a rebalance usually means oversized state stores — if a task holds gigabytes, consider more partitions or a shorter retention on the changelog.

Schema Registry

Schemas are contracts between producers and consumers. Without a registry, a renamed field ships a breaking change that you only notice when downstream services crash. With one, the producer’s serializer enforces compatibility at write time.

Register an Avro schema

from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema_registry_conf = {"url": "http://localhost:8081"}
schema_registry_client = SchemaRegistryClient(schema_registry_conf)

schema_str = """
{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "order_id", "type": "int"},
    {"name": "amount", "type": "double"},
    {"name": "status", "type": "string"}
  ]
}
"""

avro_serializer = AvroSerializer(schema_registry_client, schema_str)

producer_conf = {
    "bootstrap.servers": "localhost:9092",
    "key.serializer": "org.apache.kafka.common.serialization.StringSerializer",
    "value.serializer": avro_serializer,
}

producer = SerializingProducer(producer_conf)
producer.produce(
    topic="orders-avro",
    key="order-1",
    value={"order_id": 1, "amount": 99.99, "status": "created"},
)
producer.flush()

Compatibility modes define what changes are legal: backward (new schema reads old data — add fields with defaults only), forward (old schema reads new — delete fields with defaults only), full (both). The practical default is backward: consumers upgraded at their own pace never break on a producer change. JSON works without a registry, Avro/Protobuf/JSON-Schema don’t — the moment payloads get complex, register.

Kafka Connect

Connect moves data between Kafka and external systems — you write JSON configs, not code. It runs as a distributed worker cluster: each connector spawns tasks that read from a source (JDBC, S3, MongoDB) or write to a sink (Elasticsearch, warehouses), handling offsets, retries, and schema conversion through pluggable converters and transforms (SMTs).

JDBC Source Connector

{
  "name": "postgres-source",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://localhost:5432/mydb",
    "connection.user": "postgres",
    "connection.password": "secret",
    "table.whitelist": "orders",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "topic.prefix": "postgres-",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter"
  }
}

mode=incrementing polls on an auto-incrementing column — simple, but it misses updates to existing rows. For change data capture (updates, deletes), use Debezium instead: it tails the WAL and emits row-level events with full before/after state.

Elasticsearch Sink Connector

{
  "name": "es-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "processed-orders",
    "connection.url": "http://localhost:9200",
    "type.name": "_doc",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "schema.ignore": "true"
  }
}

tasks.max controls parallelism — bump it to match partition count for high-volume sinks. schema.ignore=true accepts schemaless JSON; keep it false once you move to Avro so Connect validates against the registry.

Stream Processing Patterns

PatternDescriptionUse Case
FilterDrop events that don’t match a predicateRemove invalid orders
MapTransform each eventEnrich with customer data
AggregateGroup and reduceCount orders per customer
WindowAggregate over time5-minute rolling average
JoinCombine two streamsOrders + payments
SplitRoute to several topicsHigh/low value orders
RepartitionRe-key a streamGroup by customer_id instead of order_id

Repartition deserves attention: it’s the invisible one. Any groupBy on a different key triggers an internal repartition topic — every key change means a full network hop for the whole stream. Minimize key changes, and size repartition topics like real topics (retention, monitoring).

Best Practices

  • Use acks=all for producers — writes data to all in-sync replicas
  • Enable idempotent producers — prevents duplicates on retries
  • Set enable.auto.commit=false — commit offsets after processing, not before
  • Handle deserialization errors — use a dead-letter topic for poison pills
  • Use Schema Registry — enforce schema compatibility, prevent breaking consumers
  • Partition by key for ordering — events with the same key go to the same partition
  • Size partitions correctly — aim for 10-50 MB/s throughput per partition
  • Monitor consumer lag — growing lag means consumers can’t keep up
  • Use exactly-once semantics for critical pipelines — set processing.guarantee=exactly_once_v2
  • Set retention wisely — use time-based (7 days) or size-based (1GB) retention per topic
  • Use consumer groups for parallelism — each consumer in a group gets a subset of partitions

Common Mistakes

  • Not handling poison pills — one bad message blocks the entire partition
  • Using auto.offset.reset=latest — new consumers miss historical data
  • Not setting max.poll.interval.ms — slow consumers get kicked from the group
  • Over-partitioning — too many partitions increase overhead and latency
  • Not monitoring lag — issues surface only when users complain
  • Using Kafka as a database — Kafka is a log, not a queryable store
  • Not using Schema Registry — schema changes break consumers silently
  • Producing without a key — events are distributed randomly, losing ordering guarantees

See Also

All client configs and the Streams topology on this page ship as runnable code in the companion repository — a KRaft cluster, the Streams app, and producer/consumer scripts ready for docker compose up.

Frequently Asked Questions

How many partitions should a topic have?

Start with 6-12 partitions for most topics. Add partitions if consumer lag grows or throughput exceeds 50 MB/s per partition. You can't reduce partitions later — only add. Plan for peak throughput, not average.

When should I use Kafka Streams vs a plain consumer?

Plain consumer if you just transform-and-write or feed another system. Kafka Streams the moment you need state: aggregations, joins, windows, or exactly-once processing. If it fits in a consumer loop without a Map of state you're maintaining yourself, you don't need Streams yet.

What's the difference between a KStream and a KTable?

A KStream is an unbounded sequence of events — every record is a new fact. A KTable is the current value per key — later records update earlier ones. Aggregations produce KTables; joining a stream against lookup data uses KTable or GlobalKTable on the lookup side.

How does windowing handle late events?

Windows have a grace period: how long they stay open for late-arriving events. TimeWindows.ofSizeWithNoGrace drops anything past the boundary; ofSizeAndGrace keeps a buffer. Late events beyond grace are dropped silently — alert on them if your SLA can't afford the loss.

How do I ensure exactly-once processing?

For Kafka Streams, set processing.guarantee=exactly_once_v2. For consumer-producer loops, use the transactional API: initTransactions(), beginTransaction(), send + commit offset, commitTransaction(). It costs latency — a few hundred ms per commit — so reserve it for pipelines where duplicates hurt.

What's the difference between Kafka Connect and writing my own consumer?

Connect handles offsets, retries, schema conversion, and parallelism for standard sources/sinks — you write JSON, not code. Write a consumer when your source/sink isn't covered by an existing connector or you need business logic in the loop. Don't write custom consumers for "read JDBC table to topic" — that's exactly what Connect is for.