Complete Guide to Kafka Stream Processing
Build real-time event pipelines with Kafka: producers, consumers, Kafka Streams, Connect, schema registry, and processing patterns.
Introduction
Moving data through Kafka is easy; processing it correctly at scale is where teams get burned. The producer/consumer APIs give you the pipe — but the moment you need to aggregate, join, window, or enrich events, you need stream processing, and that’s where most pipelines go sideways: state you didn’t plan for, ordering assumptions that break, schemas that silently change.
This guide covers the full processing stack: producers and consumers in three languages, the Kafka Streams DSL with state stores and windowing, Kafka Connect for moving data in and out, Schema Registry for contracts, and the patterns that show up in every real pipeline. Code is runnable — pick the language your service already uses.
For the broker-level side — partitions, replication, monitoring — the Kafka production operations guide pairs with this one. For the platform-agnostic version of these patterns, see Stream Processing: Event-Driven Data Pipelines.
Choosing Your Processing Model
Before writing code, pick the right tool. Kafka gives you three built-in ways to process events, and choosing wrong means either rebuilding later or running infrastructure you don’t need.
| Tool | Model | Best for | Cost |
|---|---|---|---|
| Plain consumers | Pull-based, you manage everything | Simple transforms, sink-to-DB writes | Lowest — no extra infra |
| Kafka Streams | Library inside your app, local state | Aggregations, joins, windowing, stateful logic | Medium — embedded, no cluster |
| Kafka Connect | Workers + connector configs | Moving data between Kafka and external systems | Medium — needs a Connect cluster |
| Flink / Spark | Dedicated processing clusters | Complex event-time logic, huge state, multi-source | Highest — separate cluster |
The rule that saves the most pain: start with a plain consumer if you just transform and write; go to Kafka Streams the moment you need state (counts, joins, windows); reach for Flink only when Kafka Streams’ event-time semantics or state size genuinely can’t keep up. Most pipelines never get past option two.
Core Concepts
- Topic: A named stream of events (like a category).
- Partition: A topic is split into partitions for parallelism; each partition is an ordered, append-only log.
- Offset: The position of a message within a partition.
- Consumer Group: A set of consumers that share partitions of a topic.
- Broker: A Kafka server that stores and serves messages.
- Producer: An application that publishes events to topics.
- Consumer: An application that subscribes to topics and processes events.
A stream processing topology reads from source topics, transforms events (possibly with state), and writes to sink topics — everything in between is your code.
Producer
The producer contract in every language: set acks=all for durability and enable.idempotence=true so retries can’t produce duplicates. Both are cheap insurance against the two failures you’ll actually see — broker crashes mid-write and client timeouts.
Python (aiokafka)
from aiokafka import AIOKafkaProducer
import json
import asyncio
async def produce_events():
producer = AIOKafkaProducer(
bootstrap_servers="localhost:9092",
value_serializer=lambda v: json.dumps(v).encode("utf-8"),
key_serializer=lambda k: k.encode("utf-8") if k else None,
acks="all",
enable_idempotence=True,
)
await producer.start()
try:
for i in range(100):
await producer.send_and_wait(
"orders",
key=f"order-{i}",
value={"order_id": i, "amount": 99.99 * i, "status": "created"},
)
finally:
await producer.stop()
asyncio.run(produce_events())
The key matters more than it looks: order-{i} becomes the partition key, so all events for the same order land on the same partition and stay ordered. Drop the key and Kafka round-robins — you lose per-entity ordering and any downstream join keyed on it breaks.
Java (Kafka Producer)
import org.apache.kafka.clients.producer.*;
import java.util.Properties;
public class OrderProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("enable.idempotence", "true");
try (Producer<String, String> producer = new KafkaProducer<>(props)) {
for (int i = 0; i < 100; i++) {
ProducerRecord<String, String> record = new ProducerRecord<>(
"orders",
"order-" + i,
"{\"order_id\":" + i + ",\"amount\":" + (99.99 * i) + "}"
);
producer.send(record, (metadata, e) -> {
if (e != null) {
e.printStackTrace();
} else {
System.out.printf("Sent to partition %d offset %d%n",
metadata.partition(), metadata.offset());
}
});
}
}
}
}
The callback version is async: messages batch up and flush in the background. send().get() would serialize every write and destroy throughput — use the callback, handle errors there, and let the buffer work.
Consumer
Consumers read from partitions in the context of a consumer group. The one rule that matters: commit offsets after processing, never before — otherwise a crash loses messages silently.
Python (aiokafka)
from aiokafka import AIOKafkaConsumer
import json
import asyncio
async def consume_events():
consumer = AIOKafkaConsumer(
"orders",
bootstrap_servers="localhost:9092",
group_id="order-processor",
value_deserializer=lambda v: json.loads(v.decode("utf-8")),
key_deserializer=lambda k: k.decode("utf-8") if k else None,
auto_offset_reset="earliest",
enable_auto_commit=False,
)
await consumer.start()
try:
async for msg in consumer:
print(f"Topic={msg.topic}, Partition={msg.partition}, Offset={msg.offset}")
print(f"Key={msg.key}, Value={msg.value}")
await consumer.commit()
finally:
await consumer.stop()
asyncio.run(consume_events())
auto_offset_reset="earliest" replays history for a new group — usually what you want for a processor. "latest" skips everything published before the consumer started, which silently drops backlog on first deploy.
JavaScript (KafkaJS)
const { Kafka } = require("kafkajs");
const kafka = new Kafka({
clientId: "order-consumer",
brokers: ["localhost:9092"],
});
const consumer = kafka.consumer({ groupId: "order-processor" });
async function consumeEvents() {
await consumer.connect();
await consumer.subscribe({ topic: "orders", fromBeginning: true });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
const event = JSON.parse(message.value.toString());
console.log(`Processing order ${event.order_id}, amount: ${event.amount}`);
// Process the order...
},
});
}
consumeEvents();
KafkaJS commits automatically after eachMessage resolves by default — the same at-least-once guarantee as manual commit, because a thrown error leaves the offset uncommitted and the message redelivers.
Kafka Streams (Java)
Kafka Streams is the right tool when processing needs state — aggregations, joins, windows — without standing up a separate processing cluster. It’s a library inside your Java app: topology, local state, exactly-once, and fault tolerance included.
A topology is a directed graph of stream processors: source nodes read topics, processor nodes transform, sink nodes write back. State lives in local RocksDB stores, backed by changelog topics so another instance can rebuild it if the first dies.
Filter and branch
import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;
import java.util.Properties;
public class OrderStreamProcessor {
public static void main(String[] args) {
Properties props = new Properties();
props.put("application.id", "order-processor");
props.put("bootstrap.servers", "localhost:9092");
props.put("default.key.serde", "org.apache.kafka.common.serialization.Serdes$StringSerde");
props.put("default.value.serde", "org.apache.kafka.common.serialization.Serdes$StringSerde");
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> orders = builder.stream("orders");
// Branch by amount
KStream<String, String>[] branches = orders.branch(
(key, value) -> value.contains("\"amount\":0"),
(key, value) -> true
);
KStream<String, String> zeroOrders = branches[0];
KStream<String, String> validOrders = branches[1];
// Filter high-value orders
KStream<String, String> highValue = validOrders.filter(
(key, value) -> {
// Parse and check amount > 1000
return value.contains("\"amount\":1");
}
);
highValue.to("high-value-orders");
validOrders.to("processed-orders");
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
}
}
application.id doubles as the consumer group name — every instance with the same ID shares partitions and state automatically. That’s how Streams scales: add instances, partitions reassign, work distributes.
Aggregation (count by key)
KTable<String, Long> orderCounts = orders
.groupBy((key, value) -> key, Grouped.with(
Serdes.String(),
Serdes.String()
))
.count();
orderCounts.toStream().to("order-counts");
KStream vs KTable: a stream is a sequence of events; a table is the current value per key — materialized state. Aggregation turns a stream into a table, and the table’s changelog flows to an internal *-changelog topic for fault recovery. GlobalKTable replicates the full table to every instance — use it for small lookup data, never for high-volume state.
Windowed aggregation (5-minute tumbling windows)
KTable<Windowed<String>, Long> windowedCounts = orders
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count();
windowedCounts.toStream((key, value) -> key.key() + "@" + key.window().start())
.to("windowed-order-counts");
Windowing is where semantics get subtle. Three window types: tumbling (fixed, non-overlapping — reports), hopping (overlapping, e.g. 5-min window every minute — trends), session (gap-driven — user activity bursts). The grace period controls how long a window accepts late events: ofSizeWithNoGrace drops anything arriving after the window closes — the default ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)) keeps a minute of buffer for stragglers.
Joining streams
KStream<String, String> orders = builder.stream("orders");
KStream<String, String> payments = builder.stream("payments");
KStream<String, String> enriched = orders.join(
payments,
(orderValue, paymentValue) -> orderValue + "|" + paymentValue,
JoinWindows.of(Duration.ofMinutes(5)),
Joined.with(Serdes.String(), Serdes.String(), Serdes.String())
);
enriched.to("enriched-orders");
Stream-stream joins are windowed: the join only matches events within the time window — a payment 10 minutes after the order won’t match. For reference data that doesn’t expire (customer lookups), join against a KTable or GlobalKTable instead — it never misses.
DSL vs Processor API
The DSL (stream, filter, groupBy, windowedBy) covers 90% of cases. Drop to the Processor API (process(), punctuators, manual state store access) only when you need: custom punctuation on wall-clock or stream-time intervals, direct state store reads mid-topology, or non-standard forwarding behavior. Mix them: build the DSL topology and attach one .process() node where you need the escape hatch.
Exactly-once with Kafka Streams
props.put("processing.guarantee", "exactly_once_v2");
With exactly_once_v2, Kafka Streams wraps read-process-write-commit in a transaction per task: duplicate events, state writes, and output records either all land or none do. The cost is a latency hit of a few hundred ms per commit interval — shrink commit.interval.ms (default 30s) for faster commits at higher overhead.
Dead Letter Topics and Error Handling
Every pipeline eventually meets a poison message: malformed JSON, a missing field, a payload bigger than the broker limit. Without a dead-letter path, that message blocks its partition forever — the consumer retries it until lag alerts fire and someone manually skips the offset.
The pattern is simple: catch per-message, route failures to a DLT with enough context to debug later, and continue.
async for msg in consumer:
try:
process_order(msg.value)
await consumer.commit()
except Exception as e:
# Route the poison message to a dead-letter topic with context
await producer.send_and_wait(
"orders-dlt",
key=msg.key,
value={
"original_value": msg.value,
"error": str(e),
"partition": msg.partition,
"offset": msg.offset,
},
)
await consumer.commit() # Skip the poison message, keep processing
Monitor the DLT like a real topic — a growing DLT means upstream producers are shipping bad data, not that consumers are broken. Alert on DLT rate, sample it weekly, and replay it after fixes (consume it back into the pipeline with a dedicated group).
Scaling and Testing Stream Applications
Kafka Streams scales by adding instances of the same application.id: partitions rebalance across them automatically, and each instance gets its share of state. Two knobs matter for resilience: num.standby.replicas=1 keeps a passive copy of each state store on another instance (failover without a cold rebuild), and num.stream.threads controls parallelism within one instance — match it to cores.
For testing, TopologyTestDriver runs your full topology in-memory — no cluster needed:
TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props);
TestInputTopic<String, String> input =
driver.createInputTopic("orders", Serdes.String().serializer(), Serdes.String().serializer());
TestOutputTopic<String, Long> output =
driver.createOutputTopic("order-counts", Serdes.String().deserializer(), Serdes.Long().deserializer());
input.pipeInput("order-1", "{\"amount\": 99.99}");
assertThat(output.readValue()).isEqualTo(1L);
Test stateful logic this way — windowing, joins, aggregations all work in the driver. The gap it can’t cover is distributed behavior (rebalancing, standby failover), which needs a real cluster; the companion repo’s docker-compose gives you one.
Operationally, keep an eye on rebalance frequency: every instance restart triggers partition reassignment, and stateful tasks take longer to come back than stateless ones because they replay their changelog topics first. Long catch-up after a rebalance usually means oversized state stores — if a task holds gigabytes, consider more partitions or a shorter retention on the changelog.
Schema Registry
Schemas are contracts between producers and consumers. Without a registry, a renamed field ships a breaking change that you only notice when downstream services crash. With one, the producer’s serializer enforces compatibility at write time.
Register an Avro schema
from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
schema_registry_conf = {"url": "http://localhost:8081"}
schema_registry_client = SchemaRegistryClient(schema_registry_conf)
schema_str = """
{
"type": "record",
"name": "Order",
"fields": [
{"name": "order_id", "type": "int"},
{"name": "amount", "type": "double"},
{"name": "status", "type": "string"}
]
}
"""
avro_serializer = AvroSerializer(schema_registry_client, schema_str)
producer_conf = {
"bootstrap.servers": "localhost:9092",
"key.serializer": "org.apache.kafka.common.serialization.StringSerializer",
"value.serializer": avro_serializer,
}
producer = SerializingProducer(producer_conf)
producer.produce(
topic="orders-avro",
key="order-1",
value={"order_id": 1, "amount": 99.99, "status": "created"},
)
producer.flush()
Compatibility modes define what changes are legal: backward (new schema reads old data — add fields with defaults only), forward (old schema reads new — delete fields with defaults only), full (both). The practical default is backward: consumers upgraded at their own pace never break on a producer change. JSON works without a registry, Avro/Protobuf/JSON-Schema don’t — the moment payloads get complex, register.
Kafka Connect
Connect moves data between Kafka and external systems — you write JSON configs, not code. It runs as a distributed worker cluster: each connector spawns tasks that read from a source (JDBC, S3, MongoDB) or write to a sink (Elasticsearch, warehouses), handling offsets, retries, and schema conversion through pluggable converters and transforms (SMTs).
JDBC Source Connector
{
"name": "postgres-source",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:postgresql://localhost:5432/mydb",
"connection.user": "postgres",
"connection.password": "secret",
"table.whitelist": "orders",
"mode": "incrementing",
"incrementing.column.name": "id",
"topic.prefix": "postgres-",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter"
}
}
mode=incrementing polls on an auto-incrementing column — simple, but it misses updates to existing rows. For change data capture (updates, deletes), use Debezium instead: it tails the WAL and emits row-level events with full before/after state.
Elasticsearch Sink Connector
{
"name": "es-sink",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max": "1",
"topics": "processed-orders",
"connection.url": "http://localhost:9200",
"type.name": "_doc",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"schema.ignore": "true"
}
}
tasks.max controls parallelism — bump it to match partition count for high-volume sinks. schema.ignore=true accepts schemaless JSON; keep it false once you move to Avro so Connect validates against the registry.
Stream Processing Patterns
| Pattern | Description | Use Case |
|---|---|---|
| Filter | Drop events that don’t match a predicate | Remove invalid orders |
| Map | Transform each event | Enrich with customer data |
| Aggregate | Group and reduce | Count orders per customer |
| Window | Aggregate over time | 5-minute rolling average |
| Join | Combine two streams | Orders + payments |
| Split | Route to several topics | High/low value orders |
| Repartition | Re-key a stream | Group by customer_id instead of order_id |
Repartition deserves attention: it’s the invisible one. Any groupBy on a different key triggers an internal repartition topic — every key change means a full network hop for the whole stream. Minimize key changes, and size repartition topics like real topics (retention, monitoring).
Best Practices
- Use
acks=allfor producers — writes data to all in-sync replicas - Enable idempotent producers — prevents duplicates on retries
- Set
enable.auto.commit=false— commit offsets after processing, not before - Handle deserialization errors — use a dead-letter topic for poison pills
- Use Schema Registry — enforce schema compatibility, prevent breaking consumers
- Partition by key for ordering — events with the same key go to the same partition
- Size partitions correctly — aim for 10-50 MB/s throughput per partition
- Monitor consumer lag — growing lag means consumers can’t keep up
- Use exactly-once semantics for critical pipelines — set
processing.guarantee=exactly_once_v2 - Set retention wisely — use time-based (7 days) or size-based (1GB) retention per topic
- Use consumer groups for parallelism — each consumer in a group gets a subset of partitions
Common Mistakes
- Not handling poison pills — one bad message blocks the entire partition
- Using
auto.offset.reset=latest— new consumers miss historical data - Not setting
max.poll.interval.ms— slow consumers get kicked from the group - Over-partitioning — too many partitions increase overhead and latency
- Not monitoring lag — issues surface only when users complain
- Using Kafka as a database — Kafka is a log, not a queryable store
- Not using Schema Registry — schema changes break consumers silently
- Producing without a key — events are distributed randomly, losing ordering guarantees
See Also
- Stream Processing: Event-Driven Data Pipelines
- Complete Guide to Apache Kafka in Production
- Event-Driven Architecture Guide
- Apache Kafka Streams Documentation (official)
- Kafka Connect Documentation (official)
- Confluent Schema Registry Fundamentals (official)
All client configs and the Streams topology on this page ship as runnable code in the companion repository — a KRaft cluster, the Streams app, and producer/consumer scripts ready for docker compose up.
Frequently Asked Questions
How many partitions should a topic have?
Start with 6-12 partitions for most topics. Add partitions if consumer lag grows or throughput exceeds 50 MB/s per partition. You can't reduce partitions later — only add. Plan for peak throughput, not average.
When should I use Kafka Streams vs a plain consumer?
Plain consumer if you just transform-and-write or feed another system. Kafka Streams the moment you need state: aggregations, joins, windows, or exactly-once processing. If it fits in a consumer loop without a Map of state you're maintaining yourself, you don't need Streams yet.
What's the difference between a KStream and a KTable?
A KStream is an unbounded sequence of events — every record is a new fact. A KTable is the current value per key — later records update earlier ones. Aggregations produce KTables; joining a stream against lookup data uses KTable or GlobalKTable on the lookup side.
How does windowing handle late events?
Windows have a grace period: how long they stay open for late-arriving events. TimeWindows.ofSizeWithNoGrace drops anything past the boundary; ofSizeAndGrace keeps a buffer. Late events beyond grace are dropped silently — alert on them if your SLA can't afford the loss.
How do I ensure exactly-once processing?
For Kafka Streams, set processing.guarantee=exactly_once_v2. For consumer-producer loops, use the transactional API: initTransactions(), beginTransaction(), send + commit offset, commitTransaction(). It costs latency — a few hundred ms per commit — so reserve it for pipelines where duplicates hurt.
What's the difference between Kafka Connect and writing my own consumer?
Connect handles offsets, retries, schema conversion, and parallelism for standard sources/sinks — you write JSON, not code. Write a consumer when your source/sink isn't covered by an existing connector or you need business logic in the loop. Don't write custom consumers for "read JDBC table to topic" — that's exactly what Connect is for.
Related Resources
Complete Guide to Event-Driven Systems
Design event-driven backends: event sourcing, CQRS, sagas, outbox pattern, idempotency, eventual consistency, and production patterns.
GuideComplete Guide to Microservices Communication
Compare sync vs async communication patterns for microservices. Covers REST, gRPC, message queues, event-driven, service mesh, and when to use each.
PatternPipes and Filters Pattern: Composable Data Pipelines
Chain processing steps with independent filters connected by pipes. A pattern for data transformation pipelines where each step is reusable and composable.
RecipeKafka Consumer Groups with Python for Scalable Streaming
Create Kafka consumer groups in Python with partition assignment, offset management, rebalance handling, and exactly-once semantics using confluent-kafka.
RecipeConsume Kafka Topics with Spring Boot Stream Listeners
Build reliable Kafka consumers in Spring Boot with @KafkaListener: concurrency, manual acks, retries with dead-letter topics, and batch listeners.
GuideComplete Guide to Elasticsearch Cluster Setup
Deploy and scale Elasticsearch clusters in production. Covers node roles, sharding, index templates, mapping, ILM, snapshots, security, and tuning.