Data Pipeline Architecture: Batch, Streaming, Lambda, Kappa

Introduction

The most expensive architecture mistake in data engineering isn’t picking the wrong tool — it’s building streaming infrastructure when a nightly batch job would have done the job. Teams adopt Kafka and Flink because the use case sounds real-time, then spend months maintaining event-time semantics, watermarks and replay logic for dashboards that executives look at once a day.

This guide walks through the four architectures you’ll actually encounter — batch, streaming, Lambda and Kappa — plus the ETL vs ELT decision that determines where transformations run. Each section has working code, and the decision table at the end maps requirements to the pattern that fits.

Pipeline Patterns Overview

Pattern	Latency	Codebases	Complexity	Best for
Batch	Hours–days	1	Low	Reporting, historical analysis, heavy joins
Streaming	Seconds–minutes	1	High	Alerts, live dashboards, event-driven systems
Lambda	Both	2	Very high	Exact history + real-time, regulated data
Kappa	Seconds–minutes	1	High	Streaming-first teams with replayable logs
ETL	n/a	—	—	Transform outside the warehouse, mask before landing
ELT	n/a	—	—	Transform inside the warehouse with dbt/SQL

Batch pipelines extract, transform and load on a schedule — Airflow, Spark, dbt and Snowflake are the usual suspects. Streaming pipelines process events as they arrive, through Kafka, Flink, Kinesis or Pulsar. Lambda keeps a batch layer for historical accuracy alongside a speed layer for real-time views — at the cost of maintaining the same logic twice. Kappa collapses that into a single streaming codebase, treating batch as replay of a bounded stream, which only works if your log retains enough history. ETL transforms before loading; ELT loads raw data first and transforms inside the warehouse.

Batch Processing

Batch is where most pipelines should start — and where many should stay. It’s simpler to debug, cheaper to run, and a failed run can just be re-executed.

Implementation with Airflow + Spark

For a deeper Airflow walkthrough (operators, XCom, sensors), see the Apache Airflow guide.

# dags/etl_pipeline.py — Airflow DAG for batch ETL
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "data-team",
    "depends_on_past": False,
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
}

dag = DAG(
    "daily_etl_pipeline",
    default_args=default_args,
    schedule="0 2 * * *",  # Daily at 2 AM
    start_date=datetime(2026, 1, 1),
    catchup=False,
    max_active_runs=1,
)

# Step 1: Extract from source databases to S3
extract_orders = SparkSubmitOperator(
    task_id="extract_orders",
    conn_id="spark_default",
    application="/opt/airflow/jobs/extract_orders.py",
    application_args=["--date", "{{ ds }}", "--output", "s3://staging/orders/"],
    dag=dag,
)

# Step 2: Transform with Spark
transform_orders = SparkSubmitOperator(
    task_id="transform_orders",
    conn_id="spark_default",
    application="/opt/airflow/jobs/transform_orders.py",
    application_args=[
        "--input", "s3://staging/orders/{{ ds }}/",
        "--output", "s3://processed/orders/{{ ds }}/",
    ],
    dag=dag,
)

# Step 3: Load into Snowflake
load_orders = SnowflakeOperator(
    task_id="load_orders",
    sql="""
        COPY INTO orders_staging
        FROM s3://processed/orders/{{ ds }}/
        STORAGE_INTEGRATION = orders_s3_int
        FILE_FORMAT = (TYPE = PARQUET);
    """,
    snowflake_conn_id="snowflake_default",
    dag=dag,
)

# Step 4: dbt models
run_dbt_models = BashOperator(
    task_id="run_dbt_models",
    bash_command="cd /opt/dbt && dbt run --select staging+ --vars '{date: {{ ds }}}'",
    dag=dag,
)

extract_orders >> transform_orders >> load_orders >> run_dbt_models

Note the STORAGE_INTEGRATION in the COPY INTO: it references a Snowflake storage integration backed by an IAM role. Older examples interpolate AWS keys into the SQL — those end up in query history, logs and rendered Airflow templates, so don’t copy that pattern.

Spark batch job

# jobs/transform_orders.py — Spark batch transformation
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
import argparse

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", required=True)
    parser.add_argument("--output", required=True)
    args = parser.parse_args()

    spark = SparkSession.builder.appName("TransformOrders").getOrCreate()

    # Read from staging
    df = spark.read.parquet(args.input)

    # Transform: clean, enrich, aggregate
    transformed = (
        df
        .filter(F.col("status").isin("completed", "shipped"))
        .withColumn("order_date", F.to_date("created_at"))
        .withColumn("total_amount", F.col("subtotal") + F.col("tax") + F.col("shipping"))
        .groupBy("order_date", "region")
        .agg(
            F.count("*").alias("order_count"),
            F.sum("total_amount").alias("revenue"),
            F.avg("total_amount").alias("avg_order_value"),
        )
    )

    # Write to processed zone — overwrite makes the run idempotent
    transformed.write.mode("overwrite").partitionBy("order_date").parquet(args.output)
    spark.stop()

if __name__ == "__main__":
    main()

Streaming Processing

Streaming earns its complexity when the business actually reacts to events within seconds: fraud scoring, live inventory, real-time alerting. If the consumer reads results tomorrow morning, batch was enough.

For the full streaming deep-dive — state management, exactly-once semantics, Kafka Streams vs Flink — see the stream processing guide.

Implementation with Kafka + Flink

// OrderStreamingJob.java — Flink 1.17+/2.x streaming job
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.AggregateFunction;
import org.apache.flink.api.connector.source.Source;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.connector.kafka.sink.KafkaSink;
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema;
import org.apache.flink.formats.json.JsonNodeDeserializationSchema;
import org.apache.flink.formats.json.JsonNodeSerializationSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.time.Duration;

public class OrderStreamingJob {

    // AggregateFunction: accumulator = {orderCount, totalRevenue}
    public static class OrderAggregator
            implements AggregateFunction<JsonNode, double[], JsonNode> {

        @Override public double[] createAccumulator() { return new double[]{0, 0.0}; }

        @Override public double[] add(JsonNode order, double[] acc) {
            acc[0] += 1;
            acc[1] += order.get("total_amount").asDouble();
            return acc;
        }

        @Override public JsonNode getResult(double[] acc) {
            ObjectNode out = new ObjectMapper().createObjectNode();
            out.put("order_count", (int) acc[0]);
            out.put("total_revenue", acc[1]);
            return out;
        }

        @Override public double[] merge(double[] a, double[] b) {
            a[0] += b[0]; a[1] += b[1];
            return a;
        }
    }

    public static void main(String[] args) throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Consume from Kafka
        KafkaSource<JsonNode> source = KafkaSource.<JsonNode>builder()
            .setBootstrapServers("kafka:9092")
            .setTopics("orders")
            .setGroupId("order-processor")
            .setStartingOffsets(OffsetsInitializer.earliest())
            .setValueOnlyDeserializer(new JsonNodeDeserializationSchema())
            .build();

        DataStream<JsonNode> orders = env.fromSource(
            source,
            // Event-time windows need watermarks, or they never fire
            WatermarkStrategy.<JsonNode>forBoundedOutOfOrderness(Duration.ofSeconds(30))
                .withTimestampAssigner((order, ts) -> order.get("created_at").asLong()),
            "Kafka Orders"
        );

        // Process: filter, enrich, aggregate
        DataStream<JsonNode> aggregated = orders
            .filter(node -> node.has("status") && "completed".equals(node.get("status").asText()))
            .map(node -> {
                ObjectNode enriched = node.deepCopy();
                double total = node.get("subtotal").asDouble()
                    + node.get("tax").asDouble()
                    + node.get("shipping").asDouble();
                enriched.put("total_amount", total);
                return (JsonNode) enriched;
            })
            .returns(JsonNode.class)
            // 5-minute tumbling windows on event time
            .keyBy(node -> node.get("region").asText())
            .window(TumblingEventTimeWindows.of(Time.minutes(5)))
            .aggregate(new OrderAggregator());

        // Write to Kafka output topic
        KafkaSink<JsonNode> sink = KafkaSink.<JsonNode>builder()
            .setBootstrapServers("kafka:9092")
            .setRecordSerializer(
                KafkaRecordSerializationSchema.builder()
                    .setTopic("order-metrics")
                    .setValueSerializationSchema(new JsonNodeSerializationSchema())
                    .build()
            )
            .build();

        aggregated.sinkTo(sink);
        env.execute("Order Streaming Job");
    }
}

Two details matter here and are easy to get wrong. FlinkKafkaConsumer/FlinkKafkaProducer were deprecated in Flink 1.17 and removed in Flink 2.x — use KafkaSource/KafkaSink. And event-time windows without a WatermarkStrategy never trigger, which is the classic “my job runs but produces nothing” bug.

Python streaming with Faust

# streaming/order_processor.py — pip install faust-streaming (maintained fork)
import faust

app = faust.App(
    "order-processor",
    broker="kafka://kafka:9092",
    store="rocksdb://",
)

class Order(faust.Record, serializer="json"):
    order_id: str
    customer_id: str
    status: str
    subtotal: float
    tax: float
    shipping: float
    region: str

class OrderMetric(faust.Record, serializer="json"):
    region: str
    order_count: int
    total_revenue: float

orders_topic = app.topic("orders", value_type=Order)
metrics_topic = app.topic("order-metrics", value_type=OrderMetric)

# Stateful tables — must exist before the agent that populates them
region_counts = app.Table("region_counts", default=int)
region_revenue = app.Table("region_revenue", default=float)

@app.agent(orders_topic)
async def process_order(orders):
    async for order in orders:
        if order.status == "completed":
            total = order.subtotal + order.tax + order.shipping
            region_counts[order.region] += 1
            region_revenue[order.region] += total

# Every 5 minutes, emit the accumulated window and reset it
@app.timer(interval=300.0)
async def emit_windowed_metrics():
    for region in list(region_counts.keys()):
        await metrics_topic.send(value=OrderMetric(
            region=region,
            order_count=region_counts[region],
            total_revenue=region_revenue[region],
        ))
        region_counts[region] = 0
        region_revenue[region] = 0.0

if __name__ == "__main__":
    app.main()

A caveat: faust-streaming is the maintained fork of the abandoned faust package, but its own release cadence is slow. For new projects, evaluate Bytewax, Quix Streams or plain kafka-python consumers before committing to it.

Lambda Architecture

Lambda runs a batch layer for accurate history and a speed layer for real-time approximations, merging both at query time. It was designed when stream processing couldn’t guarantee correctness — today that guarantee exists, which is why most new systems skip it.

• Pros: both historical accuracy and real-time latency; tolerant of late or replayed data.
• Cons: every business rule exists twice, in two frameworks — the classic Lambda failure mode is the two layers drifting apart and the serving layer quietly returning different answers.

Kappa Architecture

Kappa is Lambda with the batch layer deleted: one streaming codebase, and “batch” is replaying the log from an earlier offset. Backfilling is a reprocessing job, not a separate pipeline.

• Pros: one codebase, one processing model; replay gives you backfilling for free.
• Cons: requires a replay-capable log, so long reprocessing windows mean high Kafka storage; if your “batch” job is a 6-hour Spark join, replaying it through a stream processor hurts.

ETL vs ELT

This is a separate axis from batch vs streaming: it’s about where the T happens.

	ETL	ELT
Order	Extract → Transform → Load	Extract → Load → Transform
Where T runs	Outside the warehouse (Spark, Python)	Inside the warehouse (dbt, SQL)
Warehouse stores	Only clean, modeled data	Raw + transformed (staging → marts)
Choose when	Warehouse compute is limited; PII must be masked before landing	Warehouse is powerful (Snowflake, BigQuery); you want iterative dbt development
Typical stack	Airflow + Spark, Informatica, Talend	Fivetran + dbt, Snowpipe + dbt

The ETL pipeline guide covers incremental loads, SCD handling and idempotent patterns in detail.

ELT with dbt

In dbt each model is its own .sql file — the two below are shown together for readability.

-- models/staging/stg_orders.sql — Staging model (raw → clean)
SELECT
    order_id,
    customer_id,
    CAST(created_at AS TIMESTAMP) AS created_at,
    CAST(subtotal AS DECIMAL(10,2)) AS subtotal,
    CAST(tax AS DECIMAL(10,2)) AS tax,
    CAST(shipping AS DECIMAL(10,2)) AS shipping,
    status,
    region
FROM {{ source('raw', 'orders') }}
WHERE status IS NOT NULL

-- models/marts/fct_daily_revenue.sql — Mart model (clean → business)
SELECT
    DATE(created_at) AS order_date,
    region,
    COUNT(*) AS order_count,
    SUM(subtotal + tax + shipping) AS total_revenue,
    AVG(subtotal + tax + shipping) AS avg_order_value
FROM {{ ref('stg_orders') }}
WHERE status = 'completed'
GROUP BY 1, 2

Choosing the Right Pattern

Requirement	Pattern
Daily/weekly reporting, large history, heavy joins	Batch
Freshness measured in hours; cost-sensitive	Batch
Alerts and dashboards consumed in seconds	Streaming
Event-driven systems reacting to each event	Streaming
Exact history and real-time; audit/regulatory needs	Lambda
One codebase; log retention covers your backfill window	Kappa
Mask PII before data lands in the warehouse	ETL
Powerful warehouse; SQL-native team; iterative modeling	ELT

And when not to reach for the fancy option: if nobody queries the data faster than daily, streaming is pure overhead — operational complexity, on-call burden and infrastructure cost in exchange for freshness nobody consumes. The same logic applies to Lambda: maintaining two codebases only makes sense when both views are actually queried.

For a worked implementation of the Lambda pattern — Kafka Connect ingestion, Trino merge queries, deduplication — see the batch-to-streaming bridge pattern.

Putting It Together: A Real Architecture

In practice these patterns compose rather than compete. A typical e-commerce data platform looks like this:

1. Orders service writes to Postgres. A CDC connector (Debezium) streams changes into Kafka — this is your raw event feed.
2. Streaming branch: a Flink job reads the orders topic and computes fraud signals and live inventory counts within seconds. Only the consumers that genuinely need sub-minute latency live here.
3. Batch branch: the same Kafka topic lands hourly into S3 as Parquet. A nightly Airflow DAG (the one above) runs dbt models that produce fct_daily_revenue and the rest of the warehouse marts — the ELT half of the picture.
4. Serving: dashboards read the warehouse; the fraud API reads the streaming sink. Nobody keeps two versions of the revenue logic around, because there’s no speed-layer duplicate of it — that would be Lambda, and it’s the part you deliberately avoid.

The decision flow is: identify each consumer’s freshness requirement, stream only the branches that need it, and keep everything else on the boring batch path. When a consumer later needs faster data, you already have the replayable log — that’s the Kappa-style backfill escape hatch.

Best Practices

• Start with batch and add streaming only when a real consumer needs the latency — batch is simpler and cheaper.
• Default to ELT when your warehouse can handle it; dbt + Snowflake/BigQuery is where most teams land today.
• Retain your Kafka logs long enough to replay — that retention window is what makes Kappa and backfilling possible.
• Make every pipeline idempotent: re-running must produce the same result, via merge/upsert and date partitioning.
• Monitor data quality, not just job status — row counts, null rates, schema drift and freshness. See the data quality guide.
• Register schemas for every Kafka topic so producers and consumers can’t silently drift.
• Checkpoint streaming jobs — Flink/Spark checkpoints are what make exactly-once real.
• Keep zones separated: raw → staging → marts.
• Alert on pipeline failures; an Airflow retry quietly hiding a broken source is how bad data ships.
• Document lineage with dbt docs or OpenLineage before someone asks where a number came from.

Common Mistakes

• Starting with streaming: it’s the most expensive way to learn that batch was enough. Earn the complexity.
• No idempotency: re-running a pipeline appends duplicate data. Use merge/upsert patterns.
• No schema management: producers and consumers drift apart. Use a schema registry.
• Keeping Lambda too long: the two codebases diverge over time and the speed layer becomes the wrong-but-fast answer. If you already have it, plan the Kappa migration; if you don’t, don’t start one.
• No data quality checks: pipelines silently produce wrong data. Add Great Expectations or dbt tests.

Further Reading

• Apache Airflow documentation — DAGs, operators and scheduling
• Apache Flink documentation — KafkaSource/KafkaSink, watermarks, checkpoints
• dbt documentation — sources, refs and project structure
• Questioning the Lambda Architecture — Jay Kreps’ original argument for Kappa
• Apache Kafka documentation — log retention and replay semantics