Tag: pipeline
Browse 14 practical software engineering resources tagged with "pipeline". Discover code recipes, design patterns, documentation templates, and in-depth guides to help you build, deploy, and maintain production-ready solutions involving pipeline. Each resource is written for engineers who ship real systems, with copy-paste examples and practical trade-offs.
CI/CD and Data Pipelines
Pipelines automate sequences of tasks that move code or data from source to destination. A reliable pipeline is versioned, testable, observable, and easy to debug when a stage fails.
The recipes below cover CI/CD pipelines, ETL and data pipelines, DAGs, scheduling, artifact management, and pipeline testing. Each guide focuses on building automation that is repeatable and safe to change.
Every resource includes clear explanations, copy-paste code, and practical warnings. Use them to make informed decisions, avoid production pitfalls, and speed up your delivery. If you are just getting started, read the beginner-friendly articles first; if you are experienced, jump straight to the advanced patterns and architecture guides. New resources are added regularly, so bookmark this page and check back for the latest patterns.
Prompt Chaining Pattern
Chain multiple LLM calls where each step's output feeds the next step's input. Break complex tasks...
Pipes and Filters Pattern: Composable Data Pipelines
Chain processing steps with independent filters connected by pipes. A pattern for data...
ETL Extract-Transform-Load
How to build ETL pipelines with extract, transform, and load stages. Covers staging tables,...
Idempotent Load: Re-run Data Loads Safely Without Duplicates
How to re-run data loads safely without duplicates. Covers deduplication keys, MERGE upserts, load...
Chain of Responsibility Pattern
Pass requests along a chain of handlers until one handles it. A behavioral design pattern for...
Intercepting Filter Pattern
Compose cross-cutting concerns into a chain of pluggable filters that intercept requests and...
Data Pipeline Design Document Template
A template for documenting data pipeline sources, transformations, sinks, scheduling, error...
CI/CD Pipeline Design Template
A template for documenting CI/CD pipeline stages, gates, environments, deployment strategies,...
CI/CD Pipeline Security Template
A template for securing build and deployment pipelines against credential leaks, tampering, supply...
Data Pipeline Architecture: Batch, Streaming, Lambda, Kappa
Master data pipeline architecture: batch processing, streaming, lambda and kappa patterns, ETL vs...
ETL Pipelines: Extract, Transform, Load for Data Engineers
A practical guide to ETL pipelines: extracting data from multiple sources, transforming with...
CI/CD Security: Harden Your Pipelines and Prevent Supply
A practical guide to securing CI/CD pipelines: secrets management, least-privilege runners,...
CI/CD Pipeline Guide
A practical guide to building CI/CD pipelines with GitHub Actions, testing, deployment strategies,...
Complete Guide to CI/CD with GitHub Actions
Build CI/CD pipelines from scratch with GitHub Actions. Covers workflows, runners, matrix builds,...