Tag: data-engineering
Browse 8 practical software engineering resources tagged with "data-engineering". Discover code recipes, design patterns, documentation templates, and in-depth guides to help you build, deploy, and maintain production-ready solutions involving data-engineering. Each resource is written for engineers who ship real systems, with copy-paste examples and practical trade-offs.
Data Engineering
Data engineering builds the pipelines and infrastructure that transform raw data into usable products. It requires skills in storage, processing, orchestration, and quality.
The resources below cover ETL, data lakes, pipelines, orchestration, data quality, and streaming. Each guide helps you build data systems that are reliable and scalable.
Every resource includes clear explanations, copy-paste code, and practical warnings. Use them to make informed decisions, avoid production pitfalls, and speed up your delivery. If you are just getting started, read the beginner-friendly articles first; if you are experienced, jump straight to the advanced patterns and architecture guides. New resources are added regularly, so bookmark this page and check back for the latest patterns.
Data Governance Policy Template
A template for data classification, retention, access control, privacy, and compliance policies...
Data Pipeline Design Document Template
A template for documenting data pipeline sources, transformations, sinks, scheduling, error...
Data Quality Rules Template
A template for defining data validation rules per dataset and column: completeness, consistency,...
ETL Job Runbook Template
A runbook for operating, monitoring, and troubleshooting ETL jobs: startup, shutdown, health...
Apache Airflow: DAGs, Operators, Scheduling
Master Apache Airflow: DAGs, operators, sensors, XCom, scheduling, backfilling, connections,...
Data Pipeline Architecture: Batch, Streaming, Lambda, Kappa
Master data pipeline architecture: batch processing, streaming, lambda and kappa patterns, ETL vs...
Data Quality Guide: Validation, Profiling, Great
Master data quality: validation frameworks, profiling, schema enforcement, anomaly detection, and...
dbt: Models, Tests, Macros, Materializations
Master dbt for data transformations: models, tests, macros, materializations, seeds, snapshots,...