Tag: data-engineering
Browse 8 practical software engineering resources tagged with "data-engineering". Discover code recipes, design patterns, documentation templates, and in-depth guides to help you build, deploy, and maintain production-ready solutions involving data-engineering.
Data Governance Policy Template
A template for data classification, retention, access control, privacy, and compliance policies covering GDPR, CCPA, and SOC 2 requirements.
Data Pipeline Design Document Template
A template for documenting data pipeline sources, transformations, sinks, scheduling, error handling, and monitoring with schema definitions.
Data Quality Rules Template
A template for defining data validation rules per dataset and column: completeness, consistency, accuracy, timeliness, and uniqueness checks.
ETL Job Runbook Template
A runbook for operating, monitoring, and troubleshooting ETL jobs: startup, shutdown, health checks, common failures, diagnostics, and recovery.
Apache Airflow: DAGs, Operators, Scheduling
Master Apache Airflow: DAGs, operators, sensors, XCom, scheduling, backfilling, connections, variables, and production patterns for data pipeline orchestration.
Data Pipeline Architecture: Batch, Streaming, Lambda, Kappa
Master data pipeline architecture: batch processing, streaming, lambda and kappa patterns, ETL vs ELT, and choosing the right approach for your data workloads.
Data Quality Guide: Validation, Profiling, Great
Master data quality: validation frameworks, profiling, schema enforcement, anomaly detection, and monitoring with Great Expectations, Pandera, and Soda for reliable pipelines.
dbt: Models, Tests, Macros, Materializations
Master dbt for data transformations: models, tests, macros, materializations, seeds, snapshots, Jinja templating, and production patterns for analytics engineering.