Tag: data
Browse 35 practical software engineering resources tagged with "data". Discover code recipes, design patterns, documentation templates, and in-depth guides to help you build, deploy, and maintain production-ready solutions involving data.
Batch Processing Patterns
Design reliable batch processing pipelines for large datasets with retry logic, idempotency, and observability.
Caching & Memoization
How to cache expensive computations and API responses using in-memory, LRU, and distributed caches across Python, JavaScript, and Java.
Validate and Sanitize User Input Data
How to validate, sanitize, and constrain user input data at the application boundary using schemas, type checking, and validation libraries.
Date Formatting
How to parse, format, and manipulate dates across timezones using Python, JavaScript, and Java.
Deep Clone Objects in JavaScript
How to create deep copies of JavaScript objects and arrays correctly, handling circular references, Dates, Maps, Sets, and custom classes.
Deep Clone Objects in JavaScript: Beyond JSON.parse
Compare deep clone strategies including JSON.parse, structuredClone, manual recursion, and library approaches for copying nested objects with circular references and special types
Flatten and Unflatten Nested Objects
How to convert nested objects to flat key-value pairs and back again, with dot-notation, bracket notation, and custom separator support.
Money and Currency Handling
How to represent, parse, format, and calculate monetary values accurately across currencies.
Parse JSON
How to parse JSON strings into native data structures across multiple programming languages.
Schedule and Monitor DAGs with Apache Airflow
How to define, schedule, and monitor Directed Acyclic Graphs in Apache Airflow with operators, sensors, XCom, and task dependencies.
Parallel DataFrame Operations with Dask
How to use Dask for parallel DataFrame operations on datasets larger than memory, covering lazy evaluation, partitions, custom computations, and distributed scheduling.
Validate DataFrame Schemas with Pandera
How to validate pandas and Polars DataFrame schemas with Pandera, covering column types, constraints, custom checks, hypothesis testing, and schema inheritance.
Transform Data in the Warehouse with dbt
How to use dbt for SQL-based data transformations with models, tests, materializations, macros, and incremental loading in a data warehouse.
Build an ETL Pipeline with pandas and Parquet
How to build an extract-transform-load pipeline using pandas for data processing and Parquet for columnar storage with type coercion and validation.
High-Performance DataFrame Operations with Polars
How to use Polars for fast DataFrame operations with lazy evaluation, expression API, streaming, and interop with pandas for large datasets.
Large-Scale Aggregation with PySpark
How to perform group-by aggregations on large datasets with PySpark, covering window functions, UDFs, broadcast joins, and performance tuning.
Regular Expressions
How to use regular expressions for pattern matching, validation, and text extraction across Python, JavaScript, and Java.
Sort an Array
How to sort arrays and lists in ascending, descending, and custom order across multiple languages.
Recursive CTEs for Hierarchical Data Queries
How to query hierarchical data with recursive Common Table Expressions in SQL, covering tree traversal, org charts, category trees, and cycle detection.
URL Encoding
How to encode and decode URLs, query parameters, and path segments safely across Python, JavaScript, and Java.
URL Encoding and Decoding
Master URL encoding in JavaScript and other languages with encodeURI, encodeURIComponent, plus-safe handling, RFC 3986 compliance, and decoding edge cases
UUID Generation
How to generate universally unique identifiers (UUIDs) for database keys, session tokens, and resource naming across Python, JavaScript, and Java.
UUID Generation: v4, v7, and ULID Comparison
Compare UUID v4, v7, ULID, and nanoid for generating unique identifiers with different tradeoffs in randomness, sortability, performance, and database index locality
Batch-to-Streaming Bridge
How to bridge batch and streaming pipelines with a data lake. Covers Lambda architecture, Kafka Connect S3 sink, schema alignment, and unified serving layer.
CDC Pattern: Stream Database Changes to Downstream
How to stream database changes to downstream consumers with CDC. Covers log-based CDC, Debezium, Kafka Connect, outbox pattern, and consumer reconciliation.
Data Lineage Tracking: Track Origin End-to-End
How to track data origin and transformations end-to-end. Covers column-level lineage, OpenLineage, Marquez, metadata injection, and impact analysis.
ETL Extract-Transform-Load
How to build ETL pipelines with extract, transform, and load stages. Covers staging tables, incremental extraction, idempotent loads, and orchestration.
Idempotent Load: Re-run Data Loads Safely Without Duplicates
How to re-run data loads safely without duplicates. Covers deduplication keys, MERGE upserts, load IDs, partition overwrite, and transactional loads.
Schema Registry Evolution
How to manage schema versions for streaming pipelines with a schema registry. Covers Avro, backward compatibility, forward compatibility, and consumer migration.
Data Governance Policy Template
A template for data classification, retention, access control, privacy, and compliance policies covering GDPR, CCPA, and SOC 2 requirements.
Data Pipeline Design Document Template
A template for documenting data pipeline sources, transformations, sinks, scheduling, error handling, and monitoring with schema definitions.
Data Quality Rules Template
A template for defining data validation rules per dataset and column: completeness, consistency, accuracy, timeliness, and uniqueness checks.
ETL Job Runbook Template
A runbook for operating, monitoring, and troubleshooting ETL jobs: startup, shutdown, health checks, common failures, diagnostics, and recovery.
Data Classification Template
A template for classifying data as public, internal, confidential, or restricted with handling rules.