data
Practical resources about data for software engineers.
87 results
Understanding Data Processing in Software Engineering
Every application eventually deals with data — parsing user input, serializing objects for APIs, validating form submissions, or transforming records between formats. Getting data handling right is what separates fragile scripts from production-grade systems.
This collection covers practical recipes and patterns for working with JSON, CSV, XML, YAML, and custom formats. You will find copy-paste solutions for common tasks like flattening nested objects, sanitizing user input, generating slugs, and formatting dates across Python, Java, and JavaScript.
Batch Processing Patterns
Design reliable batch processing pipelines for large datasets with retry logic, idempotency, and observability.
Caching & Memoization
How to cache expensive computations and API responses using in-memory, LRU, and distributed caches across Python, JavaScript, and Java.
Convert CSV to JSON
How to convert CSV data to JSON format in Python, Java, and JavaScript.
Convert JSON to CSV
How to convert JSON data to CSV format in Python, Java, and JavaScript.
Validate and Sanitize User Input Data
How to validate, sanitize, and constrain user input data at the application boundary using schemas, type checking, and validation libraries.
Date Formatting
How to parse, format, and manipulate dates across timezones using Python, JavaScript, and Java.
Deep Clone Objects in JavaScript
How to create deep copies of JavaScript objects and arrays correctly, handling circular references, Dates, Maps, Sets, and custom classes.
Deep Clone Objects in JavaScript: Beyond JSON.parse
Compare deep clone strategies including JSON.parse, structuredClone, manual recursion, and library approaches for copying nested objects with circular references and special types
Diff JSON Objects
How to compare two JSON objects and find differences in Python, Java, and JavaScript.
Flatten and Unflatten Nested Objects
How to convert nested objects to flat key-value pairs and back again, with dot-notation, bracket notation, and custom separator support.
Format Phone Numbers
How to format and validate phone numbers in Python, Java, and JavaScript.
Generate PDF Reports with Python
Create styled PDF documents from data using ReportLab and fpdf2 in Python.
Generate URL Slugs
How to generate clean, URL-friendly slugs from strings in multiple programming languages.
Merge JSON Files
How to merge multiple JSON files into a single object or array in Python, Java, and JavaScript.
Merge JSON Files in JavaScript
Combine multiple JSON files with conflict resolution strategies using Node.js.
Money and Currency Handling
How to represent, parse, format, and calculate monetary values accurately across currencies.
Parse Command Line Arguments
How to parse command line arguments in Python, Java, and Node.js CLI applications.
Parse CSV Files
How to parse CSV files in Python, Java, and JavaScript with practical code examples.
Parse CSV Files with Python and Pandas
How to read, filter, and transform large CSV files efficiently using Python pandas and the csv module.
Parse Excel Files
How to read and write Excel (.xlsx) files in Python, Java, and JavaScript.
Parse JSON
How to parse JSON strings into native data structures across multiple programming languages.
Parse Log Files
How to parse and analyze server log files using Python, Java, and JavaScript.
Parse Markdown Files
How to parse Markdown to HTML and extract structured data in Python, Java, and JavaScript.
Parse PDF Files
How to extract text and metadata from PDF files in Python, Java, and JavaScript.
Parse XML Files
How to parse XML documents in Python, Java, and JavaScript with practical code examples.
Schedule and Monitor DAGs with Apache Airflow
How to define, schedule, and monitor Directed Acyclic Graphs in Apache Airflow with operators, sensors, XCom, and task dependencies.
Parallel DataFrame Operations with Dask
How to use Dask for parallel DataFrame operations on datasets larger than memory, covering lazy evaluation, partitions, custom computations, and distributed scheduling.
Validate DataFrame Schemas with Pandera
How to validate pandas and Polars DataFrame schemas with Pandera, covering column types, constraints, custom checks, hypothesis testing, and schema inheritance.
Transform Data in the Warehouse with dbt
How to use dbt for SQL-based data transformations with models, tests, materializations, macros, and incremental loading in a data warehouse.
Read and Write Excel Files with Python
How to read, write, and format Excel spreadsheets using openpyxl and pandas in Python.
Generate QR Codes with Python
Create QR codes for URLs, text, and contact cards using the qrcode library in Python.
Build an ETL Pipeline with pandas and Parquet
How to build an extract-transform-load pipeline using pandas for data processing and Parquet for columnar storage with type coercion and validation.
High-Performance DataFrame Operations with Polars
How to use Polars for fast DataFrame operations with lazy evaluation, expression API, streaming, and interop with pandas for large datasets.
Large-Scale Aggregation with PySpark
How to perform group-by aggregations on large datasets with PySpark, covering window functions, UDFs, broadcast joins, and performance tuning.
Extract Data from HTML Pages with Python and BeautifulSoup
Parse HTML and extract data using BeautifulSoup. Covers CSS selectors, navigation, tables, pagination, and respectful scraping with rate limiting.
Regular Expressions
How to use regular expressions for pattern matching, validation, and text extraction across Python, JavaScript, and Java.
Serialize and Deserialize Data
How to serialize and deserialize data in JSON, XML, and YAML across Python, Java, and JavaScript.
Sort an Array
How to sort arrays and lists in ascending, descending, and custom order across multiple languages.
Recursive CTEs for Hierarchical Data Queries
How to query hierarchical data with recursive Common Table Expressions in SQL, covering tree traversal, org charts, category trees, and cycle detection.
Truncate Text
How to truncate text with ellipsis and word boundaries in Python, Java, and JavaScript.
URL Encoding
How to encode and decode URLs, query parameters, and path segments safely across Python, JavaScript, and Java.
URL Encoding and Decoding
Master URL encoding in JavaScript and other languages with encodeURI, encodeURIComponent, plus-safe handling, RFC 3986 compliance, and decoding edge cases
UUID Generation
How to generate universally unique identifiers (UUIDs) for database keys, session tokens, and resource naming across Python, JavaScript, and Java.
UUID Generation: v4, v7, and ULID Comparison
Compare UUID v4, v7, ULID, and nanoid for generating unique identifiers with different tradeoffs in randomness, sortability, performance, and database index locality
Validate JSON Schema
How to validate JSON data against schemas in Python, Java, and JavaScript.
Implement ACID Transactions in PostgreSQL
How to use PostgreSQL transactions to ensure Atomicity, Consistency, Isolation, and Durability for reliable multi-step database operations
Elasticsearch Aggregations for Analytics and Search
How to use Elasticsearch aggregations to build faceted search, analytics dashboards, and real-time metrics from indexed data
Build Responsive Email Templates with MJML
Create cross-client responsive email templates using MJML markup, live Handlebars variables, and inline CSS for reliable rendering across Gmail, Outlook, and Apple Mail
RAG Hybrid Search Pattern
Combine keyword (BM25) and semantic (vector) search to improve retrieval accuracy in RAG pipelines. Fuse ranked results using reciprocal rank fusion.
Batch-to-Streaming Bridge
How to bridge batch and streaming pipelines with a data lake. Covers Lambda architecture, Kafka Connect S3 sink, schema alignment, and unified serving layer.
CDC Pattern: Stream Database Changes to Downstream
How to stream database changes to downstream consumers with CDC. Covers log-based CDC, Debezium, Kafka Connect, outbox pattern, and consumer reconciliation.
Data Lineage Tracking: Track Origin End-to-End
How to track data origin and transformations end-to-end. Covers column-level lineage, OpenLineage, Marquez, metadata injection, and impact analysis.
ETL Extract-Transform-Load
How to build ETL pipelines with extract, transform, and load stages. Covers staging tables, incremental extraction, idempotent loads, and orchestration.
Idempotent Load: Re-run Data Loads Safely Without Duplicates
How to re-run data loads safely without duplicates. Covers deduplication keys, MERGE upserts, load IDs, partition overwrite, and transactional loads.
Schema Registry Evolution
How to manage schema versions for streaming pipelines with a schema registry. Covers Avro, backward compatibility, forward compatibility, and consumer migration.
Data Migration Runbook Template
A runbook template for safely migrating data between systems including pre-migration checks, rollback procedures, and post-migration validation.
Complete Guide to Kafka Stream Processing
Build real-time event streaming pipelines with Kafka. Covers producers, consumers, Kafka Streams, Kafka Connect, schema registry, and stream processing patterns.
Data Lake vs Data Warehouse — Architecture Guide
A practical guide to Data Lake architecture: structured vs unstructured storage, lakehouse concepts, ETL vs ELT patterns, and when to choose a lake over a warehouse.
Data Mesh Architecture — Decentralized Data Ownership
A practical guide to Data Mesh: decentralizing data ownership to domain teams, treating data as a product, and enabling self-serve data infrastructure.
Lakehouse Architecture — The Best of Both Worlds
A practical guide to Lakehouse architecture: combining data lake storage flexibility with data warehouse reliability using open table formats like Delta Lake, Apache Iceberg, and Hudi.
Blob Storage: S3, GCS, and Azure Blob Patterns for Engineers
A practical guide to cloud blob storage: bucket design, access control, lifecycle policies, multipart uploads, presigned URLs, and cost optimization patterns for S3, Google Cloud Storage, and Azure Blob.
Caching Strategies: From Browser to Database, a Complete
A practical guide to caching strategies: browser caching, CDN edge caching, application caching with Redis, and database query caching. Learn when to use each and how to avoid cache invalidation nightmares.
Apache Airflow: DAGs, Operators, Scheduling
Master Apache Airflow: DAGs, operators, sensors, XCom, scheduling, backfilling, connections, variables, and production patterns for data pipeline orchestration.
Data Pipeline Architecture: Batch, Streaming, Lambda, Kappa
Master data pipeline architecture: batch processing, streaming, lambda and kappa patterns, ETL vs ELT, and choosing the right approach for your data workloads.
Data Quality Guide: Validation, Profiling, Great
Master data quality: validation frameworks, profiling, schema enforcement, anomaly detection, and monitoring with Great Expectations, Pandera, and Soda for reliable pipelines.
dbt: Models, Tests, Macros, Materializations
Master dbt for data transformations: models, tests, macros, materializations, seeds, snapshots, Jinja templating, and production patterns for analytics engineering.
Data Migration: Zero-Downtime Strategies That Work
A practical guide to data migration: planning, dual-write patterns, backfill strategies, schema evolution, validation, and rollback procedures for moving data without service interruption.
ETL Pipelines: Extract, Transform, Load for Data Engineers
A practical guide to ETL pipelines: extracting data from multiple sources, transforming with validation and business logic, and loading into data warehouses. Covers batch scheduling, error handling, and monitoring with Python, dbt, and Airflow.
Full-Text Search — Implement Search That Actually Works
A practical guide to full-text search: PostgreSQL tsvector, Elasticsearch indexing, query design, relevance tuning, and building search that users trust with autocomplete, faceting, and typo tolerance.
Real-Time Analytics: From Events to Dashboards in Seconds
A practical guide to real-time analytics: event collection, stream processing, data warehousing, and building sub-second dashboards with Kafka, ClickHouse, Druid, and modern OLAP databases.
Stream Processing: Event-Driven Data Pipelines with
A practical guide to stream processing: choosing between Kafka Streams, Flink, and Spark Streaming, designing event schemas, handling stateful operations, and building exactly-once processing pipelines for real-time data.
Complete Guide to Elasticsearch Cluster Setup
Deploy and scale Elasticsearch clusters. Covers node roles, sharding, replicas, index templates, mapping, snapshots, and production tuning for search at scale.
Complete Guide to SQL Query Optimization
Optimize SQL queries. Covers EXPLAIN plan analysis, index strategies, join optimization, N+1 query detection, query rewriting, materialized views, partitioning, connection pooling, and query caching with practical PostgreSQL and MySQL examples.
Graph Databases — Neo4j and Property Graph Modeling
A practical guide to graph databases: property graph model, Cypher query language, modeling patterns, and when to choose Neo4j over relational databases.
NoSQL Data Modeling Patterns
A practical guide to NoSQL data modeling: embedding vs referencing, access pattern-driven design, and patterns for MongoDB, DynamoDB, Cassandra, and Redis.
SQL CTEs — Common Table Expressions Explained
A practical guide to SQL Common Table Expressions (CTEs): non-recursive and recursive CTEs, readability, performance, and when to use them over subqueries.
SQL Joins — Visual Guide with Examples
A visual guide to SQL joins: INNER, LEFT, RIGHT, FULL OUTER, CROSS, and SELF joins with practical examples, performance tips, and common pitfalls.
SQL Window Functions — Complete Guide
A practical guide to SQL window functions: ROW_NUMBER, RANK, DENSE_RANK, LEAD, LAG, SUM, AVG over partitions, and real-world analytics use cases.
Vector Databases — AI/ML Embeddings and Similarity Search
A practical guide to vector databases: embeddings, similarity search, approximate nearest neighbors, and choosing between Pinecone, Weaviate, pgvector, and Chroma.
A/B Testing: Experimentation Frameworks for Data-Driven
A practical guide to A/B testing: experiment design, statistical significance, sample sizing, avoiding pitfalls, and building an experimentation culture in engineering teams.
AWS Basics — Core Services for Developers
A practical guide to AWS core services for developers: compute, storage, databases, networking, and security fundamentals with hands-on examples.
Azure Basics — Core Services for Developers
A practical guide to Microsoft Azure core services for developers: compute, storage, databases, networking, and identity with hands-on examples.
GCP Basics: Core Services for Developers
A practical guide to Google Cloud Platform core services for developers: compute, storage, databases, networking, and data analytics with hands-on examples.
Terraform Best Practices — Modules, State, and Workspaces
A practical guide to Terraform best practices: module design, remote state management, workspaces, and security for production-grade infrastructure as code.
GDPR Compliance — A Practical Guide for Developers
A developer-focused guide to GDPR compliance: data subject rights, lawful basis, data minimization, and technical measures for privacy by design.
No results found.