Skip to content

Data Engineering

Data engineering builds reliable systems that move, transform, store, and serve data for decisions and products. The core concerns are meaning, correctness, lineage, recovery, latency, privacy, and cost—not a catalog of processing tools.

Begin with the Data Product

Define:

  • consumers and decisions or features supported;
  • entities, grain, identifiers, and business time semantics;
  • freshness, completeness, accuracy, and availability objectives;
  • history, correction, retention, and deletion rules;
  • privacy classification and access;
  • expected volume, growth, query patterns, and cost;
  • owner and incident response.

A table without a documented grain is a future correctness bug.

Pipeline Shape

sources → ingestion → durable raw boundary → transform → validated models → consumers

Not every system needs every stage. Preserve a replayable boundary when source recovery is limited or transformations will evolve.

ETL transforms before loading to the target; ELT loads then transforms inside the analytical platform. Choose according to trust, privacy, compute location, latency, and recoverability. The acronym does not determine architecture quality.

Batch and Streaming

Batch processes bounded datasets on a schedule or trigger. Streaming processes an unbounded sequence incrementally. Near-real-time requirements should name a freshness objective; do not adopt stream infrastructure for vague speed.

Streaming still uses bounded windows, checkpoints, state, and replay. A practical design often combines continuous ingestion with periodic reconciliation from an authoritative source.

Event Time

Distinguish:

  • event time: when the business event occurred;
  • ingestion time: when the platform received it;
  • processing time: when a computation handled it.

Events arrive late and out of order. Windowed computations need a policy for lateness, watermarks, corrections, and finality. “Daily revenue” is ambiguous without timezone, event-time rule, refunds, and late-arrival handling.

Idempotency and Replay

Assume retries and partial completion. Give source records stable identifiers, make writes idempotent or transactional, and record checkpoints only after durable effects.

For each pipeline define:

  • replay source and retention;
  • deduplication identity and duration;
  • destination overwrite, merge, or append semantics;
  • safe backfill scope and rate;
  • separation of live and backfill progress;
  • reconciliation against source totals or invariants.

Exactly-once labels apply within specific engine boundaries; external side effects still need idempotency.

Storage Layers

  • operational databases serve application transactions;
  • object storage provides inexpensive durable files;
  • warehouses optimize governed analytical queries;
  • lakehouses add table metadata and transactional management over object storage;
  • serving stores provide low-latency projections for applications or models.

Avoid copying data unless the new representation has an owner, retention rule, and consumer value.

File and Table Formats

Columnar formats such as Apache Parquet group data for selective analytical reads and compression. Performance depends on row-group size, file size, sorting, statistics, compression, and query engine support—not the extension alone.

Avoid many tiny files: metadata and scheduling overhead can dominate. Compact deliberately without blocking ingestion, and make compaction idempotent.

Table formats add snapshots, schema evolution, partition metadata, and atomic commits over files. They do not fix poor data modeling or governance.

Partitioning

Partition by common selective predicates with bounded cardinality, often date plus a stable domain dimension. Over-partitioning creates tiny files and metadata pressure; under-partitioning scans unnecessary data.

Do not encode sensitive or mutable values casually in paths. Measure pruning from actual queries and plan repartitioning as access patterns evolve.

Analytical Modeling

Define facts at one grain and dimensions that describe them. Star schemas remain useful when stable business measures need consistent, understandable queries.

Slowly changing dimensions require an explicit history policy:

  • overwrite when only current truth matters;
  • version rows with valid-time bounds when historical context matters;
  • store selected previous values when only limited change history is needed.

Use effective-time joins carefully and define overlapping or missing intervals.

Contracts and Schema Evolution

A data contract should identify owner, schema, field meaning, nullability, units, classification, freshness, compatibility, and change process.

Prefer additive changes, tolerant readers, and staged migration. Renaming is often add–backfill–migrate–remove. Structural compatibility does not guarantee semantic compatibility: changing amount from gross to net breaks consumers without changing its type.

Data Quality

Validate where defects can first be detected and where impact matters:

  • schema and type;
  • uniqueness and referential integrity;
  • accepted ranges and categories;
  • completeness and freshness;
  • reconciliation totals;
  • distribution shifts;
  • business invariants.

Define whether failure should block, quarantine, continue with a warning, or degrade a downstream product. Every alert needs an owner and evidence to locate the bad partition or source.

Orchestration

A workflow scheduler coordinates dependencies, retries, parameters, timeouts, backfills, and status. Tasks should be idempotent, bounded, independently observable, and explicit about inputs and outputs.

Avoid encoding business transformations in scheduler callbacks. Keep transformations runnable and testable outside orchestration. A successful task means its declared output is valid, not merely that a process exited zero.

Governance and Privacy

  • classify data at ingestion and propagate classification;
  • grant access by purpose and least privilege;
  • record lineage from source to consumer;
  • define retention and deletion across replicas and derived datasets;
  • protect logs, samples, test data, and query results;
  • audit sensitive access and exports;
  • document authoritative sources and owners.

Masking identifiers does not necessarily anonymize linked datasets. Minimize collection and derived copies.

Observability and Cost

Measure input/output records and bytes, freshness, late data, duplicates, rejected records, checkpoint age, task duration, retries, queue lag, file sizes, query scan, and cost per useful dataset or consumer.

Infrastructure health can be green while data is stale or wrong. Monitor data-product objectives directly.

Checklist

  • Are grain, meaning, owner, and source of truth explicit?
  • Can the pipeline replay and reconcile safely?
  • Are time, lateness, deduplication, and correction semantics defined?
  • Do partitioning and file size match real queries?
  • Are schema and semantic changes compatible and measured?
  • Do quality failures have deliberate behavior and ownership?
  • Can sensitive data be found, restricted, retained, and deleted downstream?
  • Is cost attributable to a useful data product?