I designed and implemented this data pipeline as part of graduate-level data engineering study. It transforms raw electrocardiogram (ECG) recordings into structured datasets for machine learning and calculates heart rate variability (HRV) features that characterize variation in the time between consecutive heartbeats. The pipeline also records the source of each dataset and the transformations applied to it.
The pipeline processes publicly available MIT-BIH Arrhythmia Database research data and does not process personally identifiable information.
The engineering problem
Machine-learning workflows depend on pipelines that turn raw recordings into structured, model-ready features. ECG waveforms require several processing steps before they are useful for modeling: signal processing, feature extraction, aggregation, and representation for downstream learning tasks.
The central question for this project was therefore:
How can raw ECG recordings be transformed into ML-ready HRV features through a reproducible batch pipeline that remains consistent across reruns and partial-output scenarios?
In this context, consistency describes how the pipeline handles its data. This includes stable artifact identities, idempotent writes, and fail-fast handling of incomplete outputs. Running the same stage again should produce the same set of outputs, while an incomplete set should cause the pipeline to stop. These concerns are technical and architectural: they describe data-processing behavior, not medical or clinical safety.
From raw ECG to ML-ready features
The system follows a staged batch architecture with clear transformation boundaries:
raw → processed → curated → ml_ready
Each stage represents a distinct step in the data lifecycle.
The implemented pipeline converts:
raw ECG → RR intervals → five-minute HRV windows → ML-ready feature datasets
raw— original ECG waveform Parquetprocessed— canonicalrr_intervals_v1curated— five-minutewindow_features_v1ml_ready—window_features_ml_v1andrecord_features_v1
window_features_ml_v1 is the primary representation because it preserves HRV dynamics across five-minute temporal windows; each row is identified by (run_id, record_id, window_start_sec). record_features_v1 provides a secondary record-level baseline that summarizes HRV statistics across an ECG record, with row grain (run_id, record_id).
Three separately runnable containerized services implement those stages: ingestion, processing, and aggregation. A shell orchestrator runs them in sequence, while the services exchange artifacts and state through MinIO object storage and PostgreSQL metadata rather than calling one another directly.
Separating pipeline responsibilities
The pipeline is implemented as a containerized microservice architecture with explicit stage ownership.
Ingestion reads WFDB ECG inputs (or synthetic recordings for local testing), writes raw artifacts such as raw/.../ecg.parquet on deterministic paths derived from run date, run ID, and record ID, and registers metadata in PostgreSQL. It uses storage-first idempotency, isolates failures per record, and reports succeeded, partial-success, or failed run states.
Processing discovers registered raw ECG artifacts through metadata and reads the ECG signal from the field named lead_0. It uses NeuroKit2 with its pantompkins1985 method to detect R peaks and derive RR intervals. It then writes the canonical processed/rr_intervals_v1 artifact, stores processing metrics, and uses deterministic paths with per-record isolation and service lifecycle tracking.
Aggregation uses PySpark to read registered rr_intervals_v1 artifacts, compute fixed five-minute HRV windows as curated/window_features_v1, and derive the ML-ready datasets window_features_ml_v1 and record_features_v1. It preserves explicit row grain and schema versions, registers outputs in metadata, and applies deterministic feature rules with fail-fast invariants.
Execution is coordinated by a fixed-order shell orchestrator that runs ingestion, processing, and aggregation sequentially. In production settings, scheduling is handled externally (for example cron or workflow orchestrators).
Reliability under reruns and partial outputs
A major challenge was keeping aggregation correct under reruns and partial-output scenarios. Implementing that behaviour required careful handling of artifact-existence checks, metadata consistency, and fail-fast validation—and iterative refinement as edge cases appeared.
A batch stage may run more than once. Blindly appending results can create duplicate HRV records or leave the lake in an inconsistent state. Deterministic artifact identities and idempotent write or replacement behaviour were used so that reruns converge toward the same state rather than accumulating conflicting outputs.
Object storage is the source of truth for whether an artifact already exists. Metadata in PostgreSQL reflects that state; it does not redefine it.
Aggregation makes the requirement especially visible because it writes two ML-ready outputs. When AGG_OVERWRITE=false:
- if both outputs already exist, the stage skips;
- if neither exists, the stage executes;
- if exactly one exists, the stage fails fast as a partial state;
- recovery requires an explicit overwrite run.
A silent half-write is worse than a clear failure: it can make the data lake appear complete even though its artifacts and downstream inputs are inconsistent.
Reliability more broadly is supported by validation rules, controlled artifact generation, deterministic orchestration, and this strict aggregation idempotency.
Additional implementation challenges included integrating PySpark-based aggregation within Docker and debugging interactions between object-storage artifacts and PostgreSQL metadata registration. Those issues were addressed through iterative refinement, contract-based validation checks, and staged pipeline testing.
Scalability and maintainability
Scalability is addressed primarily in feature aggregation. Aggregation is implemented with PySpark and could move to distributed Spark execution if deployed on a cluster. Versioned, run-scoped Parquet artifacts in object storage also support larger data volumes.
Maintainability is supported by explicit service responsibilities, staged data modeling, canonical data contracts, schema versions and invariants, structured logging and lifecycle state, and a reproducible local Docker Compose deployment. Together, these decisions make the pipeline easier to debug, test, and extend.
Governance, metadata, and lineage
Governance and traceability are implemented through PostgreSQL metadata tables that record pipeline runs, service execution states, and artifact registrations. Artifact records capture the data stage (layer), artifact type, schema version, and URI referencing the stored Parquet object in MinIO.
The primary window-level dataset identifies each row by run_id, record_id, and window_start_sec; the record-level baseline uses run_id and record_id. PostgreSQL separately registers each staged artifact and its originating run, making the transformation path traceable from ML-ready outputs back to the corresponding input records.
Services run in isolated Docker containers and communicate with MinIO and PostgreSQL over the Docker network. Runtime configuration and credentials are provided through environment variables rather than embedded in code.
The project does not claim formal security certification, production-grade secret management, regulatory compliance, or clinical validation.
Reproducible local orchestration
Docker and Docker Compose define a reproducible local environment for the pipeline services, MinIO storage, and PostgreSQL metadata. The shell orchestrator (run_orchestrator.sh) runs ingestion, processing, and aggregation in a deterministic sequence using a shared RUN_ID and RUN_DATE.
Each component has a distinct role:
- Docker Compose defines and runs the local containers.
- The shell orchestrator executes the batch stages in sequence.
- External scheduling can trigger the pipeline through cron or a workflow orchestrator. The repository shows how this can be configured, but no production scheduler is deployed as part of the project.
- Cluster-level container orchestration, such as Kubernetes, is not part of the current implementation; it is documented only as a possible direction for production deployment.
Technologies evidenced by the repository: Python · Apache Spark / PySpark · Docker · Docker Compose · MinIO · PostgreSQL · Parquet · NeuroKit2 · WFDB · shell scripting · Git / GitHub · MIT-BIH Arrhythmia Database / PhysioNet
Reflection
I designed and implemented this project independently, which required end-to-end ownership from ingestion through ML-ready feature generation.
The principal challenge was aggregation behaviour that remained correct under reruns and partial-output scenarios: artifact-existence checks, metadata consistency, and fail-fast validation around dual ML-ready outputs. Integrating PySpark within Docker, and debugging interactions between object-storage artifacts and metadata registration, demanded the same iterative refinement, contract-based checks, and staged testing.
Working through these issues showed me why explicit data contracts and schema checks matter: they made failures easier to locate and reruns easier to reason about. The project also gave me practical experience defining pipeline boundaries, building HRV features with Spark, tracing artifacts through metadata, and debugging a multi-container workflow. I also became more deliberate about planning changes and documenting the system as it evolved.
The project demonstrates a reproducible local architecture for turning raw ECG recordings into versioned, lineage-tracked HRV feature datasets. Its limitations are clear: it has not been deployed or operated as a production system, and it has not undergone clinical validation.
Future work: a possible streaming path
The current pipeline processes data in batches. It could be extended with event-driven ingestion using Apache Kafka and streaming HRV computation over sliding or tumbling windows, while keeping the resulting schemas compatible with the existing batch artifacts.
In a hybrid architecture, batch processing would remain responsible for historical recomputation and model retraining, while streaming components could provide near-real-time feature updates. Shared data contracts across both paths would help maintain semantic consistency.
Apache Kafka and streaming processing are conceptual future work. They are not technologies implemented or used in this project.
Running the current batch pipeline in production would require additional work on deployment, secrets management, monitoring, and alerting. Depending on the workload, the infrastructure could later expand to managed object storage, distributed Spark, container orchestration, or a production workflow scheduler. These extensions are not part of the current repository.