How we designed a scalable, HIPAA-aligned data lake for a county health network processing 4M+ records monthly.
Health data in East Africa arrives in fragments. DHIS2 pushes aggregate reports weekly. ODK forms dump JSON submissions daily. HMIS databases export CSV files on request. EMR systems sit behind clinic LAN networks with no external access. And spreadsheets — always spreadsheets.
When Edos Centre was commissioned to build a county-wide health analytics platform, our first challenge was not dashboards or visualisations. It was answering a deceptively simple question: where does all this data actually live, and how do we build a single trustworthy version of it?
The Starting Architecture
A health data lake is not a database. It is an organised storage layer that accepts data in any format, preserves raw copies indefinitely, and exposes clean, queryable versions to downstream analytics tools. We designed ours with three zones:
Raw zone — HDFS/S3-compatible object storage (we used MinIO on-premises for data sovereignty reasons). Every file that enters the system lands here untouched. A DHIS2 export from 2019, an ODK submission from this morning, a clinic's exported Excel — all preserved as-is with a timestamp and source identifier.
Standardised zone — PostgreSQL tables with a consistent schema. Apache Airflow runs nightly jobs that read from the raw zone, apply cleaning and validation rules, and write to standardised tables. This is where business rules live: reconciling ICD-10 codes, de-duplicating patient records, standardising facility identifiers.
Analytics zone — dbt models that transform standardised data into dimensional models optimised for Power BI and Apache Superset. Star schemas, pre-aggregated indicator tables, and time-spine models for period comparisons.
HIPAA Alignment in an East African Context
Kenya's Health Act and the Data Protection Act 2019 impose data residency and security requirements that overlap significantly with HIPAA. We designed with these in mind:
- All PHI fields encrypted at rest using PostgreSQL's pgcrypto extension
- Row-Level Security (RLS) policies ensuring facility staff can only see their own facility's patient-level records
- Audit logs capturing every SELECT on patient tables, stored in a separate append-only audit schema
- Network-level isolation: analytics queries run against read replicas with no write access
- Automated data retention policies: raw data retained for 10 years per health records legislation
The ETL Patterns That Actually Work
After 3 years of health data integration projects, we have converged on a small set of patterns that work reliably in East Africa's infrastructure environment:
DHIS2 → Data Lake: Use the DHIS2 Analytics API rather than the Data Values API for aggregate data. It pre-applies the organisation unit hierarchy and period logic that would otherwise take hours of custom code. Schedule pulls at 2am to avoid peak DHIS2 load.
ODK → Data Lake: KoboToolbox and ODK Central both expose REST APIs for form submissions. Pull incrementally by filtering on _submission_time > last_run_timestamp. This avoids re-processing the full submission history every night.
Excel → Data Lake: Accept the reality. Many county government systems still produce Excel exports. Build Airflow DAGs that pick up files from a shared folder, validate structure, and ingest. Automate the monitoring rather than trying to eliminate the Excel.
Real-time streams: For facilities with stable internet and critical monitoring needs (ICU metrics, disease surveillance), we use PostgreSQL logical replication into Debezium → Kafka → a streaming consumer that writes to the lake.
What We Would Do Differently
If we were starting this project again, we would design the organisation unit hierarchy upfront and enforce it as a controlled vocabulary from day one. Mid-project changes to how counties, sub-counties and facilities are identified cascaded through dozens of tables and cost us two weeks of remediation.
Open Source Stack
Our entire health data lake stack is open source: Apache Airflow (orchestration), PostgreSQL + pgcrypto (warehouse), dbt (transformation), MinIO (object storage), Apache Superset (analytics). The licensing cost is zero. The implementation cost is expertise — which is where Edos Centre comes in.