Healthcare Data Platform — Ingestion Automation
PalTech ↗ · Jul 2024 — PresentEvent-driven AWS Lambda pipeline that cut manual data-loading effort by 30–50% on a US healthcare platform.
The problem
The platform depends on high-volume healthcare data published by CMS and other external sources as CSV files. Developers were downloading and loading these files by hand — slow, error-prone, and a recurring tax on every data refresh cycle.
What I built
An event-driven ingestion pipeline using AWS Lambda functions to fetch, validate, and load source files automatically. On the database side, I designed partitioned tables for monthly source data and added materialized views with post-load refresh mechanisms, cutting data redundancy and keeping reporting consistent across the application.
Outcome
Manual developer effort on data loading dropped by 30–50%, and the schema work keeps query performance predictable as monthly volumes grow. I own this platform end to end as the team's dedicated offshore engineer, working daily with an onshore lead.