Data Engineer

Architected a multi-layer AWS data lake using S3, Glue, Redshift, and Snowflake to integrate EHR, claims, and flat-file sources for enterprise healthcare analytics and machine learning use cases. Supported more than 20 Apache Airflow-orchestrated workflows processing over 300 GB daily at a 99% SLA.

Built and optimized Python-based ETL pipelines on Apache Airflow, reducing end-to-end runtime by 30%. Redesigned data models and tuned Snowflake and Redshift SQL through indexing, clustering keys, and partition pruning, improving query and Tableau dashboard performance by 35%.

Developed an automated Great Expectations data-quality framework covering null checks, schema enforcement, and referential integrity, reducing reporting defects by 25%. Automated CI/CD deployments with Jenkins and GitHub Actions, reducing manual deployment effort by 50%.

Implemented HIPAA-compliant governance using IAM access controls, field-level encryption, audit logging, and AWS Glue Data Catalog lineage and metadata management for PHI datasets, reducing unauthorized-access incidents by 30%.

Designed a PySpark ingestion framework on AWS using S3, Glue, and Lambda to process more than 500 GB of nightly financial transaction data. Applied dynamic partitioning and optimized shuffle configuration for efficient large-scale processing.

Implemented a watermark-based incremental loading strategy with timestamp tracking and change detection, reducing batch processing time by 40%. Built modular and parameterized Apache Airflow DAGs to orchestrate more than 15 production ingestion workflows with SLA monitoring.

Developed Star Schema dimensional models and curated data marts in Snowflake and PostgreSQL to support Tableau and Power BI dashboards. Implemented dbt- and PySpark-based data-quality checks for regulatory KPI accuracy.

Containerized ETL workloads with Docker, reducing environment-related defects by 30%, and mentored three junior engineers on ETL and Airflow best practices.