Principal Data Engineer

Lead a high-performing team of 5 data engineers in designing and maintaining enterprise-scale data pipelines and infrastructure across AWS and Azure. Engineered containerized, Docker- and Kubernetes-based data pipelines using Kafka, Python microservices, and REST APIs, reducing processing time by 40%. Designed scalable, high-performance data architecture supporting real-time analytics across 10+ business units. Built multi-source streaming platforms with Apache NiFi and Apache Airflow to support data lakes and warehouses. Optimized SQL and Bash scripting for automated data ingestion, transformation, and workflow orchestration. Managed Git-based CI/CD pipelines for version control and automated deployment of data pipelines. Standardized logging and audit frameworks for ETL processes, enabling faster troubleshooting and root-cause analysis. Collaborated with architecture leadership to define governance, lineage, and access frameworks using Unity Catalog and schema registry.

Designed and optimized large-scale ETL/ELT pipelines using Spark and Airflow to process billions of retail and consumer data records. Migrated legacy Teradata and Oracle systems to Snowflake and AWS S3, cutting costs by 40% and improving performance. Integrated processed datasets into Amazon Redshift to support real-time business intelligence and reporting needs. Created Power BI dashboards to track key KPIs and provide actionable insights for business stakeholders. Implemented data validation, monitoring, and lineage tracking to ensure data consistency and governance compliance. Used Git for version control and deployed pipelines via automated CI/CD workflows to ensure reliability and maintainability. Collaborated with cross-functional teams to improve data accessibility, reliability, and overall data engineering best practices.

Modernized legacy ETL pipelines and data warehouse workflows, achieving a 30% improvement in processing speed. Implemented Python, Kafka, Elasticsearch, FluentD, and GCP (GKE) solutions, supporting 5+ TB daily data ingestion. Assisted in migration from on-prem Oracle systems to AWS Redshift improving scalability and reporting performance. Designed and implemented Star Schema data models for internal reporting, improving query performance and simplifying business analytics. Created ELK-based logging pipelines to capture operational metrics and generate audit-ready reports for internal teams. Created lightweight data lineage maps to help stakeholders trace data flow from source to report.