AWS Data Engineer

Working as an AWS Data Engineer at Birlasoft Limited, designing and delivering cloud-native data engineering solutions, scalable data pipelines, enterprise data lakes, and analytics platforms for banking, retail, and enterprise clients.

Develop ETL/ELT, batch, incremental, and real-time streaming pipelines using AWS services, PySpark, Python, SQL, Snowflake, and Apache Airflow. Contribute to architecture discussions, code reviews, production deployments, documentation, issue resolution, and team mentoring.

Designed and implemented a real-time ingestion pipeline using Kinesis Data Streams and Kinesis Firehose to capture high-volume application events and route them to Amazon S3 and Amazon Redshift. Built Lambda-based processing functions for real-time transformation, validation, and enrichment.

Created centralized monitoring with CloudWatch and Kinesis Firehose, and configured EventBridge and SNS alerts for pipeline failures, data-quality breaches, and shard lag anomalies. Optimized Kinesis shard configurations and Lambda concurrency for burst traffic while implementing IAM policies and KMS encryption for secure data delivery.

Designed enterprise-scale AWS Glue and PySpark ETL pipelines to ingest and transform data from more than six API and database source systems into Snowflake, reducing data latency by 35%. Built a metadata-driven ingestion framework that enabled configuration-based onboarding of new data sources.

Developed Apache Airflow DAGs with task dependencies, retry logic, and SLA alerts. Implemented Python-based data quality validations, reducing manual remediation effort by 80%, and established CI/CD and infrastructure-as-code deployment using GitHub Actions and Terraform.

Designed Snowflake fact and dimension models, clustering strategies, and materialized views for analytics reporting. Applied SCD Type 2 logic in PySpark to preserve historical dimension records.

Migrated legacy on-premises ETL workloads to AWS Glue and re-engineered batch jobs with PySpark for cloud-native scalability. Developed transformations for cleansing, enrichment, and business-rule processing across banking transaction datasets.

Built a metadata-driven ETL framework, designed watermark-based incremental and full-load strategies, and optimized Glue performance using DPU tuning, S3 partitioning, and Parquet. Loaded curated datasets into Amazon Redshift and configured distribution keys, sort keys, and workload management settings for analytical performance.

Implemented structured logging, error handling, and CloudWatch monitoring to improve pipeline observability and support SLA adherence.

Bachelor of Science in Mathematics, Statistics and Computer Science.