Data Engineer
About Me
I am a Data Engineer with over 12 years of experience designing and implementing large-scale, cloud-native data pipelines using Azure, GCP, and Databricks. My expertise lies in batch and real-time data integration, data lakes, and modern warehousing technologies such as Snowflake, PySpark, and BigQuery. I have a proven track record of improving data quality, reducing processing time, and driving cost optimization through automation and architectural enhancements.
Throughout my career, I have delivered high-impact analytics solutions across various domains including healthcare, retail, and automotive. I am skilled in building scalable data pipelines, developing monitoring systems for data quality, and implementing real-time ETL solutions. I am also experienced in cross-cloud integration and have worked extensively with cloud platforms like Azure and Google Cloud.
I hold a Master of Computer Science and Application degree from Bharathiar University, India, and recently completed a Post Graduate Program in Cloud Computing from GenAI University of Austin, Texas. I am certified in multiple technologies including Databricks Apache Spark, Azure Data Solutions, Google Cloud Professional Data Engineer, Apache Cassandra, Tableau, and Oracle SQL & PL/SQL.
In my current role as a Senior Data Engineer at Copart, I design and implement data pipelines for operational and financial data, optimize query performance, and develop analytics tools that enhance cost efficiency. I collaborate closely with stakeholders to translate business requirements into scalable data solutions and reporting frameworks.
Previously, I worked at Cognizant as a Senior Associate, where I managed data pipelines for healthcare clients such as Kaiser Permanente. I implemented ETL workflows using Azure Data Factory, Synapse Analytics, Informatica, and Databricks, and led global teams to deliver technical solutions. My work contributed to improved data quality, faster release cycles, and increased revenue through enhanced claims acceptance rates.
Earlier in my career, I served as a System Engineer and ETL Developer at Tata Consultancy Services, working on large-scale Hadoop and ETL projects for clients like The Home Depot. I developed ETL workflows, managed production migrations, and supported big data initiatives using Hive, Spark, and MapReduce.
I am passionate about leveraging cloud technologies and data engineering best practices to build efficient, reliable, and secure data ecosystems that empower organizations to make data-driven decisions.
Skills
PythonSQLAzureCI CDGitAutomationKafkaFlask
Education
Master of Computer Science and Application
Post Graduate Program in Cloud Computing
Experience
Designed and implemented data pipelines to load operational and financial data into Data Lake and analytics databases. Developed monitoring systems for data quality and pipeline status using Google Cloud, BigQuery, Snowflake, Airflow, Azure Data Factory, and DBT. Optimized data loading performance resulting in 30% reduction in query execution time. Built real-time ETL pipelines using Kafka, Azure ADLS, and Snowflake. Developed Flask + DuckDB microservice for cost optimization analytics reducing slot usage by 15-20%. Created CI/CD-enabled pipelines with Git and Jenkins. Implemented data access controls using IAM roles, authorized views, and policy tags in BigQuery. Collaborated with stakeholders to translate business requirements into scalable GCP-based data solutions and Tableau reporting.
Worked on NCAP project for Kaiser Permanente, implementing data pipelines in Azure Data Factory and Synapse for healthcare claims cost analysis. Designed ETL workflows extracting data from Oracle and loading into Synapse. Built reverse pipelines for outbound data submissions to ADLS. Implemented Star Schema data models and automated data validation reports. Integrated Azure ADF & Synapse with Git and Jenkins for CI/CD. Developed Databricks (PySpark) data pipelines and conducted data profiling and root cause analysis.
Managed Informatica-based data pipelines for healthcare data extraction and claims loading into SQL Server for CSR submissions. Analyzed requirements and developed ETL applications using Informatica, Oracle, SQL Server, and Unix. Orchestrated data processing for Medicare and ACA risk adjustments. Managed global teams and technical deliverables. Introduced error management system improving Medicare claims acceptance rate from 95% to 98.3%. Developed SAS-based ETL pipelines and worked on healthcare claims, membership, and benefits data.
Worked on .NET and SQL Server applications for tool rental maintenance. Developed ETL workflows using Informatica for retail data, collaborating with business teams for UAT and data issues. Developed Teradata BTEQ scripts and generic shell scripts for ETL and reporting. Worked on Hadoop (HDFS, Hive) ETL processes and big data projects using Spark and HiveQL. Responsible for design documentation, coding, testing, support, deployment, and production migration coordination.