Shiva Kotagiri
Shiva Kotagiri

Senior Data Engineer

Open to offers · Member since 12 Oct 2026
Location
Amherst, United States
Desired salary
Unspecified
Work preference
On-site / Full Time
Experience level
Senior

About

Professional summary

I am a data engineering professional with over four years of experience in data analytics, data integration, and cloud-based data platforms. I build reliable, analytics-ready datasets that support reporting, operational analysis, and governed data products.

I specialize in Python, SQL, PySpark, Apache Spark, Databricks, Delta Lake, and ETL/ELT development. My work includes transforming high-volume structured data, developing reusable transformation logic, and improving data pipeline reliability.

I have strong experience with dimensional data modeling, including fact and dimension tables, star schemas, and curated reporting datasets. I apply data reconciliation, validation, profiling, deduplication, and completeness checks to ensure trusted data outputs.

I work with lakehouse and medallion architecture patterns, organizing data across Bronze, Silver, and Gold layers. I am familiar with batch and streaming concepts, incremental processing, schema evolution, access controls, metadata, and data lineage.

I use collaborative engineering practices such as Git, pull requests, code reviews, CI/CD, documentation, monitoring, logging, and root-cause analysis. I partner with business and analytics stakeholders to translate requirements into scalable data solutions.

I am Databricks Certified Data Engineer Associate and AWS Certified Data Engineer Associate. I am currently based in Amherst, Massachusetts, and open to relocation.

Skills

24 capabilities

Tech stack & tools

Working toolkit

Application Hosting

Development

Languages & Frameworks

Experience

Career history

Senior Data Engineer Worcester Commons

Develop and maintain scalable data transformation workflows that convert operational and business data into reliable, analytics-ready datasets. Use SQL, Python, Spark, PySpark, and Databricks-oriented patterns to create reusable logic, standardize business rules, and process growing datasets efficiently.

Design dimensional models and layered Bronze, Silver, and Gold data processing patterns. Implement ETL/ELT workflows, incremental processing, data-quality validation, reconciliation, exception handling, logging, and monitoring. Apply Git, pull requests, code reviews, and CI/CD practices while partnering with analysts and stakeholders on data models, documentation, and issue resolution.

Data Engineer Ujjivan Small Finance Bank

Built and maintained SQL- and Python-based data preparation workflows for banking and transaction datasets used in operational analysis and management reporting. Integrated relational sources and structured files, applying standardization, transformation, and validation before publishing analytical datasets.

Created and optimized complex SQL queries, joins, aggregations, and reusable transformations. Supported data warehouse modeling, source-to-target reconciliation, data-quality checks, and investigation of missing, duplicate, or inconsistent records. Worked with Snowflake and SQL Server, automated recurring preparation tasks, and maintained transformation, mapping, and validation documentation using Git-based development practices.

Education

Learning history

University of Massachusetts

Master of Science, Business Analytics

Master of Science in Business Analytics, University of Massachusetts, Amherst, Massachusetts.

This professional hasn’t added portfolio projects yet.

This professional hasn’t listed any services yet.

Jobs Talent AI Tools Salaries
Menu