Senior Data Engineer

Working as a Senior data Engineer with 9+ years of expertise in Big Data Analytics, Microsoft Azure, and Google Cloud Platform (GCP), adept at leading end-to-end data pipeline projects and ensuring client satisfaction through strong project management and communication skills.
• Proficient in a wide array of Big Data technologies including Spark, Hadoop, Hive, and Apache Airflow, with hands-on experience in cloud platforms such as GCP (BigQuery, Dataproc, Dataflow), Azure (Azure Data Lake, Data Factory, Databricks, Azure Synapse, Microsoft Fabric), Snowflake and AWS, enabling effective data processing and analytics solutions.
• Expertise in implementing high-performance ETL processes using Azure Data Factory (ADF) and Azure Databricks to ensure efficient data movement and transformation.
• Monitor Scheduled Jobs, Live Production Environment, and other automation Jobs.
• Proficient in managing databases such as Teradata, Cosmos DB, Oracle, and MySQL, BigQuery with hands-on experience in data warehousing tools like Infoworks and Azure Data Factory. Skilled in Python, Scala, Java, Unix Shell, and SQL for effective data manipulation and transformation.
• Proficient in designing and implementing secure architecture solutions leveraging Azure technologies such as Azure Application Gateway, API Management Gateway, and Service Bus to ensure data privacy and compliance.
• Skilled in leading data migration projects, including the migration of data from legacy systems to modern data warehouse stacks using Azure Data Factory (ADF), Azure Databricks, Azure Synapse.
• Experienced in processing and transforming distributed data in staging layers using Azure Databricks to provide insights and analytics, ensuring data accuracy and integrity.
• Proven ability to identify and address data quality issues through comprehensive data cleansing strategies, ensuring data reliability and consistency.
• Strong visualization skills, with experience in building interactive dashboards and reports using Power BI, Tableau, and Azure Databricks to provide actionable insights to stakeholders.
• Proficient in cloud-native data processing services such as Google Cloud Data Fusion and Google Cloud Composer (Airflow) for streaming and real-time data processing.
• Skilled in implementing end-to-end pipelines on Google Cloud Platform (GCP) for data ingestion, transformation, and analytics, ensuring optimal performance and scalability.
• Experienced in automating data workflows and tasks using Google Cloud Composer (Airflow) and scripting with Python, enabling efficient and reliable data processing.
• Knowledgeable in data governance best practices and ITIL principles, with experience in implementing security measures and monitoring tools to ensure data integrity and availability.
• Experienced in working with SQL and Kusto Query Language (KQL) for data querying and manipulation tasks.
• Skilled in utilizing visualization tools like PowerBI and Tableau for building interactive dashboards and reports to provide actionable insights.
• Experienced in using CVS tools like Git, Jenkins, and Azure DevOps for version control and continuous integration/continuous deployment (CI/CD) processes.
• Experienced in designing end-to-end real-time data processing pipelines using Spark Streaming, Kafka and Azure Event Hubs. Proficient in building and deploying Spark Streaming applications to ingest, process, and analyze streaming data in near real-time.