Senior Data Engineer

Built and managed Apache Airflow DAGs to orchestrate Apache Spark batch workloads on AWS EKS, with standardized outputs written to AWS S3.

Developed Python backend services using FastAPI and Celery for data processing and analytics. Added integration tests for Airflow workflows with pytest and used Glitchtip to monitor and resolve production issues.

Migrated dependency management from requirements.txt to uv and pyproject.toml, improving CI/CD speed and simplifying local development setup.

Designed and deployed an Apache Kafka event-streaming pipeline for real-time market trade and order events. Maintained price snapshots in MySQL to support near-real-time margin and risk monitoring.

Implemented a Lambda architecture combining Kafka streaming data with daily Apache Airflow batch jobs for raw-trade archival and reconciliation against official historical data. Migrated legacy Talend ETL jobs to Dockerized Airflow and created idempotent DAGs for safe recovery and duplicate prevention.

Built a FastAPI service to efficiently deliver MySQL warehouse data to internal dashboards handling millions of records. Also developed an internal FAQ solution with n8n and Supabase vector search to reduce support requests.

Developed Python scripts to collect, clean, and automate daily ingestion of financial data from multiple sources.

Used Pandas and Polars to analyze market trends and stock lists for ad hoc business reporting. Built automated real-time stock-news scrapers that filtered events relevant to margin lending.

Connected processed datasets to Power BI to provide daily business performance reporting.

Optimized SQL queries for core reports, improving execution performance by approximately 30%.

Collaborated with developers on Data Mart table structures and indexing strategies to improve data access performance. Resolved errors in daily data-loading processes to maintain accurate reporting.