Data Engineer Haystack Search LLC
• Data Pipelines: Design and automated ETL pipelines using Python, SQL, and Apache Airflow to process over 25M+ business records for real-time ML search applications.
• Collaboration: Partner with co-workers to reduce ETL pipeline runtimes by 75% through asynchronous ETL Python scripts.
• Data Manipulation: Configure Docker containers on AWS EC2 instances for scheduled Python and SQL scripts that ingest and cleanse small business data for efficient and high-quality user search results.