Senior Software Development Engineer (SDE2) | Web Scraping & Automation Loti AI
Designed, developed, and optimized large-scale distributed web-scraping systems processing millions of data points daily across more than 110 domains. Built resilient Python-based scrapers using Scrapy and Playwright to manage dynamic websites, complex navigation, authentication barriers, pop-ups, and CAPTCHAs.
Implemented automation workflows, proxy rotation, request optimization, distributed task execution, JSON data storage, and AWS S3 uploads. Used Scrapydweb and cron jobs for scheduling, management, monitoring, and continuous operation of high-availability data pipelines.