About this role.
Serve Robotics is seeking a senior machine learning engineer to build and optimize distributed training systems for autonomy models used in sidewalk-delivery robots. The role centers on petabyte-scale multimodal robotics data, including video, point clouds, and other sensor inputs, processed across large GPU clusters. Key work includes eliminating training bottlenecks, improving data pipelines and GPU utilization, refining model architectures and loss functions, and operating reliable distributed jobs. The engineer will partner closely with ML researchers and infrastructure teams to productionize experiments and accelerate model iteration. Candidates need at least five years of production ML experience, strong Python skills, and deep familiarity with distributed training and large-scale datasets.
Role DNA
A quick view of the complexity, pace, ownership and collaboration implied by the job description.
Job Complexity
5/5Pace & Pressure
5/5Autonomy Level
4/5Communication Load
4/5Salary analysis
Estimated compensation compared with the broader US market for similar roles.
Core skills
Skills and capabilities most closely associated with this opportunity.
Sample interview questions
I would establish baseline metrics for GPU utilization, data-loader wait time, host CPU and memory usage, network throughput, communication overhead, and step-time breakdowns. I would then profile the highest-impact constraint, validate a targeted change with controlled experiments, and monitor both throughput and model-quality effects before rolling it out.
I would first define robust time synchronization, calibration, and quality checks across modalities. The pipeline would use scalable sharding and preprocessing, modality-specific encoders, and a fusion architecture appropriate to the downstream task, while tracking performance by modality availability and environmental scenario.
I would address input-pipeline parallelism, storage locality, batching strategy, mixed precision, gradient accumulation, communication overlap, and efficient distributed data parallel configuration. I would also inspect load imbalance between workers and ensure checkpointing and evaluation do not unnecessarily interrupt training.
I would use structured ablations: verify data integrity and label quality, compare training and validation curves, inspect representative failures, and hold the dataset constant while changing optimization or architecture choices. This makes it possible to isolate whether the issue is poor signal, unstable training dynamics, capacity limitations, or generalization.
I would version datasets, code, configurations, and model artifacts; record environment and hardware details; and use experiment tracking with clear lineage. For production, I would add automated validation, fault-tolerant orchestration, checkpoint recovery, monitoring, and documented runbooks so that successful experiments can be repeated and scaled reliably.
At Serve Robotics, we’re reimagining how things move in cities. Our personable sidewalk robot is our vision for the future. It’s designed to take deliveries away from congested streets, make deliveries available to more people, and benefit local businesses.
The Serve fleet has been delighting merchants, customers, and pedestrians along the way in Los Angeles, Miami, Dallas, Atlanta and Chicago while doing commercial deliveries. We’re looking for talented individuals who will grow robotic deliveries from surprising novelty to efficient ubiquity.
Who We Are
We are tech industry veterans in software, hardware, and design who are pooling our skills to build the future we want to live in. We are solving real-world problems leveraging robotics, machine learning and computer vision, among other disciplines, with a mindful eye towards the end-to-end user experience. Our team is agile, diverse, and driven. We believe that the best way to solve complicated dynamic problems is collaboratively and respectfully.
This role develops and scales large-scale machine learning training systems for multimodal robotics data, enabling the creation of high-performance autonomy models. By optimizing distributed training pipelines, neural network architectures, and data processing workflows, the position improves training efficiency, accelerates model iteration, and maximizes GPU utilization. The role collaborates closely with ML researchers and infrastructure teams, influencing the design, deployment, and performance of end-to-end autonomy models and the large-scale data pipelines that support them.
Responsibilities
Design and maintain training systems that can process and learn from petabyte-scale multimodal datasets (e.g., video and point cloud data). This includes ensuring data is efficiently loaded, distributed, and processed across large GPU clusters.
Identify and resolve bottlenecks in the training pipeline, including data loading, preprocessing, model computation, and inter-node communication, to maximize GPU utilization and reduce training time.
Work with the ML team to develop and refine neural network architectures suitable for autonomy tasks, particularly those handling high-dimensional and sequential sensor data.
Create and adjust loss functions and training strategies that help the model learn effectively from complex multimodal inputs and improve autonomy performance.
Configure, monitor, and maintain large-scale distributed training jobs across multiple machines and GPUs, ensuring stability, fault tolerance, and efficient resource usage.
Implement scalable systems to preprocess, transform, and augment large robotics datasets so that they are suitable for model training.
Work closely with ML scientists and other engineers to integrate new models, experiments, and training approaches into the production training pipeline.
Analyze training metrics, model outputs, and experiment logs to assess model performance and guide improvements in architecture, data usage, or training strategies.
Develop tools and workflows that allow teams to run experiments, track results, and iterate quickly on new model ideas or training approaches.
Qualifications
Master’s or PhD in Computer Science, Robotics, Electrical Engineering, Machine Learning, or a closely related technical discipline.
Minimum of 5 years of professional experience developing, training, and deploying machine learning models in production environments.
Hands-on experience training machine learning models across multiple GPUs or compute nodes, including familiarity with distributed training frameworks and large dataset handling.
Strong programming skills in Python for implementing machine learning models, data pipelines, and training workflows.
Solid knowledge of core concepts such as neural networks, optimization algorithms, loss functions, model evaluation, and training methodologies.
What Makes You Stand out
Experience identifying and resolving training bottlenecks related to compute utilization, memory usage, and data throughput in machine learning systems.
Experience training machine learning models on robotics or autonomous driving datasets involving multimodal sensor inputs such as camera video, LiDAR point clouds, radar, or telemetry data.
Experience developing models that combine multiple data modalities (e.g., images, point clouds, and structured sensor data) into a unified learning system.
Peer-reviewed publications or significant research contributions in machine learning, robotics, or related areas.
*Please note: The listed base salary range applies to candidates based in the US. Compensation may vary depending on location, experience, and role alignment. We are open to qualified candidates working remotely in Canada
Canada – ALL: $177k – $215k CAD
Upload your resume now
- To unlock remote work opportunities and be discovered by global employers.
This job listing has been manually reviewed by the Jobicy Trust & Safety Team for compliance with our posting guidelines, including verification of the company's legitimacy, accuracy of job details, clarity of remote work policy, and absence of misleading or fraudulent content.
