Senior ML Engineer (Token Factory)

Remote from: UK, Europe +4 more, Germany, Netherlands, Israel, Czechia
Annual salary: Undisclosed
Salary information is not provided for this position. Check our Salary Directory to estimate the average compensation for similar roles.
Department: Software Engineering
Employment type: Full Time,
Job posted: 4 Jul 2026
Apply before: 4 Aug 2026
Experience level: Senior
Views / Applies: 23 / 2

Nebius is the AI cloud company, delivering a unified platform that spans the complete AI journey from data and model training and tuning to production runtime and deployment.

Computer Software
2023

Actively Hiring

AI Summary

Nebius is building a full-stack AI cloud platform and is seeking a Senior ML Engineer for their Token Factory team, which focuses on high-performance inference and fine-tuning for foundation models. The role involves optimizing LLM inference, implementing speculative decoding, and designing low-precision training pipelines. Candidates need deep ML theory, GPU profiling skills, and strong software engineering abilities. The company offers competitive compensation, career growth, and a collaborative culture.

Role DNA

Job Complexity

Easy Hard

Pace & Pressure

Relaxed Fast-paced

Autonomy Level

Guided Full Ownership

Communication Load

Independent Highly Collaborative

AI Insight This role requires deep expertise in ML theory, GPU optimization, and performance engineering, pushing models to hardware limits, which is highly challenging. The expectation of contributions to open-source inference engines adds complexity, justifying a difficulty score of 4.

Salary Analysis

AI Insight No specific salary was provided in the listing. The estimated market median for a Senior ML Engineer in the US is around $200,000, with a typical range of $150,000 to $250,000. The offered salary is likely competitive within this range, considering the role's specialized focus on inference optimization and GPU performance.

Core Skills Required

Python GPU Profiling LLM Inference Transformers PyTorch CUDA vLLM Computer Architecture Distributed Systems Machine Learning

Cover Letter Sample

Dear Hiring Team,

I am writing to express my strong interest in the Senior ML Engineer (Token Factory) position at Nebius. With a profound understanding of transformer architectures and extensive experience in GPU workload profiling using tools like Nsight and PyTorch profiler, I am excited about the opportunity to maximize inference throughput and minimize latency across tens of thousands of GPUs.

My background includes contributing to open-source inference engines such as vLLM and TensorRT-LLM, and I have successfully implemented low-precision training pipelines using FP8. I thrive in fast-paced startup environments, take full ownership of projects, and collaborate effectively with cross-functional teams.

I am eager to bring my expertise in LLM inference optimization and distributed systems to Nebius, and I look forward to the possibility of discussing how I can contribute to the Token Factory's mission. Thank you for considering my application.

Sincerely,
[Your Name]

Sample Interview Questions

Can you explain how you would profile and identify bottlenecks in a large-scale LLM inference pipeline using NVIDIA tools?

I would start by using NVIDIA Nsight Systems to get a high-level overview of kernel execution, memory transfers, and CPU operations. Then, I would drill down with Nsight Compute to analyze individual CUDA kernels for occupancy, memory bandwidth, and compute utilization. I would also use PyTorch Profiler to trace the model execution and identify bottlenecks like attention computation or feed-forward layers. Based on the profiling results, I would optimize by fusing kernels, using Flash Attention, or adjusting batch sizes.

Describe your experience with speculative decoding and how you would implement a novel architecture for a transformer model.

I have implemented speculative decoding using a smaller draft model to generate candidate tokens accepted by the target model in parallel. For a novel architecture, I would explore using a multi-layer perceptron as the draft model or adapt the Medusa approach with multiple heads. I would carefully manage the acceptance rate and latency trade-offs, and profile to ensure the overhead does not outweigh the speedup.

How do you approach designing a low-precision training pipeline for FP8, and what challenges have you faced?

I start by converting model weights and activations to FP8 using quantization-aware scaling. I use per-tensor scaling factors to maintain precision and handle outliers. Challenges include loss of accuracy due to limited dynamic range, which I mitigate with mixed-precision training (e.g., keeping master weights in higher precision). I also handle gradient scaling and ensure that batch normalization layers are computed in FP32. I evaluate the model performance on validation datasets to tune the scaling factors.

Explain the trade-offs between throughput and latency in LLM serving and how you would optimize for both.

Throughput is maximized by batching requests, while latency is minimized by reducing batch sizes or using dynamic batching. I would use continuous batching to interleave requests and increase GPU utilization without sacrificing latency. Additionally, using tensor parallelism and pipeline parallelism can help. I would also optimize memory management with PagedAttention and use kernel fusion to reduce kernel launch overhead.

Given your experience with distributed systems, how would you design a scalable inference system for serving multiple models across thousands of GPUs?

I would use a microservices architecture with a load balancer distributing requests to inference workers. Each worker would manage a set of GPUs running model instances with tensor parallelism. I would implement model sharding to distribute model layers across GPUs. To handle dynamic loads, I would use orchestration tools like Kubernetes and auto-scaling groups. For low latency, I would optimize network communication with techniques like RDMA and use caching for frequently used KV-cache entries.

About Nebius:

Nebius is leading a new era in cloud infrastructure for the global AI economy. We are building a full-stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, without the cost and complexity of building large in-house AI/ML infrastructure.

Built by engineers, for engineers. From large-scale GPU orchestration to inference optimization, we own the hard problems across compute, storage, networking and applied AI.

Listed on Nasdaq (NBIS) and headquartered in Amsterdam, we have a global footprint with R&D hubs across Europe, the UK, North America and Israel. Our team of 1,500+ includes hundreds of engineers with deep expertise across hardware, software and AI R&D.

The role

Token Factory is a part of Nebius Cloud, one of the world’s largest GPU clouds, running tens of thousands of GPUs. We are building a high-performance inference and fine-tuning platform designed to push foundation models to their hardware limits. Our mission is to maximize throughput, minimise latency, and optimise cost-per-token across tens of thousands of GPUs.

Some directions we are currently working on, and which you can be a part of:

Inference Optimization: Identifying LLM inference bottlenecks to drive production speedups. Squeezing the maximum performance for a wide range of LLM architectures at scale (e.g., GPT-OSS, Kimi K2.5, DeepSeek V3.1/V3.2, GLM-5).
Inference engines support: Implement novel speculative decoding architectures, optimise components of various LLM designs (dense/MoE, autoregressive/parallel), and contribute to open-source inference engines.
Low Precision Training & Inference: Design and productionise low-precision (FP8, NVFP4/MXFP4) training and inference pipelines with measurable gains in throughput and cost-efficiency.

We expect you to have:

A profound understanding of theoretical foundations of machine learning and transformer architecture.
Experience profiling GPU workloads using Nsight, PyTorch profiler, or similar tools
Understanding of GPU memory hierarchy and compute/memory tradeoffs
Familiarity with important ideas in LLM space, such as MHA, RoPE, KV-cache, Flash Attention, and quantisation
Understanding of performance aspects of large neural network training (sharding strategies, custom kernels, hardware features etc.)
Strong software engineering skills (we mostly use Python)
Deep experience with modern deep learning frameworks
Proficiency in contemporary software engineering approaches, including CI/CD, version control and unit testing
Strong communication and leadership abilities

Nice to have:

Experience working with open-source inference engines (vLLM, SGLang, TensorRT-LLM), including contributions
Experience with kernel languages or DSLs such as Triton, Cute, CUTLASS, CUDA
A track record of building and delivering products (not necessarily ML-related) in a dynamic startup-like environment.
Strong engineering skills, including experience in developing large distributed systems or high-load web services.
Open-source projects that showcase your engineering prowess
Excellent command of the English language, alongside superior writing, articulation, and communication skills.

Benefits & Perks:

Competitive compensation
Career growth and learning opportunities
Flexibility and ownership
Collaborative and innovative culture
Opportunity to work on impactful AI projects
International environment and talented teams

What’s it like to work at Nebius:

Fast moving – Bold thinking – Constant growth – Meaningful impact – Trust and real ownership – Opportunity to shape the future of AI

Equal Opportunity Statement:

Nebius is an equal opportunity employer. We are committed to fostering an inclusive and diverse workplace and to providing equal employment opportunities in all aspects of employment. We do not discriminate on the basis of race, color, religion, sex (including pregnancy), national origin, ancestry, age, disability, genetic information, marital status, veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by applicable law.

Applicants must be authorized to work in the country in which they apply and will be required to provide proof of employment eligibility as a condition of hire.

If you need accommodations during the application process, please let us know.

Apply now >