Senior AI Infrastructure Engineer (Model Training)

Join Kodiak Robotics as a Senior AI Infrastructure Engineer, where you'll play a crucial role in optimizing the training of our AI models. You'll design high-throughput data loading and streaming systems, build and optimize distributed training infrastructure, and maximize the utilization of modern accelerators. This remote-friendly position offers a generous benefits package, including a 401k, PTO, and various medical plans.

• Design and implement high-throughput data loading and streaming systems for multimodal sensor data, ensuring GPUs are kept saturated.
• Build and optimize distributed training infrastructure across multi-node GPU clusters, applying various parallelism techniques to models that don't fit on a single device.
• Profile end-to-end training pipelines to identify and eliminate bottlenecks across storage, network, CPU preprocessing, and GPU compute.
• Passion for building the infrastructure that lets AI for the physical world train faster, scale further, and improve continuously
• Hands-on experience with distributed training frameworks and techniques (PyTorch DDP/FSDP, DeepSpeed, Megatron, NCCL) and a strong grasp of parallelism trade-offs
• Experience building high-performance data pipelines for large-scale training, including streaming dataset formats (WebDataset, MosaicML Streaming/MDS, or similar), sharding, and storage/network-aware loading
• Deep understanding of GPU performance: mixed precision, memory hierarchy, kernel fusion, profiling tools (Nsight, PyTorch Profiler), and interconnects (NVLink, InfiniBand)
• BS, MS, or PhD in Computer Science or a related field, and at least 2-3 years of industry experience in ML systems or infrastructure
• Strong Python skills and proficiency in PyTorch internals; systems-level experience (C++/CUDA/Triton) a plus