Software Engineer, ML Serving - Rime Ai

What You'll Own

• -
• Architecture and implementation of Rime's TTS serving infrastructure, from GPU-backed inference engines to the API surface.
• -
• Model optimization from a single-node to disaggregated fleet serving.
• -
• Compatibility with different NVIDIA hardwares from Hopper to Blackwell and beyond for on-prem and cloud deployments.
• -
• Continuous integration and deployment workflows for the model serving pipeline.
• -
• Site reliability: on-call rotation, monitoring, alerting, and observability across the serving stack.
• -
• Resource provision, cost management across our GPU fleet.

What We're Looking For

• -
• Hands-on experience with real-time multinode ML serving infrastructure — ML serving framework experience: NVIDIA Dynamo/Triton, vLLM, SGLang, or equivalent.
• -
• Experience with distributed or disaggregated model serving (Tensor Parallel, Pipeline Parallel, or equivalent).
• -
• Strong cloud infrastructure fundamentals: Linux internals, networking, containerization (Docker, Kubernetes).
• -
• IaC experience — Terraform, Packer, or comparable. You should have opinions about how to do this right.
• -
• On-call is part of the job. You treat production reliability as a shared responsibility.

Nice to Have

• -
• Experience with multinode training (DDP, FSDP, etc.).
• -
• Experience with gRPC or other bidirectional binary streaming protocols.
• -
• Experience with audio streaming and related technologies (WebRTC, WebSockets, etc.).
• -
• Experience with a multilingual monorepo where you pick the best language out of merit more than personal experience.
• -
• Experience with multi-cloud infrastructures (AWS, GCP, OCI, etc.).
• -
• Comfort with configuration management tooling (Ansible, Chef, Puppet, or similar).
• -
• SRE, DevOps, or platform engineering background at a startup.
• -
• Experience at an early-stage company.

Why Join Rime

• -
• Build the serving infrastructure behind a category-defining voice AI company from the ground up.
• -
• You will bring in experience that no one else currently has at the company: you can help us set the vision.
• -
• Direct collaboration with the inference, platform, and ML teams — no handoff culture.
• -
• The systems you build determine what experiences our customers can deploy at scale.
• -
• Meaningful equity upside at an early stage.
• -
• High ownership, high standards, low bureaucracy.
• -
• SF / Bay Area.
• At Rime, we...
• -
• Are outliers
• -
• Cut through the hype to focus on the craft
• -
• Move fast with agency and freedom
• -
• Maintain a growth mindset, finding joy in the struggle
• -
• Do the right things, knowing that it'll lead to making money
• If that sounds like you too, you'll be a great fit for Rime!