Senior Applied AI Scientist

Join Ro, a company focused on revolutionizing healthcare through AI-powered products. As a Senior Applied AI Scientist, you will be responsible for evaluating, measuring, and optimizing our AI systems. This role requires strong skills in data science, applied machine learning, and product engineering. You will work closely with engineers and product teams to improve the accuracy, reliability, and cost-effectiveness of our AI products.

• Design and own evaluation frameworks for production LLM features, including LLM-as-a-judge evaluations, regression suites, synthetic datasets, golden datasets, and human review workflows.
• Analyze production behavior to identify quality issues, hallucinations, latency bottlenecks, cost regressions, and emerging failure modes.
• Define the metrics that matter and build dashboards that make AI performance visible across the organization.
• Strong Python and SQL skills with experience working on production data pipelines and experimentation
• Bonus: Experience with evaluation platforms (e.g. Braintrust, LangSmith, OpenAI Evals), experimentation platforms, causal inference, healthcare, or operations-heavy environments
• 5+ years of experience in data science, applied machine learning, experimentation, or a closely related field, with at least the last year focused on applied LLMs or AI evaluation
• You have experience designing reproducible evaluation frameworks rather than relying on manual spot checks or qualitative assessments
• You’re comfortable working closely with engineers and product teams to translate experimental findings into production improvements
• You have strong statistical intuition: you think in terms of distributions, confidence intervals, variance, and sample sizes rather than anecdotes