Senior Software Engineer (AI Evals)

Join Sentry's AI/ML team as a Senior Software Engineer, where you'll build the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This critical role involves designing datasets, benchmarks, and test harnesses to ensure our AI-powered features behave correctly and predictably as they scale. You'll collaborate closely with applied AI engineers and product leaders, own the evaluation lifecycle for major AI initiatives, and enjoy a range of benefits including competitive compensation, medical insurance, flexible PTO, and more.

• Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems.
• Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data.
• Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria.
• Care deeply about correctness, rigor, and measurement in AI systems
• Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
• Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
• Thrive in cross-functional environments and enjoy influencing model design through better evaluation
• Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
• Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
• Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools
• Comfort writing production-quality code (we use Python and TypeScript)
• Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
• Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines