AI Engineer (Evaluation)

Join Distyl, a company that builds AI systems using Evaluation-Driven Development. As an AI Evaluation Engineer, you will design and implement evaluation frameworks, build and maintain test cases, develop evaluation pipelines, and work closely with various teams to guide system development and deployment. You should have strong Python engineering skills, the ability to translate human judgment into code, a systems-oriented mindset, and experience with evaluation-driven or experiment-driven development. Benefits include equity options, healthcare, and a collaborative work environment.

• Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments.
• Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases.
• Develop and maintain evaluation pipelines—offline and online—that integrate directly into system iteration loops, ensuring that system changes are driven by measurable improvements.
• Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments. You treat evaluation code with the same rigor as application code
• Ability to Translate Human Judgment into Code: Work with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders that scale
• Systems-Oriented Mindset: Understand how evaluation interacts with prompts, agents, data, and deployment. You design evaluation systems that support fast iteration while maintaining trust and safety in production
• AI-Native Working Style: Use AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration
• Experience with Evaluation-Driven or Experiment-Driven Development: Experience using structured evaluation or experimentation frameworks to drive system iteration, and understand the pitfalls of overfitting to metrics that don’t reflect real outcomes
• Travel: Travel between 10-50% of the time, depending on the project, your role and level of interest in doing so
• 2+ years of software engineering experience