Senior AI Reliability Engineer (Platform)

Join Flatiron Health as a Senior AI Reliability Engineer (Platform) to help scale AI-enabled workflows across engineering, product, and business teams. This role focuses on building systems, standards, and feedback loops for evaluating, monitoring, and improving AI workflows. You will design evaluation frameworks, define quality metrics, develop reliability engineering practices, and partner with various teams to evaluate AI solutions. The position offers flexible work hours, a 401(k) contribution, financial health resources, parental benefits, mental well-being tools, travel support for healthcare services, and programs supporting fertility, adoption, and surrogacy.

• Design, build, and continuously improve evaluation frameworks, benchmarks, and automated testing pipelines for AI, LLM-powered, and agentic workflows.
• Define and monitor quality, reliability, safety, performance, and cost metrics for AI systems, including observability, drift detection, hallucination risk, retrieval quality, and end-to-end workflow behaviour.
• Develop reliability engineering practices for AI-enabled systems, including SLOs, SLIs, monitoring, alerting, incident response, runbooks, and root-cause analysis of AI failure modes.
• We’re looking for a Senior AI Reliability Engineer (Platform) to help us accomplish our mission to improve and extend lives by learning from the experience of every person with cancer. Are you ready to be the next changemaker in cancer care?
• You understand that AI systems fail differently from traditional software
• A model may not crash, but it may silently degrade, become less accurate, respond inconsistently, produce poor outputs, or create business risk in ways that are hard to detect without the right evaluation and observability patterns
• You are comfortable operating in ambiguous spaces where the right answer is not always obvious, and you are motivated by turning emerging AI capabilities into production-ready systems that teams can actually trust
• This role is focused on that production behaviour and system health, not on pure model research or training
• You’re a senior technical practitioner with experience working across data science, machine learning, software engineering, platform engineering, or reliability engineering
• 5+ years of experience in platform engineering, SRE, machine learning, MLOps or a related technical field, with strong Python skills and experience building production-quality systems
• Strong understanding of AI reliability and observability, including logging, tracing, monitoring, drift detection, statistical analysis, uncertainty, alerting, and production system health
• Experience designing experiments, evaluation frameworks, statistical analyses, and quality metrics for ML or AI systems, with familiarity in LLMs, RAG, AI agents, prompt evaluation, and model behaviour
• Knowledge of agentic and multi-agent systems, including orchestration, state management, tool execution, governance, reliability, human-in-the-loop controls, and selecting the appropriate level of AI autonomy for a given problem
• Experience with modern cloud and ML infrastructure, including AWS, containers, Kubernetes, CI/CD, data pipelines, workflow orchestration, versioning, and distributed compute platforms
• Fluent in English
• Strong communication and collaboration skills, with the ability to explain AI behaviour and tradeoffs to technical and non-technical stakeholders and thrive in a fast-moving, ambiguous environment with a pragmatic, enablement-focused mindset
• Experience with LLM evaluation, red-teaming, adversarial testing, AI safety, RAG evaluation, retrieval quality measurement, embedding drift, or AI observability and model monitoring
• Experience working in healthcare, life sciences, or other regulated, privacy-sensitive environments
• Hands-on experience with observability and data/ML platforms such as Datadog, Splunk, OpenTelemetry, Databricks, Spark, Airflow, dbt, Ray, SageMaker, GitLab CI/CD, or similar technologies