Staff Backend Engineer (Application Core Services)

Join Grafana, a leading open-source analytics and monitoring platform. As a Staff Backend Engineer, you will work on critical systems that power our business, collaborating with various teams to ensure smooth operations and scalability. You will design, build, and operate reconciliation systems, improve operational efficiency, and support new region and cluster rollouts. This is a remote-first position with a focus on developer productivity and professional development. - Design, build, and operate reconciliation systems, including the SSS backend, to track desired stack state, detect and repair drift across stack templates, state, Hosted Grafana, and actual customer stack configuration. - Collaborate across SSS, , and deployment configurations to ensure stack lifecycle workflows remain reliable, observable, and resilient. - Improve operational efficiency by reducing deployment complexity (e.g., aiming for single PR regional SSS deployment) and contributing to the Stack Config Reconciliation project. - We are looking for engineers who enjoy solving complex workflow and systems problems, improving reliability and developer experience, and building software that directly supports both customers and internal stakeholders - We are seeking a Staff Backend Engineer who thrives on building production systems where correctness, scalability, and operational clarity are paramount. As a remote-first organization, you should be comfortable collaborating asynchronously across time zones and taking full ownership of the critical systems powering Grafana Cloud - You should be adept at breaking down complex systems work into safe, iterative increments while clearly communicating technical tradeoffs to both internal stakeholders and adjacent product teams - You will be particularly successful in this role if you enjoy solving challenges related to stateful systems, eventual consistency, and reconciliation loops. We value engineers who can take ambiguous lifecycle requirements and transform them into explicit, modular solutions - Experience participating in blameless incident response and writing high-quality post-incident reviews - Have professional experience with Golang and be willing to work across both backend service and application code - Can take on complex challenges and break them down to achieve tight learning loops: to analyze, design, and build modular solutions, deliver MVPs, gather data and feedback, and then progress iteratively - You write clean, robust, well-tested software that other engineers can understand, operate, and maintain - Care deeply about developer and user experience and the quality of the products that you work on - Have some experience with delivering projects from gathering requirements, and brainstorming ideas to shipping a product to the customer’s hands in a self-driven way - You are willing to work across teams. Your work has to be aligned with the needs of other squads and external stakeholders. You make your plans transparent, bring stakeholders on board, and are open to feedback and suggestions - Strong Kubernetes experience in AWS, GCP, or Azure, and familiarity with infrastructure-as-code tooling (Helm, Terraform, Jsonnet, etc.) - You have worked on a big SaaS platform and dealt with common distributed systems problems (e.g. scalability, multi-tenancy, data isolation, HA, …) - You have at least 1 year of fully remote work experience - Have experience with mentoring junior engineers in a collaborative but asynchronous environment - Experience with TypeScript/Node.js - Experience with Jsonnet/Tanka, Terraform, Flux, Argo, or similar deployment/configuration tooling - Experience with Kubernetes control-plane patterns, operators, reconcilers, or desired-state systems - Experience working on SaaS provisioning, tenancy, regional expansion, plugin rollout, or customer lifecycle systems - Experience with incident response involving configuration drift, partial failure, or cross-service state mismatch