Staff Software Engineer - O11y, Asserts at Grafana Labs

Asserts.ai team joined Grafana Labs as part of an acquisition and is now growing inside the larger Observability group. The backend team is responsible for building and operating a set of low-latency microservices built in Java and databases like Graph DB. The services provide automated Root Cause Insights by actively querying the Grafana Cloud Metrics, Traces, and Logs databases and integrating with the Grafana platform components like Dashboarding and Alerting. The Asserts services are multi-tenant, highly available and process and store data in Graph and Time Series DB. We will soon be rolling it out to thousands of self-service customers, and its critical that as we grow, we improve our performance, increase our reliability, and do it all more efficiently.

Were looking for a senior or staff level engineer with a strong distributed systems background to both build and lead the Asserts backend initiatives.   

As a company we are remote-first and global, we embrace people of different experiences and backgrounds to build diverse teams where every person brings a unique perspective to the software. 

What will you be doing?

• Drive technical and business strategy in the Observability department. 

• Influence the product roadmap. Drive innovations from ideation to customer adoption.

• Drive system design. Create design documents, collaborate within and across teams. 

• Work with your team to deliver new features, then use the results to iterate and improve.

• Build and operate critical systems. Own their reliability, performance, and availability.

• Participate in on-call rotations. 

• Mentor and support other team members.

• Strive to become a subject matter expert for observability products and systems.

• Gain a deeper understanding of our cloud product, our customers and get to know the codebase of a large distributed system.

What are we looking for in you?

• You are a motivated self starter with a bias towards action.

• You are customer focused. We build everything with our users in mind. You have a passion for creating intuitive products that fit customers’ needs. 

• You have experience delivering projects from gathering requirements, brainstorming ideas all the way to shipping a product to the customer’s hands in a self-driven way.

• You have experience in building and deploying SaaS software on any one of the cloud providers like AWS, GCP or Azure. 

• You have strong experience in Java programming and have built and deployed Java based microservices.

• You have experience with Kubernetes.

• You have been responsible for operating production services and organizing/participating in on-call rotations.

• You actively mentor other team members, identifying areas for focus and improvement.

• You like to share your knowledge by creating blog posts, giving tech talks at meetups and conferences.

• You’re curious and enjoy learning new programming languages and frameworks, setting up examples, and figuring out how things work.

Nice to haves:

• Been a power user of Grafana and Prometheus in operational roles (including on-call for your team at a previous employer or just using these tools on hobby/homelab projects)

Why Join Us

• Be part of a remote-only, globally distributed team.

• Collaborate with passionate professionals across Grafana Labs.

• Participate in company-wide hack-a-thons that help drive the direction of real-world innovation.

• Attend cross-functional team off-sites

• Contribute to open-source projects and make a real impact.

• Enjoy a culture that values work-life balance, personal growth, and professional development.

About Grafana Labs

Grafana Labs is an open-source SaaS company specializing in observability and monitoring solutions. We help organizations make sense of their data through open, composable, and enterprise-ready monitoring and analytics tools. As a remote-only company, we embrace a global team culture that values collaboration, innovation, and personal growth.

Grafana operates in 32+ countries. We try to operate as one team and focus on global benefits which our whole team can enjoy. Inevitably there are some regional variations and we discuss the benefits offered in your country of residence through our interview process.