Software Engineer - Platform Productivity at Grafana Labs

Grafana Cloud moves millions of metrics, log lines, and traces per second from our customers environments into a highly available, low-latency stack that processes and stores these data, and serves them to dashboards and alerting tools. We aim to grow this to hundreds of millions per second, and its critical that as we grow, we improve our performance, increase our reliability, and, of course, do it efficiently and effectively.

The Internal Engineering Platform delivered by the Platform department provides application engineers with the tools, systems and Kubernetes clusters they need to build, deploy and run their workloads. Platform roles at Grafana Labs have an eye for engineers with a passion for performance and reliability, and who enjoy taking projects from conception to production. We organize ourselves into squads to allow focus on Cloud Infrastructure, Networking and Security; engineering Productivity; Capacity management, Client Administrative Tooling (CAT); and US Federal compliance.

Because we deploy production services, we have on-call rotations to ensure the health of the system. Everyone at Grafana Labs tries to incorporate and use our product line up into their day-to-day, so being on call is an important way to understand our system and how people use our products.

About the squad & the role:

We are hiring for the Platform Productivity squad. The squad is mostly responsible for helping our internal engineers release their software onto our infrastructure, in secure and measurable ways. They lead automation of the release processes (anywhere from CI/CD to bootstrapping) and help our internal engineering teams get on board with them using ‘golden path’ techniques, but also helping edge cases and making sure that teams can get the most out of our tools. 

At the end of the day, we’re the Platform Team for the teams that are building some of the most cherished observability tools– from Grafana, Mimir and Loki, to Tempo.

Who Are We Looking For?

You enjoy working with engineers, as well as with the management structures that are there to support you and enable you and your team to do your very best. 

You are comfortable working in a remote-first company; communication is key. For us, working together means being collaborative, friendly, kind, and respectful. We operate by consensus, you can contribute to a discussion but then commit to the team decision.

You are eager to learn and grow. There is a lot of room for growth and development, and the team has quite a lot of knowledge to share for those who are wanting to learn. 

You approach development holistically. The team owns the full life cycle of our code; from writing design docs, looking at developer feedback, testing and deployment, all the way through to decommissioning. We appreciate engineers who enjoy looking at the big picture, and also notice the details of the brush strokes. Youre a flexible software engineer. In a typical day, we might spend time responding to incidents, integrating existing systems, or designing and implementing our own systems. While our primary language is Go, we value flexibility and well choose the best tool for the job, even if that is a shell script.

You have experience with operating your code. Since a lot of operators and developers use our software, having some grounding in both of these spaces really helps us with building better platforms for our users. Kubernetes and Docker are what we use, mostly. We also work with a lot of Infrastructure as Code, so be sure to check out grafana/tanka.

Responsibilities

The squad is responsible for setting its own roadmap, and as a part of the team youll have a part to play in that process. Youll help us maintain, improve and extend what we already have. Youll be involved in choosing what we focus on next and, just as importantly, when and how to gracefully sunset systems which are no longer needed.  Your responsibilities will also include helping the team to design, compare, and choose appropriate solutions for (at least some) of the following things:

• Development and maintenance of our Internal Engineering Platform (IEP)

• CI/CD platform management and development

• Build, release and deployment automation 

• Application configuration management tooling

• “Up to date” software automation

• Artefact management

• Working with diverse internal teams, from application development to security, to support implementation of their requirements 

• Being part of an on-call rotation to support Platform tooling 

Some skills/experience we like to see:

• Engineering/software development experience within a Platform group delivering services to internal engineering teams

• Experience working in a cloud environment

• Infrastructure as Code with Terraform/Crossplane.

• Familiarity with Kubernetes administration - very cool if experience with Tanka.

• Experience/Interest in implementing, integrating, and maintaining observability systems and processes

A few of our upcoming projects:

• Development of our next gen CI/CD systems

• Development of productivity tooling/automation in our IEP portal (Backstage)

• CI/CD observability

• Development of security automation as part of our CI/CD systems