Request access

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Scorecards in Platform Engineering

John Demian
John Demian
Head of Product Marketing
Rely.io
John Demian
May 17, 2024
6
 min read
Drive alignment, prioritization, and quality with Scorecards

Scorecards are the unsung heroes that help engineering leaders take their teams to new heights. Their appeal lies in the capacity to convert complex objectives into clear, measurable outcomes which help align the entire team around a common goal.

What is a scorecard?

A scorecard functions as a critical tool for gauging and managing the performance of a service or an app within an organization. These scorecards measure a variety of important metrics and indicators like DORA metrics, observability maturity, production readiness, and operational maturity among others.

Take the DORA metric for example. It provides a holistic view of your DevOps team's performance by leveraging critical metrics like deployment frequency, number of failures, and change failure rates enabling teams to ensure that they are meeting the operational benchmarks and progressing steadily towards defined goals.

The role of scorecards in Engineering Teams

The DevOps philosophy is about more than just tools and technology. It’s a culture that fosters collaboration, automation, and continuous improvement. Scorecards are powerful tools for DevOps and Engineering teams that use them to gamify the adoption of standards such as production readiness, DORA metrics, operational maturity, and observability coverage among others. 

By integrating scorecards in your engineering day-to-day you can drive awareness to individual contributors around the priorities and standards of the company. This will add visibility down to the individual contributors around specific actionable insights which drive progress and operational excellence.

Implementing Scorecards 

Your typical implementation of scorecards is not a one-time thing. It’s not a “build-it-and-forget-it” type process and requires a cycle that includes setting goals, measuring progress, and tweaking those goals.

Your typical process would look like this:

  1. Set the goals: Start with defining what you want to achieve and track. It could be anything from reducing deployment time, improving code quality,  mean time to resolve, etc.
  2. Define Metrics: Decide on what metrics you will use to measure and get them into the software catalog whether via plugin data, automation, or public API.
  3. Measure Progress. Keep track of these metrics over time.
  4. Learn from Results: Analyze the results to identify areas of improvement.
  5. Update Goals: Based on your learnings, you’ll update these goals to better fit your end goals.

The idea is not to create a rigid system that puts pressure on your team but rather to implement scorecards that will promote a culture of continuous learning and improvement.

Scorecard examples 

To effectively integrate scorecards into your internal developer portal, consider the following common scorecards that are widely recognized for their ability to promote and measure engineering excellence:

Software Development Lifecycle Efficiency Scorecards

  • Time to Prod: How long on average does a change request seat wait for a review? And how long does that review take on average? And after approval, how long does it take to be deployed to production? Keeping Track of these durations allows you to identify the bottlenecks in your SDLC process.
  • Sprint Burndown: What’s the amount of work left undone by the end of each sprint?
  • Alert and Incident Count: How many inaccuracies are being inserted into your systems as you develop? 

Production Readiness Scorecards

  • Service Stability: How frequently does the service experience downtime or performance issues?
  • Monitoring: Are the right metrics being tracked and monitored? Are their values healthy? How effective are the alerting mechanisms in place for identifying and responding to issues?
  • Documentation: Is there comprehensive and up-to-date documentation for the service?
  • Disaster Recovery: Are there clear strategies and procedures for disaster recovery? Are your escalation policies up to date? Is an on-call rotation properly established?
  • Vulnerabilities and Versioning: Has any critical vulnerability been identified in one of the packages you’re using? Is the service compliant with the SDK versioning that your services are using?

DORA Metrics Scorecards

  • Deployment Frequency: How often does your team deploy code to production?
  • Lead Time for Changes: How long does it take to go from code commit to code successfully running in production?
  • Time to Restore Service: How quickly can the team restore a service following a downtime event?
  • Change Failure Rate: What percentage of changes to production fail?

Operational Maturity Scorecards

  • Configuration Management: How effectively is the service's configuration managed and applied across environments?
  • Security Practices: Are best practices in place for ensuring the security of the service?
  • Peer Review: How rigorously are code and infrastructure changes reviewed before deployment?

Observability Coverage Scorecards

  • Logging: Does the service produce sufficient logs for debugging and understanding user behavior?
  • Tracing: Are there effective tracing mechanisms in place to diagnose problems?
  • Metrics: Are key performance indicators and service level indicators being measured and adhered to?

These scorecards are not exhaustive, but they can serve as a starting point for engineering teams looking to measure and enhance their engineering best practices. 

Tailor these scorecards to your team's unique context and objectives, periodically review them, and adapt them as necessary to maintain a trajectory toward engineering excellence.

How to create a scorecard for your internal developer portal

Developing a scorecard for use within an internal developer portal begins with a blueprint that outlines the key activities in your engineering processes. 

Following the 5 processes I’ve mentioned above let’s run an example of setting up DORA metrics in Rely.io and I’ll start with the first one.

1) Set the goal: In my example, I’d like to create a scorecard to track my DORA metrics with the goal to better understand how to improve efficiency and productivity within my team.

2) Define the Metrics: Next we’ll select the most pertinent metrics which will help me measure the indicators that correlate directly to my objectives. In this case it would be things like the Deployment Frequency, Lead Time for Changes, Mean Time to Recovery, and Change Failure Rate.

3) Track Progress: Since I want to easily show the progress I’ll need to be able to display them in a way that’s easy to understand. To make it easier for me to understand the performance of different teams and services I’ll split them into 3 tiers: Bronze, Silver, and Gold.  

4) Analyzing Results: Because of the way I’ve created my scorecards, I’ll have access to easy-to-understand metrics that are updated in real time. This way I can take these insights into meaningful action plans that will directly impact my team's performance.

5) Update Goals As the team evolves and industry standards change, so will my requirements so I’ll have to go and revisit the metrics that go into my Scorecards and adjust the objectives so they remain both relevant and challenging.

Create your first Scorecard right now!

With Rely.io, you'll use data-driven insights, transparent metrics, and AI capabilities, to ensure your team's objectives stay aligned with your goal of creating a culture of engineering and operational excellence. 

Start using Rely.io to create and integrate scorecards into your very own internal developer portal today! 

Need advice with a scorecard you want to set up? Get in touch! We’re here to help.

John Demian
John Demian
Head of Product Marketing
Rely.io
John Demian
On this page
Contributors
Previous post
There is no previous post
Back to all posts
Next post
There is no next post
Back to all posts
Our blog
See related articles
What are Day 1 and Day 2 Operations for Platform Engineers
What are Day 1 and Day 2 Operations for Platform Engineers
For technical leaders, platform and DevOps engineers, mastering both day 1 and day 2 operations is crucial for ensuring smooth operations.
John Demian
John Demian
July 12, 2024
9
 min
How to Implement Developer Self-Service Successfully
How to Implement Developer Self-Service Successfully
Developer self-service empowers developers to build and manage their services and resources independently from DevOps, accelerating development cycles without any compromises on quality or standards.
John Demian
John Demian
June 21, 2024
9
 min
The ultimage guide to Dora metrics
The ultimate guide to DORA metrics
As platform engineers, DevOps engineers, and technical leaders, embracing DORA metrics can propel your teams towards operational excellence and streamlined processes.
John Demian
John Demian
June 14, 2024
7
 min