Engineering Metrics Glossary

Insights that drive action.

Understand every Pinpoint performance metric, why it matters, and how it's derived, so you have a clear view of the actions that boost software engineering performance.

Average Days Late

Average days late for completed issues with due dates.Expand

Insights

Average days late for completed issues with due dates.

Why this matters

Average Days Late reflects how well a team is meeting its targeted deadlines. It is a measure of both planning and delivery effectiveness.

How to use

If Average Days Late is higher than target, it could be a sign that there are bottlenecks or inefficiencies that warrant investigation. Examine Throughput to determine if this is the case.

High Average Days Late could also signal that the team is overly-optimistic in the planning process or is not building slack into the schedule for the inevitable complication or addition of unplanned work. Consider using the historical Throughput as a yardstick to sanity check the level of work teams sign up for as they plan. Scrutinizing Sprint Health will reveal if scope control and/or excessive unplanned work is the culprit.

Insights

Backlog Change shows the percent change in the number of open issues for the period specified. A negative figure means more issues were closed than opened; a positive figure means more issues opened than closed.

Why this matters

An increase in backlog is often a natural outcome of the planning process, as new work is defined and captured. However, teams whose backlogs only ever increase may suggest too much feature ideation against too little execution.

How to use

Keep an eye on any spikes in Backlog Change that occur independent of planning cycles, which could signal quality issues or a resource imbalance.

Steady change in backlogs over time (whether growing or shrinking) may also signal a need for resource adjustments. Growing backlogs may indicate additional team members are needed. A shrinking backlog may signal that a team has resource capacity that could be allocated elsewhere.

Insights

We use the 80/20 rule as a means for understanding Workload Balance—that is, how evenly work is spread across a team. Specifically, Workload Balance evaluates what percent of the available team capacity is involved in delivering 80 percent of the work.

To calculate a person’s work capacity, we use Issue Days. Issue Days is essentially a proxy for workload—importantly, one that’s based on actuals, not estimates. With Issue Days for each person, we can see how workload is distributed across a team, or a department, or even the engineering organization as a whole.

This is where the 80/20 rule comes in. Let’s assume a team with ten people. We might see that for a given period, six of the ten team members, 60 percent, accounted for 80 percent of the Issue Days in the period.

To convert this into Workload Balance, we divide that 60 percent by 0.8. Why? Because we’re using the 80/20 rule, and because we want an even distribution of work to score as 100 percent. If eight of ten people on a team are responsible for 80 percent of its work—i.e., a perfect distribution according to the 80/20 rule—then we want to reflect their Workload Balance as 100 percent ((8÷10)÷0.8), not 80 percent.

In the example above, this team has a Workload Balance of 75 percent: (6÷10)÷0.8. Not bad, but still with room for a better balance of work across the team.

Why this matters

Knowing and being able to show a team’s Workload Balance makes it possible to:

  • Find underutilized capacity
  • Anticipate overwork / burnout risk
  • Quantify the case for adding more people
Insights

To calculate this, we take the total code additions plus total code deletions and divide by the number of commits within the period selected.

Why this matters

Generally speaking, the fewer the changes per commits, the better. With fine-grained changes, root cause analysis and fixes are easier in the event of any problem. You have a more precise view into what changed, when. There is also much higher alignment between the description of the commit and everything it actually includes.

How to use

Guidelines of course can help encourage developers to favor smaller, more frequent commits, and to only include related changes in the same commit. But guidelines only go so far. This is where Pinpoint comes in. When people see that a best practice is more than just hot air—that the practice is actually being measured, and used to inform performance evaluation—the adoption rate is... remarkable.

Insights

Code Ownership evaluates the total lines of existing code a person has contributed across all repositories.

Why this matters

In addition to showing one important part of developer contribution, Code Ownership also helps to show a developer’s code base familiarity, and domain expertise.

How to use

Note that Code Ownership reflects existing, not total, lines of code contributed by a person. That is, as a person’s code is reduced or rewritten by others, those lines are removed from his/her Code Ownership total.

Also note that the Code Ownership signal ignores 3rd party modules and libraries. These are not counted toward a person’s Code Ownership.

Insights

Where Workload Balance evaluates the balance in issue work, Contribution Balance performs a similar function for coding work. The two signals share the same 80/20 logic, except that Contribution Balance uses code changes (additions and deletions) instead of Issue Days to evaluate workload. For more, see Workload Balance.

Insights

Review Rate is calculated by dividing the total amount of merged code changes reviewed by the total amount of all code changes merged.

Why this matters

The pull request and review process exists for a reason: to vet code quality and functionality before updating master. Review Rate indicates the rigor with which people and teams are abiding the PR process.

Insights

To calculate this:

  • For teams, we measure the time in days from an issue first being set to in progress, to the time it is completed. We exclude from this calculation any issues that are never set to in progress (for example, that go straight from open to closed), as these issues presumably didn’t engage development. Cycle Time also makes allowance for issues that are closed and subsequently re-opened. In these instances, the latest closed date is used.
  • For people, we sum of the total days in progress for all Completed Issues that person worked in the time period selected. This figure is then divided by the total Completed Issues that person worked for the same period. So, a person who works on five Completed Issues in a given period, and whose total in progress duration during that period is 50 days, would have a Cycle Time of 10 days (50 / 5) for the period.
Why this matters

Cycle Time is one of the key efficiency signals for software development. It measures the fundamental capability of a software team — namely, the timely completion of issues. Shorter Cycle Time indicates an optimized development process and faster time to market. Longer (or lengthening) Cycle Time may signal an understaffed team, or inefficiencies in the team or process.

Cycle Time also informs Forecast Work Months, which helps to drive more accurate, more predictable issue completion dates.

How to use

A word of caution in using Cycle Time to compare teams. If one team is building with the benefit of a loosely-coupled microservices architecture, while another works on a legacy application with a monolithic code base, there are bound to be differences in their Cycle Times. The objective should be for each team to improve its score over time.

For teams looking to improve Cycle Time, start by examining the workflow. The detailed view for Cycle Time shows the distribution of days across time to start (not included in the Cycle Time calculation), in progress duration, and verification duration. This will help identify bottlenecks like wait time.

It also helps to think smaller. Are teams composing user stories in the smallest possible increments of valuable functionality? This allows for quicker iterations at lower risk, since the amount of change is contained. Are the teams themselves as small as possible? Smaller, self-contained teams usually means faster cycles, higher quality and less rework. And as always, look to automate as much as possible — regression tests, provisioning, and the like.

Insights
Why this matters

Defect Density helps to reveal the quality of code owned by a team. It may also be used as a proxy for technical debt, helping to determine the relative cost to maintaining the code as-is.

How to use

Code bases with higher Defect Density may be in line for refactoring. The trend in Defect Density over time, as shown on the detail view, will help decide whether a code base is reaching a point where the costs (and risks) to maintain are outweighed by the costs to refactor.

Note: Some teams may not have an associated code repository. In these instances, Defect Density is excluded from the performance summary calculation.

Insights
Why this matters

Together with Delivered vs. Final Plan, Delivered vs. Initial Plan gives teams a view of how effectively they do what they say they’re going to do.

Delivered vs. Initial Plan is one of the variables in determining Sprint Health.

Insights
Why this matters

Defect Ratio tells you whether you’re creating more bugs than you’re closing. If so, there’s a quality issue somewhere in the pipeline.

How to use

Defect Ratio helps surface how good is the output of your engineering pipeline. It serves as a natural check and balance against other pipeline signals. For example, if your Backlog Change is negative (meaning, you’re burning through your backlog), your Throughput is holding steady or improving, but your Defect Ratio is rising, that indicates your speed is impacting quality.

Insights

Where Rework Rate measures reversion in issue state, Review Rework Rate looks at how often code requires further change following code review.

Why this matters

Review Rework Rate helps you to see how efficiently developers are working code to a finished state, according to the standards of the team or organization.

How to use

A high Review Rework Rate may signal that a developer is contending with code that’s higher in complexity, or whose function is poorly defined, or both. It may also be a symptom of speed over thoroughness. In either case, a high Review Rework Rate suggests a person in need of help, maybe in the form of further training or coaching.

Insights

Let’s say a standard issue flow is open -> in development -> in verification -> closed. If an issue proceeds to in verification but subsequently is returned to in development—that is, it reverts to a prior issue state—we flag this as rework. We calculate Rework Rate by tallying all the Completed Issues that had at least one reversion. We then divide this total by Completed Issues for the selected time period.

At the individual level, this means that whenever an issue reverts, the Rework Rate for the person who worked the issue is updated. If multiple people have worked the issue, each of their Rework Rates are updated.

The number of times an issue may revert isn’t relevant to the signal calculation. That is, even if an issue were to revert multiple times to a prior state, for purposes of Rework Rate it is counted only once.

Why this matters

Rework Rate helps you to see how efficiently issues move through your workflow.

How to use

Teams or people showing higher Rework Rates may be contending with issues that are higher in complexity, or poorly defined, or both. A higher Rework Rate may also be a symptom of haste making waste. Check the team’s or person’s Cycle Time. Poor test coverage and/or low test automation also affect Rework Rate.

Insights

Sprint Health is a composite signal for teams to track and improve their sprint planning and efficiency.

To derive Sprint Health, we examine a sprint’s work at three points: the initial plan, the final plan and what is actually delivered. Looking at these, we calculate:

  • Plan Survival Rate: the percent of issues from the initial plan that remain in the final plan
  • Plan Growth Rate: the percent change in the total number of issues from the initial plan to the final plan
  • Delivered vs Initial Plan: the percent of issues in the initial plan that are completed in the sprint
  • Delivered vs Final Plan: the percent of issues in the final plan that are completed in the sprint
  • Unplanned Work: the percent of issues completed in the sprint that were not in the initial plan

These five measures are averaged to produce the Sprint Health value for all sprints completed during the selected time period.

Why this matters

A team’s ability to accurately plan and execute sprints is the bedrock of effective, predictable delivery. Sprint Health is a way for teams to keep themselves honest about their planning maturity and the ability to hit their targets.

How to use

Sprint Health can be a revealing diagnostic for optimizing a team’s sprint mechanics or for troubleshooting pipeline signals like Average Days Late. A lower Sprint Health score can indicate a team is too optimistic in its planning, isn't allowing for unplanned work, and/or is suffering from scope creep. Examining the Sprint Health funnel graph (the distribution view on the Sprint Health page) should point to which of these factors are at play.

A low score may also be a sign the team not striving to deliver a specific set of scope in their sprints, but instead is using the sprint plan as a rolling backlog, where unfinished work gets pushed from one sprint to the next. Depending on the type of work the team performs, this may be fine—for example, if a team’s primary focus is addressing support tickets. Teams delivering functionality will likely want to get more deliberate in their planning and the ability to predictably hit their targets, so they can improve their results over time.

Insights

Sprint Volatility measures the issue churn within sprints: that is, issues being added to or removed from a sprint after the sprint has begun. The more issues churning in or out of a sprint after start, the higher the Sprint Volatility.

Why this matters

A high Sprint Volatility may signal confusion over the business requirements. When delivery teams and stakeholders are misaligned, the result is often an unsettled, changing list of issues. More specifically, it may be a sign that stakeholders aren’t conveying priorities clearly or consistently.

How to use

Signal-wise, Sprint Volatility exists on a kind of continuum with Delivered vs. Planned. The former helps to gauge how sprint teams are engaging with stakeholders and managing scope; the latter shows how efficiently sprints are planned and executed once the scope is set.

Insights
Why this matters

There are several ways Traceability is important:

  • Understand the real distribution of work: When it comes to understanding the kinds of work that are consuming people, it’s not what you think is happening or what people say is happening—it’s what’s actually being written. Truth in code. With Traceability, Pinpoint can show you how much of the coding effort is being spent on new features, how much is being spent fixing bugs, and how much is spent doing maintenance—and you can see these splits by person, team, or for the company as a whole.
  • Quantify technical debt: By analyzing the connections between code and issues, we can also show you the phase of your different repositories—which are used mostly for maintenance work, versus those used for new features. It also uncovers potential hotspots for technical debt, by showing which files within a repo have a high percentage of bugs, as well as the people and teams whose code tends to produce the most bugs. These data are a starting point to putting real numbers to your technical debt.
  • Find the experts: For larger teams, especially those taking on new initiatives, it’s useful to know at a glance which people in the organization have skills most suited to the work at hand. Refactoring an app? Maybe you want to see who’s squashed the most bugs in JavaScript. Pinpoint uses traceability to show who’s done what kind of work, and in what language(s), making it easy to identify the right pros for the job.
  • Measure your own contributions: Pinpoint’s analysis of the code-issue connection means a developer can see, for example, how her code contributed to an improved user experience or better conversion rate.