Measure Things That Really Matter

As organisations seek to adapt and improve, so they look to steer the direction and demonstrate success. But what should they be measuring, and what are the pitfalls of getting the measures wrong? As organisations transform to Enterprise Product, it is essential that measurement KPIs align with the aims of the approach if success is to be realised and sustained.

Measures directly drive behaviour; they are a vital and incredibly powerful way to chart boundaries or track the progress of things over time, but care must be taken to ensure they are used correctly and not gamed. Incorrect use (for example, justifying decisions already made), will drive damaging behaviour – for individuals, teams, leadership, and the organisation overall.

The first, and most obvious problem is measuring the wrong things. The problem is two-fold, in that focus is then diverted to the wrong areas (driving incorrect behaviour) and true problems are overlooked. Of course, this is frustrating for those under the microscope who are under pressure to adapt but are relatively powerless to positively affect successful outcomes.

For some time, we have gotten into the habit of measuring the wrong things to fit a narrative, rather than what is really there. It’s easy to see why this occurred; the way traditional funding and management are structured, as well as bureaucratic requirements, means that proving often takes precedence over improving.

The purpose of this paper is to discuss the rights and wrongs of measurement and to look at what really matters.

Metrics and Vectors

There are two main types of measures in business – Metrics and Vectors. The former is often over-used and affiliated with systems and structures, while the latter is often under-utilised and more closely associated with culture and leadership.

Metrics – Metrics are a point-in-time snapshot for specific things. They should not apply outside their These are fine-resolution and cannot be easily applied over time (e.g. achievement of a milestone).
Vectors – Vectors are the “movement over time” of things and are akin to the measurement of the journey. This is much more useful overall and applies across contexts. These are coarse resolution – “scanning the horizon” – and cannot apply in a single instance easily. These are much more important to show sustainable and ongoing success (e.g. revenue growth).

KPIs, which measure a single metric at a point in time, can be dangerous. For example, hitting a specific target representing a significant increase in year-on-year revenues can easily hide diminishing forward-looking business, which will lead to poor performance in the future.

The revenue Vector will ideally take into account revenue over time, order book, pipeline etc thereby providing insight into current and future performance.

Typically, measures are empirical and objective and therefore measure “hard” numbers. However, measurement through feedback can be used to provide powerful subjective measurement.

Feedback

Feedback mechanisms can (and should) be used as a measurement. Feedback is often intangible, and may be as strong as direct loud voices, or as subtle as weak signals which require specific approaches to detect. It is often informally delivered or collected in many cases.

Feedback is a directional pointer to measure the direction travelled, rather than measuring the distance along that travel (weathervane approach). This can be best collected via sensemaking (surveys should be avoided, as feedback must be taken in its entirety and not biased or cherry- picked). Feedback is one of the most powerful mechanisms over time to determine the true state/context, and real direction of an organisation.

Ignoring or avoiding uncomfortable or weak-signal feedback is nearly always catastrophic for the organisation over time; efforts must be made to embrace what may be seen as “unpopular opinions” as well as glowing reports and disassociate any egotistical and emotional attachment.

Things We Should And Should Not Measure

When thinking about measurement, it’s worth bearing the following in mind:

Measure things that genuinely reflect reality and progress and are as objective as possible
(for example, investment, increasing revenue, decreasing cost). These are not usually comfortable for individuals, as they are often difficult to attribute and harshly revealing.
Know that ALL measurements will be gamed, so pick measurements for which gaming is an advantage (for example, Throughput)
Not only measure the right things, but measure at the right time. Think about the vector –
the direction of travel – rather than simply the point in time.
Measuring things that reflect reality and progress, may not feel comfortable, particularly if progress is not positive. Individuals like to succeed, but empirical evidence suggesting otherwise may not have a positive impact on morale.

Things We Should Not Measure

Similarly, the following should be avoided:

Metrics which apply only in a very specific context, especially where they are markers for
interest and not true performance metrics.
Inputs over outcomes. Over the last few years, professional organisations have moved from requiring a number of hours CPD per annum, allowing individuals to design their learning journey and the outcomes achieved.
Measures specifically designed to prove things (“backside-covering”)
“Hours worked” as a measure of value or output – this has no correlation to quality or quantity of work in complex spaces. Of course, hours worked are relevant in that it is typically part of their employment contract.

Example Measures

When transforming to Enterprise product, there are three areas of measurement that should be used to sustain the change and ensure continual improvement. These are:

Success at team level: Is each team being established effectively, is the coaching support effective, and are they delivering improved value?
Success at programme level: Is the programme delivering at the pace and quality intended?
Success at organisational level: Is the organisation implementing an effective operating model, is it having the benefits intended?

The three are intrinsically linked – team-level success rolls up to organisational success, and success of the programme is evidenced by the other two. Specific metrics in each area are discussed below:

Team Success

The measurement of team success is critical, as this is the area which will determine overall success and typically, this is the area which individuals will look to game. Ideal measures include:

Value delivered: Ideally, this should be linked to strategic KPIs.
Flow efficiency: This provides a good indication of the true “velocity” of the team. It is evaluated by looking at the waste in the system.
Deployment frequency: Taking into account that this may be constrained by external platforms and technology factors.
Quality: Churn on delivered stories, number of defects.
Service metrics: Trends on availability, average fix times, incident numbers etc.
Empowerment: The proportion of the team’s effort that is spent on ‘discretionary’ work (i.e., work they have put into the backlog) vs. ‘imposed’ work – e.g. fixes, other project support etc.
Colleague engagement: This is a strong leading indicator of success at team level as an engaged team is typically a productive one. A short “10 question survey” is undertaken by the colleagues in a product team before the mobilisation activity, and then at regular intervals thereafter.

Organisational Success

In many regards, organisational success is a summation of the success or otherwise of specific teams. So where appropriate. aggregating team metrics can provide an indication of broader success. However, this must be done carefully as team metrics are typically context-specific and should not be applied or compared outside of the team. That said, agreeing on a standard form for colleague engagement questions would enable an organisation-wide view, for example.

Value delivered: Ultimately, this is the primary purpose of moving to a Product model. Establishing, or aligning to, a set of strategic KPIs, and demonstrating value delivered against these, is the most effective way of measuring success. These KPIs can span customer satisfaction, revenue, cost to serve, risk reduction and other business metrics.
Colleague engagement is another candidate KPI and can be assessed through an aggregation of colleague engagement scores from those teams that have transitioned to product, acting as a strong leading indicator at the team level.
Reduction of dependencies: Aggregating individual team’s metrics for the proportion of the team’s effort that is spent on ‘discretionary’ work vs. ‘imposed’ work will give a strong indication of the extent to which there are dependencies between teams to ‘get things done’. Over time, one would expect to see a reduction in this figure, as those dependencies are steadily removed through evolution of the model.
Adoption by other areas: Increasing requests from other parts of the business (beyond, say Group IT) to explore the use of product-oriented approaches will indicate that the model is having positive effects.

Over the longer term, the following metrics could be considered:

Staff retention: increased engagement and empowerment should result in reduced staff turnover.
Permanent to temporary headcount ratio: Fewer dependencies, more efficient, iterative and predictable delivery of change should reduce the need for temporary resources to deal with spikes in change demand and large programmes of work.
Spans of control: The creation of a practice model, and empowered, self-managing teams, reduces the need for management overhead of individuals. This means that those in management positions can have greater spans of control and still provide the management and leadership necessary. Ultimately this makes the organisation leaner, with more people focused on ‘doing’ rather than managing.

Transitional Success

The success of the transformation to Enterprise Product is ultimately determined (and reflected) by the successful transition of teams, and the overall organisational, operating model establishment. The metrics outlined in the sections above are effective proxies for measuring programme success. However, we propose two additional measures:

Transition rate (delivery against the plan): Are teams transitioning to Product-oriented ways of working as outlined in the plan (once it is agreed). Although there are many factors that can influence this, of which many are out of the Programme’s control, this can still be seen as an effective measure.
Colleague survey scores: Regularly sampling the view of all colleagues in Group IT, including those not in teams transitioning to product is a good indicator for the success of the programme.

Warnings

Measurements, if used inappropriately (e.g. as targets), out of context (e.g. measuring a team as individuals), or at the wrong time (e.g. before realistic results can be achieved), can be worse than damaging.

They can not only skew data and allow manipulation of results to misrepresent reality, set incorrect expectations, hide issues, and damage outcomes over time, but can drive pathological behaviour, create a hidden space for gaming and abuse, and cause suffering and/or ultimately exodus for the people involved.

Measurements showing failure are extremely valuable in achieving better future successful outcomes, and thus information should never be hidden or cherry-picked.

Incorrectly used measurements are often a major contributor to a persistently toxic environment, and a downward trend of the organisation at large.

Goodhart’s Law

Inappropriate, management-set measurements are at high risk of falling afoul of the economic principle of Goodhart’s Law:

“when a measure becomes a target, it ceases to be a good measure”

This arises when an indicator or signifier of something (i.e. a measurement) becomes a target for direct policy action. If action is then taken on what is only an indicator, the informational value is lost and the actions will be as inappropriate as the indicator will now be inaccurate.

This induces gaming behaviour, cherry-picking, and misdirection, and remains true from individual or team level all the way up to executive; the difference here is that at high levels it is problematic for the long-term viability of the company.

Many traditional measures are victims by design – why KPIs and individual measurements are not representative of the truth; why velocity should never be used comparatively or incompletely (velocity is not merely composed of “speed”).

Goodhart’s Law must be kept in mind to encourage us to use measurements both short-term (metrics, points in time, limited use, indicator of direction) and long-term (vectors, movement over time, deeper use, systemic and sustainable coherence) as feedback mechanisms.

Summary

If we are to succeed, we must measure what is correct, and these measurements must be set by the right people with the right decision-making authority, in the right language, to provide information on what is really there, not to give us something to aim for.

It is more realistic to designate a timebox based on real-world constraints and then collect measurements within those constraints, and then measure movement on tight feedback loops to allow us to flex the workflow, than it is to predefine a metric set of what we want and then force adherence, when this may not be in line with reality.

The measurements (and the subsequent rewards or legitimacy from those measurements) will be a core factor in helping define a culture that not only attracts people, but helps them do what they do better, and optimise workflow according to the system.

Get in touch with our team to learn more about our services, explore partnership opportunities or discuss how we can help with your challenges.