Developing indicators can be quite challenging, and all too often M&E experts are not involved in the process. As a result, M&E staff often have to work with difficult indicators. Based on our experience as M&E consultants, in this article we will give you some examples of difficult indicators and suggestions on how to report and present their progress.
The ideal indicator
Ideally, an indicator should cover only one aspect. Then it is easy to show progress on a graph because there is a clear and unidimensional trajectory of progress. For example, if the indicator is "Three national bills for safe working conditions in the manufacturing sector are passed", then once one bill is passed, the indicator is at 33% progress, two bills are at 66% progress, and three bills are at 100% progress. This is both easy to report and easy to communicate visually in a chart:
In practice, however, we often have to deal with indicators that describe more than one aspect, i.e. there is an explicit or implicit 'and'. Where the context allows, it is usually best to simply report all the details. But sometimes it may be necessary to summarise progress in a single figure. How to do this? Here are some examples and suggestions.
Cumulative aspects
An example of an indicator with cumulative aspects would be "A business process map is developed for 4 services using a structured analysis mechanism".
The product we count is a completed business process map. We want four of them. But there is a second condition, namely that they are developed using a structured analysis mechanism. First of all, we would have to define in more detail what that analysis mechanism is. But then it is not really an additional aspect, but a necessary implicit condition, meaning that we can only count a business process map IF it has been developed using this specified mechanism. Any business process map developed in any other way does not count. So the progress measure is still one-dimensional:
"One completed business process map" AND "It was developed using the structured analysis mechanism" = 25% progress.
Additive aspects
An example of an indicator with additive aspects could be “80% of the beneficiaries report that the improved cookstove uses less firewood and emits less smoke”.
In this case, we face the challenge that it is very possible for a beneficiary in our survey to report that the stove does emit less smoke but still uses the same amount of firewood as the regular stove. We now have two options:
1. Only consider the conditions met if a beneficiary reports both improvements. So any beneficiary who reports only one of the two improvements will not be counted for this indicator (progress option 1 in the graph below).
2. Report the average. Assume 50% of beneficiaries report that the cookstove uses less firewood, but 90% report that it emits less smoke. Then the overall progress is (50% + 90%) / 2 = 70% (progress option 2).
Multiple target values
An indicator with multiple target values might look like this: “Of the 1000 young people who have participated in the vocational training, 50% are in a qualification-related, employed or self-employed job within 8 months after the end of the training".
The problem here is that we have two target values: On the one hand we have the absolute number of planned beneficiaries (1000) and on the other hand, the share of them that will find employment after completing the job trainings (50%). It is therefore quite possible that we will achieve a 50% employment rate for the beneficiaries but only reach a total of 600 young people instead of 1000. So how do we report and visualise progress for this indicator?
Either we report both figures, or
- If we are forced to choose a main progress number, we could ask the donor which is more important, the absolute number of young people reached with the vocational training or the percentage of young people in employment. If the percentage of people in employment is more relevant, we could, for example, report that 55% found employment and then add a note that we did not reach the total number of people expected (see progress option 1 in the figure below).
- We can merge the progress values of both targets into one by taking the average. For example,
- We only reached 600 instead of 1000 (600 / 1000 = 60% progress)
- Their employment rate was 55% (55% / 50% *100 = 110% progress)
This gives us an average progress of (110%+60%)/2 = 85% (see progress option 2 below).
Sub-groups
Example: “The satisfaction of users (of whom 20% are women and 10% are from vulnerable groups) of public services has increased by 30%.”
Assuming we have clearly defined how exactly to measure user satisfaction and how to calculate the 30% improvement, we are still faced with the challenge of having one main figure to report (of all users) and the two figures for the two sub-groups.
Now we can either:
1. Report only the main number (progress option 1).
2. Take the average of the three figures.
The tricky part is: how exactly? Just divide the three progress numbers by three? This would mean that the main number and the subgroups are equally important. However, the main figure is probably more important than the two sub-groups. So we could introduce a weighting factor: The main figure would weigh twice as much as each of the sub-groups. So the progress figure would be calculated as follows: ((progress all users * 2) + progress women + progress vulnerable groups)/4 (see progress option 2 in the figure below).
Indicator sheets in WebMo
The standard version of our monitoring software WebMo offers various kinds of indicators. The baseline / target / current indicator is suitable for quantitative indicators. It allows you to collect data at different points in time and display the history over time. The data can be entered in a disaggregated way, divided into classes such as age groups or gender. The milestone indicator can be used for both quantitative and qualitative data. You can define any number of milestones on the way from baseline to target. At the output level, you can select activity indicators that measure the progress of the activities assigned to the indicator. For trickier indicators, there are really no limits with our customised version of WebMo. We have many years of experience in creating indicator sheets to meet our clients' needs, no matter how complex their indicators are.
