Indicator development can be quite a challenge and far too often, no M&E experts are involved in the process. Consequently, M&E staff often have to work with difficult indicators. Based on our experience as M&E consultants, in this article we will give you some examples of tricky indicators and make suggestions on how to report and present their progress.
The ideal indicator
Ideally, an indicator should cover only one aspect. In that case, it is easy to display its progress in a chart, because there is a clear and unidimensional progress trajectory. For example, if the indicator says "Three national bills for safe working conditions in the manufacturing sector are passed", then as soon as one bill is passed, the indicator is at 33% progress, two bills mean 66% progress and three bills will be 100% progress. This is both easy to report and to communicate visually in a chart:
In practice, however, we often have to deal with indicators that describe more than one aspect, meaning there is an explicit or hidden "and". If the context allows, it is usually best to simply report all the details. But sometimes it may be required to summarize the progress in one single figure. How to do this? Read some examples and suggestions below.
Cumulative aspects
An example for an indicator with cumulative aspects would be: “A business-process map is developed for 4 services using a structured analysis mechanism.”
The product we are counting is a finished business-process map. We aim for four of these. But there is a second condition, namely that they were developed using a structured analysis mechanism. First, we would need to define more in detail what that analysis mechanism entails. But then it is not really an additional aspect, but a necessary implicit pre-condition, meaning that we can only count a business-process map IF it has been developed using that specified mechanism. Any business-process map developed in another way does not count. So, the progress display is still unidimensional:
"One finished business-process map" AND "It was developed using the structured analysis mechanism" = 25% progress
Additive aspects
An example of an indicator with additive aspects could be: “80% of the beneficiaries report that the improved cookstove uses less firewood and emits less smoke”.
In this case, we face the challenge that it is very well possible that in our survey a beneficiary reports that the stove does emit less smoke, but still uses the same amount of firewood as the regular stove. Now we have two options:
- Only consider the conditions fulfilled if a beneficiary reported both improvements. So any beneficiary reporting only one of the two improvements will not be counted for this indicator (progress option 1 in the graphic below).
- Report the average. Assume 50% of beneficiaries report that the cookstove uses less firewood, but 90% report that it emits less smoke. Then the overall progress is (50% + 90%) / 2 = 70% (progress option 2).
Multiple target values
An indicator with multiple target values might look like this: “Of the 1000 young people who have participated in the job trainings, 50% are in a qualification-related, dependent or independent employment within 8 months after the end of the trainings.”
Here, the problem is that we have two target values: On the one hand we have the absolute number of planned beneficiaries (1000) and on the other hand, the share of them that will find employment after completing the job trainings (50%). So it is very well possible that we may achieve a 50% of employment for the beneficiaries, but overall only reach 600 instead of 1000 youngsters. How, then, do we report and visualize progress for this indicator? Either we report both numbers, or:
- If we are obliged to decide on one main progress number, we could ask the donor what the more important aspect is, the absolute number of young people reached with the job trainings or the share of them finding employment. If the share of employed people is more relevant, we can report for example that 55% found employment and then add a note that we failed to reach the anticipated total number of people (see progress option 1 in image below).
- We can merge the progress values of both targets into one by taking the average. E.g.
- We reached only 600 instead of 1000 (600 / 1000 = 60% progress)
- Their employment rate was 55% (55% / 50% *100 = 110% progress)
This gives us an average progress of (110%+60%)/2 = 85% (see progress option 2 below).
Sub-groups
Example: “The satisfaction of users (of whom 20% are women and 10% are from vulnerable groups) of public services has increased by 30%.”
Assuming that we have clearly defined how exactly user satisfaction is measured and how 30% improvement is calculated, we will still have the challenge of having one main figure to report (of all users), as well as the two numbers of the two sub-groups. Now we can either:
- Report the main figure only (progress option 1).
- Take the average of the three figures. The tricky question is: how exactly? Just dividing the three progress numbers by three? This would mean the main figure and the sub-groups are equally important. However, the main figure is probably more important than the two sub-groups. So we could introduce a weighing factor: The main figure would weigh twice as much as each of the subgroup. So the progress figure would be calculated like this: ((progress all users * 2) + progress women + progress vulnerable groups)/4 (see progress option 2 in image below).
Indicator sheets in WebMo
The standard version of our monitoring software WebMo offers various kinds of indicators. The baseline / target / current indicator is suited for quantitative indicators. It allows you to collect data at different points in time and display the history over time. The data can be entered in a disaggregated way, divided into classes such as age groups or gender. The milestone indicator can be used both for quantitative and qualitative data. You can define as many milestones as you want on the way from the baseline to the target. At the output level, can you select activity indicators that measure the progress of activities assigned to the indicator. For trickier indicators, there are really no limits with our customized version of WebMo. We have many years of experience in creating indicator sheets to fit our clients’ needs, no matter how complex their indicators are.
