Had a question from a client today that sent me off on a rabbit chase…
As a launching off point, in the guidance on “What data should I publish?”, it says that:
- Data should be cumulative. If you’re publishing every quarter, each activity should include data for the quarter being reported along with existing data from previous quarters. This will also enable you to amend your existing data as necessary.
This threw me initially because I read too much into the term “cumulative”, but the text in italics makes clear that the intention of this guidance is to prevent publishers from removing historic data in subsequent updates. In other words, we’ve established that we want to make it easy for downstream data users to have full information with regards to the activity in question. Got it. Moving on…
In the reference page for result/indicator/period/actual, it states that:
A record of the achieved result for this period.
At first glance, my read is that this refers to incremental results, that which was achieved “for this period”. In other words, if I trained 5 people last year, but only 3 people this year, I would publish 3 as my actual for this year, not 8 (which would be the cumulative result, from the beginning of time to the current period). Then again, I wouldn’t fault anyone for interpreting this the other way, that the (cumulative) result “for this period” is 8, just as the result for the previous period was 5. Keep your finger on that minor ambiguity…
In this Discuss post (which covers several issues related to results), @pelleaardema says:
At the Netherlands Red Cross we keep track of changes over time by defining different periods for targets and actuals:
- the target is usually valid for the entire activity period (but it’s also possible to work with annual targets)
- the actual values will reflect the progress made over a shorter period of time, e.g. a quarter. In this case it is important that the actual value shows the increment in that period, rather than the cumulative value.
This explicitly states that at least one publisher out there uses incremental values rather than cumulative. In fact, DevResults clients also publish incremental results, which is all that our platform allows for at present. But…
Both the Dutch and the DFID guidance (which many organizations use as their primary reference for what to publish) do two things:
- Avoid stating explicitly whether they’re looking for cumulative or incremental results.
- Include illustrative examples that are similarly ambiguous…
<period>
<period-start iso-date=”2012-01-01” />
<period-end iso-date=”2015-12-31” />
<target value=”2000000”>
<comment>
<narrative>
In 2015, 2 mln people in Country X should have access to
independent media on a regular basis (60% of the population)
</narrative>
</comment>
</target>
</period>
<period>
<period-start iso-date=”2012-01-01” />
<period-end iso-date=”2012-12-31” />
<actual value=”300000”>
<comment>
<narrative>
Narrative explanation of the actual value, e.g. how it was
reached, in which areas the most progress was made.
</narrative>
</comment>
</actual>
</period>
<period>
<period-start iso-date=”2013-01-01” />
<period-end iso-date=”2013-12-31” />
<actual value=”500000”>
<comment>
<narrative>
Narrative explanation of the actual value, e.g. how it was
reached, in which areas the most progress was made.
</narrative>
</comment>
</actual>
</period>
<period>
<period-start iso-date=”2014-01-01” />
<period-end iso-date=”2014-12-31” />
<actual value=”1200000”>
<comment>
<narrative>
Narrative explanation of the actual value, e.g. how it was
reached, in which areas the most progress was made.
</narrative>
</comment>
</actual>
</period>
</indicator>
If you just look at the 3 actuals for 2012, 2013, and 2014 – 300,000 + 500,000 + 1,200,000 – you get exactly the target value (2,000,000), which suggests that the values should be considered as incremental, but alternatively, one could read these results as cumulative and determine that they have yet to achieve the target. (For the record, I’m sure whomever wrote this example was mostly illustrating what the elements mean rather than trying to illustrate how results should be interpreted, but even so, this is the most detail published in any authoritative source that I can find about whether results should be incremental or cumulative.)
SO! With all that in mind, my questions:
- Am I missing something obvious and explicit about how publishers should publish indicator results by period? (I did initially look through the archives to see what the intention was when results and indicators were introduced, but gave up quickly:-)
- If I’m not missing anything, is IATI’s guidance/reference in need of clarification?
- If IATI intentionally avoided coming down on one side of incremental vs. cumulative, does that suggest that yet another attribute is needed in the standard to clarify how
<indicator>actuals should be interpreted/aggregated? - With regards to data use, what do we actually prefer? I assume it would be more straightforward for the entire world to agree that incremental results are where it’s at, as this would speed both manual aggregation (i.e. adding this period to last period at a glance or in Excel) as well as automated aggregation (i.e. in a data portal or analysis app). But that would suggest that we’re only interested in things that tend to move in only one direction - i.e. things we’ve done, outputs, etc. - and not metrics that can fluctuate up and down, because having a negative incremental result would be weird, right? Additionally, if donors are using IATI as progress reports, are they expecting to see cumulative or incremental results per period?
- I’m tempted to ask if clarifying this ambiguity between incremental vs. cumulative was the/an intention behind
@aggregation-status, but I know from other threads (which I can’t be bothered to locate at the moment) that there’s a host of known reasons why publishers say that results should/should not be aggregated - percentages, methodological inappropriateness, etc. Are these two threads at all related, or have I truly gone down the rabbit hole now?
Would appreciate any clarity yall can bring to this situation: @pelleaardema @stevieflow @Herman @SJohns @mikesmith @bill_anderson