Could I get clarification on 2)?
Do we want @value to be optional or for it to remain mandatory with an ordinal scale?
In response to the above discussion and concerns it is proposed that @mikesmith’s original submission is adopted. Namely:
- Add “9 - Qualitative” to the IndicatorMeasure codelist
- Make the following value attributes optional
- result/indicator/baseline/@value
- result/indicator/period/target/@value
- result/indicator/period/actual/@value
These changes have been included on the summary table
The timetable at that link says:
17 June - 7 July
The Community can help refine the proposals and discuss in detail what has been included or left out.
Maybe a bit late, but I would still like to add my 2 cents from a measurement- technical point of view:
Qualitative measurements are fundamentally different from value-based measurements and ordinal measurements. As other commenters rightly remarked, we shouldn’t force people to reduce everything to values. However, you can’t in all circumstances reduce qualitative measurements to an ordinal scale either.
Ordinal scales are a breed on their own. Should we wish to integrate ordinal scales in the activity standard then it’s not sufficient to have tags for target and actual values. You need additional provisions to be able to register the list of codes/values and their respective explanation (text).
In some cases you can make a summary of qualitative information by using a nominal scale rather than an ordinal one. This means you can use values but one value is not better than the other.
About targets: in some cases you can have a target even when you use qualitative information in the sense that you describe a desired situation. But in other cases this may not be possible. Qualitative information is also useful to describe unexpected but very important effects, in which case you don’t have a target.
Finally, there is more than just numbers or (qualitative) text. Working on project management software I identified 17 different kinds of indicators so far and that is certainly not the end of it.
1 Like
Notes from consultation calls w/c 3rd July
Outcome:
The proposal was reviewed by those on the call and there was no objection from the group.
Back from vacation I read through all comments resulting in the decision to add this proposal to 2.03. I still have severe doubts that this is the correct way forward.
The example given by @mikesmith does i.m.o. clearly shows that an ordinal value could easily be defined to measure the success of reaching the goal of the intended intervention:
0 - not achieved (it is not feasible to integrate hygiene promotion into the vaccination program)
1 - partly achieved (it is partly feasible to integrate hygiene promotion into the vaccination program)
2 - fully achieved (it is fully feasible to integrate hygiene promotion into the vaccination program)
Adding this type of ordinal values, would enable answering very interesting questions, especially when multiple organizations and many activities are involved in the implementation of such a programme eg:
- which programme improvement interventions are actually successful?
- which programme intervention prove unsuccessful and what can we learn from it?
- in which countries is such an intervention successful?
- …
From the viewpoint of a single activity, it does not matter very much if an result value is not defined. From the viewpoint of portfolio management when you are dealing with many activities implemented by many organizations, you lose a lot of analytical power by allowing qualitative results only. I.m.o. you should always be able to define a measure for success. If you can’t, than what is the point of doing the activity at all?
@Herman Thank you for your comments. During the consultation call the proposal about using ordinal scale was mentioned but no views were expressed either way (for or against).
The consultation call was attended by 8 representatives from different organisations, including @mikesmith and other members who have previously engaged with the results agenda.
We welcome further discussion on this proposal.
@Herman in my view the decision to classify qualitative results as not/partly/fully achieved is one for the M&E community to agree upon. Are there similar approaches used in other systems? (@mikesmith et al?)
Also, I’m not sure that having a meaningful value of ‘0’ is helpful for data users needing to distinguish between different meanings for zero and null.
1 Like
As stated earlier by my colleague, the Netherlands Enterprise Ageny is in favor of having the opportunity to represent qualitative indicators using the IATI standard.
@ErikH I think there is consensus to include qualitative indicators. The outstanding issue is whether to make @value optional and/or assign an ordinal value.
@bill_Anderson The real point I tried to make was that the whole design of the results framework in IATI is geared towards publishing MEASURABLE results. Therefore we have baseline, target and actual values which enable the publisher to express in howfar the results of the activity are being achieved in a certain timeframe.
Therefore the value is mandatory.
If you want to publish qualitative results, the results are by definition not measurable. Therefore I think qualitative results should be published as a linked document as suggested by @rolfkleef, using a document category ‘Qualitative results’.
The other part of the discussion was that what is seen as a qualitative result, can often be expressed on an ordinal scale (see example above). This type of results does belong i.m.o. the IATI results framework. In that case, as always, the value is mandatory.
So in summary: I suggest to add the ordinal scale as a measure type , thus allowing a lot more flexibility in results reporting (no need for objective quantification). True qualitative results can easily be accommodated with the current link-document functionality. Hope this makes sense.
Yes, @Herman that does make sense. To make sure I do understand …
- A value should be added to the IndicatorMeasure codelist that is associated with a rule that the related
@valuecan only be not/partly/fully achieved. (Naming this ‘Qualitative’ is perhaps misleading as it is defining a specific way of measuring a qualitative condition) - The proposal to accept qualitative results where the ‘measure’ is explained in text, not in the
@valueshould not be accepted.
- I would suggest to add and value in the IndicatorMeasure codelist “Ordinal” (see WiKi). The rule is that when you use this measure type the @value must be a positive integer. In this way also other ordinal scales can be accommodated (see the example in the WiKi on ordinal data). The meaning of the ordinal values can be described in the indicator narrative.
This setup would allow great flexibility to model all kind of results, eg responses in surveys, questionnaires, etc.
- Yes definitely: measures should not be in text. Text measures can not be summarized, aggregated or otherwise analyzed thus making the use of results-data very difficult if not impossible.
@Bart_Stevens made the interesting point that there are also nominal scales, in which one value is not better than the other. That kind of ‘result’ could also be expressed by adding a measure type ‘nominal’. I have the feeling though that this does not represent an actual result. But I might be wrong here: an typical example showing that a nominal scale can represent a valid result would be helpful in this discussion.
I’ve just re-read this thread following Bill’s email asking us to review the discussion as I’m disappointed this hasn’t reached consensus (as it’s a big challenge in our work). To summarise:
The IATI Technical Team put forward the original submission (which was based on a consultation with M&E people in a wide variety of organisations):
Add “9 - Qualitative” to the IndicatorMeasure codelist
Make the following value attributes optional
result/indicator/baseline/@value
result/indicator/period/target/@value
result/indicator/period/actual/@value
At the consultation call the proposal received no objections.
I may have missed something someone said in the thread, but I think the subsequent comments all agreed with the proposal with the exception of @Herman (sorry to single you out!).
If I’ve understood correctly, the points of disagreement are views on whether:
- All results can be translated meaningfully and usefully into a quantitative scale of some kind
- If they can’t, qualitative results should only be communicated through a document link
This may have already happened, but would a call between @Herman and the original proposers of the change, such as @mikesmith be helpful to talk directly about the point of disagreement and see if we can find a way forward?
@annadownie I agree that another call - involving all interested parties reviewing all outstanding results issues - is a very good idea.
We can draft an agenda (summarising where we are on all outstanding issues) and a doodle poll to agree on a date.
If I may be so bold I think it would also be extremely helpful if the interested NGO communities in the UK and Netherlands could also speak with a more collaborative voice on some of these issues.
I said in an earlier contribution that just because something can be quantified, it doesn’t mean it’s desirable or practical to quantify it. Qualitative research is a standard part of participatory studies and is widely used in the sector, so by including it we are not doing anything that is outside the scope of the subject matter. The M&E people asked for this to be included in the standard, and therefore in my view the ability to report qualitative results should be included – I can certainly see no reason not to. This implies that for qualitative results the @value attributes must be optional; however, if for some reason they remain mandatory then the associated documentation should state that the numbers in the attribute are meaningless by definition.
The question then remains of how to implement it. The choices seem to be between a narrative field and a document link. Technically speaking I have a slight preference for a narrative, and indeed I think the best approach might be to allow both (at least one being mandatory), so that the narrative can contain a summary that is conveniently displayed by visualisation tools, and people are not obliged to follow links (sometimes over slow and difficult connections) to arbitrary document types which are not easily amenable to further analysis / use by query & other tools.
Bob Walkden
Business Architect
Save the Children UK
1 Like
(apologies for my lack of responsiveness)
Without forcing a change in M&E practice and/or distorting/shoehorning data (incorporating @Herman’s view about allowing ordinal scales, @Bart_Stevens’ about nominal scales, and the M&E and @Bart_Stevens’ view about allowing qualitative data) can we agree to:
Add three codes to the IndicatorMeasure codelist:
- Nominal - the indicator is measured as a quantitative nominal scale.
- Ordinal - the indicator is measured as a quantitative ordinal scale.
- Qualitative - the indicator is qualitative
and make the following value attributes optional
- result/indicator/baseline/@value
- result/indicator/period/target/@value
- result/indicator/period/actual/@value
Using only document-link to record qualitative actuals (per Add document-link to Results indicator (included 2.03) ) would also require @value to be optional, but suggest “Qualitative” could be used for IndicatorMeasure rather than adding a fourth code to the codelist.
1 Like
TL;DR:
+1 on using narrative rather than <document-link>
+1 on making @value optional
Fully agree with this. It would be much easier to use such data.
I have a lot of time for the goal of making all the data easy to process and use, for assessing programmatically whether projects have succeeded (in the sense that they have met their intended goals). However, on the question of whether the IATI Standard should require qualitative results to be quantified – my strong suggestion is not. I think we should be guided by the M&E community (and we have received some pretty clear feedback from them).
There are clearly good reasons for not quantifying particular results (I guess particularly when looking at complex change, where pre-planned quantitative indicators can just lead to isomorphic mimicry; and where qualitative results may provide a richer depth and more accurate picture than quantitative ones).
Secondly, the IATI Standard shouldn’t be trying to force changes in the way that monitoring and evaluation proceeds especially when M&E experts are telling us that this is not a good idea.
Finally, from a practical perspective, many systems that have some good results data do not have that data available in ways that would allow results to be quantified. Requiring them to be quantified is likely to lead to less useful data (e.g. just putting “unquantifiable” as the value).