 |
 |
@Herman in my view the decision to classify qualitative results as not/partly/fully achieved is one for the M&E community to agree upon. Are there similar approaches used in other systems? (@mikesmith et al?)
Also, I’m not sure that having a meaningful value of ‘0’ is helpful for data users needing to distinguish between different meanings for zero and null.
As stated earlier by my colleague, the Netherlands Enterprise Ageny is in favor of having the opportunity to represent qualitative indicators using the IATI standard.
@ErikH I think there is consensus to include qualitative indicators. The outstanding issue is whether to make @value optional and/or assign an ordinal value.
@bill_Anderson The real point I tried to make was that the whole design of the results framework in IATI is geared towards publishing MEASURABLE results. Therefore we have baseline, target and actual values which enable the publisher to express in howfar the results of the activity are being achieved in a certain timeframe.
Therefore the value is mandatory.
If you want to publish qualitative results, the results are by definition not measurable. Therefore I think qualitative results should be published as a linked document as suggested by @rolfkleef, using a document category ‘Qualitative results’.
The other part of the discussion was that what is seen as a qualitative result, can often be expressed on an ordinal scale (see example above). This type of results does belong i.m.o. the IATI results framework. In that case, as always, the value is mandatory.
So in summary: I suggest to add the ordinal scale as a measure type , thus allowing a lot more flexibility in results reporting (no need for objective quantification). True qualitative results can easily be accommodated with the current link-document functionality. Hope this makes sense.
Yes, @Herman that does make sense. To make sure I do understand …
@value can only be not/partly/fully achieved. (Naming this ‘Qualitative’ is perhaps misleading as it is defining a specific way of measuring a qualitative condition)@value should not be accepted.This setup would allow great flexibility to model all kind of results, eg responses in surveys, questionnaires, etc.
@Bart_Stevens made the interesting point that there are also nominal scales, in which one value is not better than the other. That kind of ‘result’ could also be expressed by adding a measure type ‘nominal’. I have the feeling though that this does not represent an actual result. But I might be wrong here: an typical example showing that a nominal scale can represent a valid result would be helpful in this discussion.
I’ve just re-read this thread following Bill’s email asking us to review the discussion as I’m disappointed this hasn’t reached consensus (as it’s a big challenge in our work). To summarise:
The IATI Technical Team put forward the original submission (which was based on a consultation with M&E people in a wide variety of organisations):
Add “9 - Qualitative” to the IndicatorMeasure codelist
Make the following value attributes optional
result/indicator/baseline/@value
result/indicator/period/target/@value
result/indicator/period/actual/@value
At the consultation call the proposal received no objections.
I may have missed something someone said in the thread, but I think the subsequent comments all agreed with the proposal with the exception of @Herman (sorry to single you out!).
If I’ve understood correctly, the points of disagreement are views on whether:
This may have already happened, but would a call between @Herman and the original proposers of the change, such as @mikesmith be helpful to talk directly about the point of disagreement and see if we can find a way forward?
@annadownie I agree that another call - involving all interested parties reviewing all outstanding results issues - is a very good idea.
We can draft an agenda (summarising where we are on all outstanding issues) and a doodle poll to agree on a date.
If I may be so bold I think it would also be extremely helpful if the interested NGO communities in the UK and Netherlands could also speak with a more collaborative voice on some of these issues.
I said in an earlier contribution that just because something can be quantified, it doesn’t mean it’s desirable or practical to quantify it. Qualitative research is a standard part of participatory studies and is widely used in the sector, so by including it we are not doing anything that is outside the scope of the subject matter. The M&E people asked for this to be included in the standard, and therefore in my view the ability to report qualitative results should be included – I can certainly see no reason not to. This implies that for qualitative results the @value attributes must be optional; however, if for some reason they remain mandatory then the associated documentation should state that the numbers in the attribute are meaningless by definition.
The question then remains of how to implement it. The choices seem to be between a narrative field and a document link. Technically speaking I have a slight preference for a narrative, and indeed I think the best approach might be to allow both (at least one being mandatory), so that the narrative can contain a summary that is conveniently displayed by visualisation tools, and people are not obliged to follow links (sometimes over slow and difficult connections) to arbitrary document types which are not easily amenable to further analysis / use by query & other tools.
Bob Walkden
Business Architect
Save the Children UK
(apologies for my lack of responsiveness)
Without forcing a change in M&E practice and/or distorting/shoehorning data (incorporating @Herman’s view about allowing ordinal scales, @Bart_Stevens’ about nominal scales, and the M&E and @Bart_Stevens’ view about allowing qualitative data) can we agree to:
Add three codes to the IndicatorMeasure codelist:
and make the following value attributes optional
Using only document-link to record qualitative actuals (per Add document-link to Results indicator (included 2.03) ) would also require @value to be optional, but suggest “Qualitative” could be used for IndicatorMeasure rather than adding a fourth code to the codelist.
TL;DR:
+1 on using narrative rather than <document-link>
+1 on making @value optional
Fully agree with this. It would be much easier to use such data.
I have a lot of time for the goal of making all the data easy to process and use, for assessing programmatically whether projects have succeeded (in the sense that they have met their intended goals). However, on the question of whether the IATI Standard should require qualitative results to be quantified – my strong suggestion is not. I think we should be guided by the M&E community (and we have received some pretty clear feedback from them).
There are clearly good reasons for not quantifying particular results (I guess particularly when looking at complex change, where pre-planned quantitative indicators can just lead to isomorphic mimicry; and where qualitative results may provide a richer depth and more accurate picture than quantitative ones).
Secondly, the IATI Standard shouldn’t be trying to force changes in the way that monitoring and evaluation proceeds especially when M&E experts are telling us that this is not a good idea.
Finally, from a practical perspective, many systems that have some good results data do not have that data available in ways that would allow results to be quantified. Requiring them to be quantified is likely to lead to less useful data (e.g. just putting “unquantifiable” as the value).
I have no issues with these suggestions.
@annadownie @bwalkden Yes, you are right. these are the two major points. The discussion is not about whether or not IATI should support qualitative results for a single activity, but about how qualitative results can be modelled in IATI without complicating data-use. When you want to do an statistical analysis over multiple activities, then you really need non-qualitative results.
@mikesmith Thanks for your comments. I think we could have an agreement
if we add 2 rules for the IATI guidelines:
1 - The @value may only be optional if the indicator measure is 'Qualitative’
2 - The @value must be a valid number for all non-qualitative measures.
Does this make sense?
Makes good sense to me.
Actually, I’d go slightly further than my previous reply and suggest that for qualitative results the @value must be omitted (not even an option), and for quantitative results it must be present. However I’d be perfectly content with Herman’s suggestion rather than delay further.
Agree! To pull it all together I think its:
Add three codes to the IndicatorMeasure codelist:
and make the following value attributes optional
add 3 rules for the IATI guidelines:
1 - the @value must be omitted for qualitative measures
2 - The @value must be included for non-qualitative measures
3 - The @value must be a valid number for all non-qualitative measures.
Thanks so much @mikesmith @bwalkden @Herman @markbrough. I will change the status of this issue to a big YES.
This proposal has been been included in the 2.03 upgrade. It can be viewed in the following two Discuss posts: