Results – represent more than quantitative data (included 2.03)

Standard Management 2.03 Decimal Upgrade Proposals
#1

Title
Results – represent more than quantitative data

Standard
Activity

Schema Object

  1. IndicatorMeasure Codelist
  2. iati-activities/iati-activity/result/indicator/baseline
    iati-activities/iati-activity/result/indicator/period/target
    iati-activities/iati-activity/result/indicator/period/actual

Type of Change

  1. Add code to codelist
  2. Change to Schema

Issue
• Currently, only quantitative indicators (units or percentages) may be represented using the IATI standard.
• Why is this a problem?: While some quantitative indicators can provide direct meaning (for example: percentage of patients that responded positively to drug X) they cannot reflect the qualitative context that is often necessary to articulate the meaning behind the values.

There are many interventions in international development where quantitative measures are not relevant or meaningful, for example in relation to policy-influencing and advocacy work, and in some forms of capacity-development and empowerment interventions. These are common areas of work for civil society organisations.

If there is not scope to report such results, CSOs will try to ‘play the game’ either by doing more of the measurable work and less of the hard-to-measure work, even if the latter is vital, or as we are already seeing, by converting qualitative results into quantitative results. Conversion of results into a quantitative format can be time-consuming and expensive, provides little or no value to the initiative, risks poor quality and redundant data, and provides little value for money.

Proposal

  1. Add a new code to the Indicator Measure codelist for Indicator attribute @measure
    Code = qualitative
    Description = This indicator is qualitative

(see http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/ and http://iatistandard.org/202/codelists/IndicatorMeasure/ for relevant sections of the standard)

  1. In order to avoid having to use a unit or a percentage, make the @value attribute for indicator/baseline, and period/target and period/actual elements optional
    (http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/period/target/ for example relevant section of the standard)
    This will have the additional advantage of avoiding having to record ‘no baseline available’ as a zero (0) value – no value and zero value are very different concepts in monitoring and evaluation and having to record a zero value may cause issues with data quality.

Standards Day
Workshopped at the TAG 2017 and mentioned at the end of the Standards day as part of the results section. Although there was very little time to discuss the proposal, no criticism of the proposal was offered. Proposal has been on IATI Discuss since March 2017.

Links
• This topic is discussed here: Results: represent more than quantitative data
• This topic addresses Principle 1 from a consultation driven by Monitoring and Evaluation experts from UK CSOs Jan – Mar 2017 – see Results: discussion space and TAG 2016/17 path. Technical suggestions were devised by technology specialists at the Nethope Athens conference March 2017. In all around 30 M&E and technical specialists were involved in this consultation and it builds on a previous consultation by Bond 2015-16 (https://www.bond.org.uk/resources/publishing-results-to-iati - also on discuss.iatistandard : Sharing Results using IATI data standard: will it improve learning and accountability? ).

4 Likes
Add document-link to Results indicator (included 2.03)
Add indicator measure "other" to the codelist (excluded 2.03)
#2

I agree that only quantitative measures are not enough, but do not agree to make the baseline, target and actual values optional. I suggest to add an ordinal scale as an extension, e.g. to the aggregation status code list. The mean reason, being that you lose a lot of analytical power when you do not value a result. I.m.o. you can always value a result by using an ordinal scale (e.g. '1 - fair, ‘2 - good’, ‘3 - very good’). This allows for e.g. surveys and expert opinions of very complex situations to be translated to a value.

See for detailed comments:

#3

Thank you for adressing this issue, possibilities for qualitative indicators would be very welcome. Another solution would be to include a new code which is binary (in addition to percentage and unit): 1. achieved, 0. not achieved.
An ordinal scale could be a good alternative in case there is flexibility in defining the scaling, e.g. the scaling could be 1. achieved, 0. not achieved in this case (the result indicator description could then describe the result to be achieved).

#4

Hi all,

Thank you to everyone for your input so far. This is a very interesting topic, and something I heard come up many times while I was working with Bond, and has been brought up by IRC staff while since I have been here.

I’m afraid I can’t add a technical solution, but I have spoken to a colleague who works with results data regularly and I can pass on her experience around capturing quantitative vs. qualitative data.

"I don’t see the purpose of ascribing a value to qualitative data. They are just fundamentally different methodologies. Part of the reason why one conducts qualitative research is to better understand constructs, beliefs, attitudes, etc. that are not easily quantifiable.

For example, we conduct focus group discussions with women to learn about the barriers they face when accessing health services. We end up with a rich qualitative dataset and we pull out common themes across the groups of women we talked to. In this case, while it’s possible to say 52% of women described XX theme, it may not make sense to describe qualitative data in these terms because the sample is not representative and the %s are therefore less important than the actual themes that emerged. Moreover, qualitative and quantitative data collection are carried out for very different reasons, so while it may be technically feasible to quantify qualitative data, this doesn’t really make sense methodologically."

In short, while it is possible to wrangle qual data into a quant measure, it is time-consuming, costly and may make the data itself meaningless. This statement suggests that the original proposal is an effective way forward.

Many thanks,

Daniel

#5

I don’t think we should try to shoe-horn qualitative data into a quantitative element. There is room to add documents with qualitative results in various formats (documents, video’s, datasets, etc).

In Daniel’s example: I would also suspect that the qualitative analysis of what themes are mentioned across group discussions will lead to other, probably more quantative indicators about what you then will try to do or to achieve in the activity. Or as a means of verification whether you achieved certain other goals.

I’ve proposed a measure type “other” to cater more broadly for anything other than units or percentages. But you’d still have to provide some value (even if just “0” or “1”), to be able to do something meaningful with the data.

I’d love to see examples of what the target and actual of a qualitative indicator would look like, to be meaningful.

The qualitative methods I have seen produce more elaborate data structures such as lists, graphs, or more complex models. That’s a topic for the results working group.

#6

@DanielMackenzie When using qualitative results, how do you assess if you achieved your intended goals or not (e.g. in your example)? Wasn’t the current IATI definition for results designed especially to answer this question?

@rolfkleef Yes I agree, some examples of baselines, targets and actuals of qualitative results might be very helpful for this discussion.

#7

This is so important for us- particularly when it comes to reporting on progress on programmes that are focused on policy change and advocacy where qualitative data is often far more valuable than quantitative. Whilst I can see the value in having baseline, target and actual values that could be translated for qual data (eg ‘1, on track, 0, not on track’ etc), I’d still want to see them as optional to give people the freedom to show progress against the indicator in the way that makes most sense to the programme and the data.

#8
  1. In order to avoid having to use a unit or a percentage, make the @value attribute for indicator/baseline, and period/target and period/actual elements optional […] This will have the additional advantage of avoiding having to record ‘no baseline available’ as a zero (0) value

The value attribute is of type xsd:string. As such, it is possible at present to provide text data as the value attribute value - it is not necessary to convert data to something numerical such as a unit or percentage.

#11

Per Rolf and Herman’s request, below is an example of a qualitative indicator. It’s for a project that aims to promote safe hygiene practices through the routine immunisation programme in Nepal.
Note: I have used <indicator measure="Qualitative"> and have not included a value for eg <actual> per this proposal. Although technically per @hayfield’s comment the current standard requires a string “value”, using freetext like <actual value="qualitative">, <actual value="none">, <actual value=" "> or <actual value="0"> seems to provide a data quality issue that is easily avoided by making value optional (per proposal).

<result type="2">
  <title>
    <narrative>Potential for inclusion of hygiene promotion into national vaccination programme beyond pilot project</narrative>
  </title>
  <description>
    <narrative></narrative>
  </description>
  <indicator measure="Qualitative">
    <title>
      <narrative>feasibility of integrating hygiene promotion into vaccination programme under normal programme delivery circumstances</narrative>
    </title>
    <baseline year="2014">
      <comment>
        <narrative>Scoping study in 2012 demonstrated the desire of health sector to pilot initiative through identifying missed opportunity to combine hygiene promotion into national vaccination programme – see published scoping document http://washdev.iwaponline.com/content/3/3/459.</narrative>
      </comment>
    </baseline>
    <period>
      <period-start iso-date="2014-01-01"></period-start>
      <period-end iso-date="2016-04-05"></period-end>
      <actual>
        <comment>
          <narrative> recommendations copied from the document 16_04_051-Nepal vaccines FCP narrative report template April 2016 - March 2017_Final_13 May 2017_A_B_Om from the activity’s document-link:
o	Both the FCHV and health worker should be involved to run the session smoothly.
o	Some incentive for health worker to motivate them to work actively.
o	Regular supervision from district and center level for proper feedback.
o	Vaccinator should also involve in review program.
o	This program should give continuity in future and scale up in other districts also because hygiene promotion is directly/indirectly related to decrease/minimize the cases of child mortality and morbidity.
o	Number of games should be decrease so that it will takes less time to run session.</narrative>
        </comment>            
      </actual>
    </period>
    <period>
      <period-start iso-date="2016-04-05"></period-start>
      <period-end iso-date="2017-02-10"></period-end>
      <actual>
        <comment>
          <narrative> conclusion copied from review meetings document with title” Central Level Review Meeting of "Hygiene Promotion through Routine Immunization Programme in Nepal" on 10th Feb, 2017”:
            Representatives of four District/Public Health Office and FCHVs have shared that the Hygiene Promotion through Routine Immunization Programme can be implemented successfully using the existing health system. It has shown its mutual synergy to increase the positive hygiene behaviour change and increasing the coverage of routine immunization. They have also flag that the program has played a vital role in reducing the dropout and wastage rate of the vaccine. All four districts mentioned that there is no outbreak of diarrhoeal and cholera diseases in their districts after the implementation of this program.
            The integration of hygiene promotion package in Routine Immunization sessions has been well appreciated and accepted by FCHVs / health workers and the mothers. So, this review meeting has recommended for the need to continue the program in these four districts and should be scaled up in other districts.</narrative>
        </comment>
      </actual>
    </period>
    <period>
      <period-start iso-date="2014-01-01"></period-start>
      <period-end iso-date="2017-06-01"></period-end>
      <target>
        <comment>
          <narrative>fully feasible to integrate hygiene promotion into vaccination programme under normal programme delivery circumstances.</narrative>
        </comment>
      </target>
    </period>        
  </indicator>
</result>

Attempting to codify and “shoehorn” the indicator into numbers would either trivialize, be unrepresentative and undermine IATI data; or require a disproportionate amount of effort (the project only needs and uses the qualitative result, not a quantitative result) and be very complex (for example different quantitative values are relevant for each evaluation context which would require additional verification with stakeholders) – this provides poor value for money, especially given this is just one (very important) indicator of many for the project.

I’m happy to provide the source documents if needed and see related proposal Add document-link to Results indicator (included 2.03) for alternative to embedding references to documents in the freetext.

#12

Apologies for the late posting. Here are two example indicators provided by a colleague at IRC which demonstrate how assigning a value to a qualitative measure can undermine data quality, creating a greater reporting burden and even have a detrimental effect on programme strategy.

"Indicator 1) National policy on sexual violence is adopted in X country.

Why use qualitative narrative to report this?

There are many nuances. First, what if the government adopts a national policy but it doesn’t cover all the issues that we find critical (like marital rape). Second, even if adopted, it may be poorly enforced, and may even have detrimental effects (it can whitewash the issues when a government can claim that the policies are in place). And if either of those cases happen, does it still count as “one” policy changed in our logframe?

In the past, IRC have reported on these indicators using qualitative narrative: IRC’s actions, and reporting on any outcomes. Sometimes we’ll conduct focus groups, for example, to get feedback from citizens on how these policies impact (or not) their daily lives. These are more nuanced account of how policies change than a number."

A quantitative measure may mask political tensions and misrepresent the situation.

Indicator 2) Women are increasingly empowered to participate in local governments.

Why use qualitative narrative to report this?

Evidence has shown that time and time again, quotas for gender in government structures do not always trickle down to women (both those in the government and the broader community) feeling more empowered to voice their opinions, needs & concerns. Nor do women in government necessarily act primarily as representatives for their entire gender rather than other segments of their local constituents. Qualitative data can help us better understand not just the numbers but also whether women can and do actively participate."

As in many cases, it would be possible to add a quantitative measure, but this would add two extra layers of reporting - 1) decided what the quant measure is 2) adding narrative report to compensate for the first.

I hope these examples contribute to the discussions in a productive way. Any questions, please let me know.

1 Like
#13

It is proposed that a measure “Other” is added to the IndicatorMeasure codelist now, and that further discussion on a structured solution to qualitative indicators is carried forward for the next upgrade

#14

I completely agree with the need to represent more than quantitative data. Most of our work (at Anti-Slavery International) aims for results that can’t meaningfully be represented by a number or percentage. Most of our projects look at the level of political interest in a range of anti-slavery measures and willingness to implement positive changes.
Partners and communities give feedback on the sincerity and quality of governments’ interactions and implementation of their recommendations into policies and laws. A number of projects report on communities’ perceived changes in gender equality and female leadership/empowerment and acceptance of that.

Echoing the points made from the IRC, narratives inform about the majority of our work far more comprehensively and meaningfully than numbers could. As the whole point is often not simply about whether something ‘is’ or ‘is not’, but it is about subjective perceptions and experiences of factors that are complex and shifting. Quantitative measures of these are often meaningless and fail by their own standard of supposed accuracy. To only have quantitative measures assumes that everything can be assessed that way which isn’t the case with our work.
As I understand it, the point of IATI is to improve transparency and inform - insisting on quantifying qualitative information seems to me to be counter to this purpose as it overly simplifies data and information to the point where integrity and meaning are lost.

I’m afraid that I can’t contribute to the technical side of the conversation about how this might be implemented in IATI but looking forward to the conversation and what comes out of it all!

#15

Sorry as well for the late feedback. Handicap International can only agree on and further highlight the importance and the need for qualitative data to be reported on. Most of our work relates to the improvement of people with disabilities’ or vulnerable people’s quality of life and we use many qualitative indicators alongside with quantitative ones to account for the quality of our actions. Here are a few examples of these indicators:
· Satisfaction of the beneficiaries in relation to the services received.
· Improved quality of services (against a set of quality criteria)
· Increased level of understanding of people about specific issues
· Improved self-esteem and self-confidence of beneficiaries
· Improved quality of life of beneficiaries (quality for life indicators being defined according to each context).
Hope this is useful at this point of the discussion.
Best,
Laurence

#16

Thanks for this @IATI-techteam. However I wondered why the team had a preference for IndicatorMeasure = “Other” over IndicatorMeasure = “Qualitative” (the original proposal), and how using the word ‘Other’ adds to the usefulness of the data where the word 'Qualitative would not?

In the interests of transparency, please can you also explain the decision-making process behind this as there was clearly articulated support on this forum for the original proposal.

#17

Can I first make it clear that @IATI-techteam hasn’t decided anything. We have proposed a way forward. The consultation is ongoing.

@SJohns we made our choice from the above discussion based on a ‘lowest-common denominator’ approach in the hope of reaching consensus. If there is, in fact, consensus for the original proposal we stand corrected and will gladly revert. If there is no consensus can I suggest that the M&E community continue to work it out?

This is, in my opinion, a matter for M&E experts, not the Tech Team. We have no vested interests and will follow any consensus reached.

#18

The original proposal was a consensus from a range of M&E experts and IATI users that are involved in using IATI to provide data on results. This is all clearly documented on IATI Discuss. I’m not inclined to go back to the M&E experts to ask for more evidence when they have already contributed beyond what was originally asked of them (see the posts above for examples). The burden of proof is really on the IATI technical team to justify their proposal to use ‘other’ and punt the rest of it into a future upgrade.

Although you say that the IATI technical team have not made a decision, the perception from the IATI team’s post on this thread is that they have clearly stated a preference for a particular way forward.

In order to maintain the integrity and transparency of the consultation, I would like to see the whole original proposal be put forward for consultation and for the community themselves to be allowed to refine the proposal between now and 7 July, as per the timeline. http://iatistandard.org/202/upgrades/decimal-upgrade-to-2-03/

#19

Fair comment.

At least it would have been If I hadn’t already tried to answer your concerns above.

At Standards Day in Dar I think we attempted to consolidate and improve on the governance of the standard through consensus. Personally I think it was a good step forward for the community as a whole. For the technical team it was another day of learning how best to balance the disparate interests and concerns of the community. We may not always make the right call, but let me assure you again that we do so in good faith. We have no vested interests.

It is your prerogative to question the assurances I have given you. It is mine to defend the integrity of our team.

#20

@bill_anderson So will the original proposal be added to the consultation?

#21

Hi all,

Many thanks to @SJohns and @bill_anderson for your comments.

I agree with Sarah that the original proposal should go through. I’d also like to emphasise that no-one is attacking the integrity of the Technical Team. We all recognise the great work that goes into the upgrades and the difficulties in mediating many differing views. It is the variety of perspectives that make it such a rich and diverse community!

The proposal should go through in its original form for the following reasons:

  • While Rolf’s suggestion for ‘Other’ marker may be seen as the lowest common denominator, there has been no support for this suggestion, on this thread or the one linked to in the post. This is not to suggest the proposal has no value in itself - indeed, it would be better than the original position! - but it has not been approved by the community in relation to the issues raised in the original proposal. I do not see how this relates to consensus.

  • A suggestion was raised by @hayfield that value can be recorded as a string, so there is no need to make it optional. @mikesmith countered with a concern about data quality. This concern has not been addressed, and relates to @Herman’s point about analytical power.

  • Herman and @rolfkleef requested examples of qualitative indicators that would lose meaning if they were assigned a value. Examples have been provided by three organisations from across the sector, with support for the original proposal. This counter has not been addressed.

It is confusing to those of us who put in a lot of work to answer all of the challenges why the proposal has not made it through to be discussed and refined further. Nor has our evidence been addressed. If the proposal goes through in full to the next stage we can work to refine it together.

As a separate note, we may want to consider whether consensus is an effective way to move forward. It is very easy for someone to post a negative comment without providing evidence, and a proposal stalls.

I hope these comments add positively to this fruitful discussion.

Best,

Daniel

3 Likes
#22

Hi All. I have to admit I’m a bit confused by the overall process - I thought the in/out discussion would happen from 17/06/17 till 07/07/17 per http://iatistandard.org/202/upgrades/decimal-upgrade-to-2-03/ and the technical team undergoes consultation with the community? It seems on some topics this is the case, but others, such as this one, there’s already been a proposal made? I’m not sure how it was arrived at, specifically:

Re: 1) from original proposal (the need for “qualitative” IndicatorMeasure):

Re: 2) from original proposal (“value” should be optional)

  • @hayfield pointed out a string is currently required so no need to put “0”
  • @mikesmith countered this will cause a data quality issue (eg more difficult to analyse etc.)
    No-one has responded to say that data quality is not an issue for IATI

Given there have been no response to the counters it seems reasonable to assume that the concerns have been addressed and that at this time the original proposals should go through, not that they should be dropped/amended?

Thankfully I see that @bill_anderson has clarified that nothing has been decided yet (and I think this only happens on the 28/07/17 per http://iatistandard.org/202/upgrades/decimal-upgrade-to-2-03/)? How do I tell what the current position is for the proposals - is there a topic/document I’ve missed somewhere?