Indicator Comprehensive Methodology - consultation space

Consultations
#1

Summary

Averages of comprehensiveness calculations of core, financial and value-added elements as defined in associated tabs. The Weighted Average gives a double weighting to the Core Average as these are essential for IATI data to be valid for use. The unparenthesised values count valid data while those in parenthesis count all populated elements.

{Weighted Average} = (({Core Average} * 2) + {Financials Average} + {Value Added Average}) / 4

Core

Core elements are those that are mandatory in version 2.01 of the IATI Activity standard. This table shows the percentage of current activities where elements are populated with valid data. (Values in parentheses indicate percentage of activities where elements are populated with any data.) The proposed scoring for the transparency indicator recognises the importance of the core and gives it double weighting for the comprehensiveness component.

Financials

Four aspects of financial reporting are tracked: the reporting of commitments and spend, the ability to track funds across activities and organisations, and the existence of activity budgets (which are also given added weight in the forward-looking component). Only current activities are assessed.

Value Added

This table attempts to capture the breadth, richness and usefulness of high quality IATI data. The purpose and methodology for each facet is described in the narrative below the table.

Comprehensiveness info on the IATI Dashboard
#2

Hi Bill,
A few thoughts:
I would suggest adding “Aid Type” and “Disbursement Channel” (ideally to core, but at least to value added) as these fields are de facto the only way in which a partner country can determine if an activity is on/off budget and thus should always be reported. I would also suggest increasing the weighting on publication in the recipient’s language, as the data are not nearly so useful to a partner country if they are not in their local language (as noted in our paper on the Use of IATI in Country Systems).

Best,
Josh

#3

Further to Josh’s comment, Canada would recommend that the indicator methodology be informed by the Development Gateway paper “Use of IATI in Country Systems” and the important feedback offered on what partner countries view as priorities in IATI data. This would be in line with the Busan commitment to meet the information needs of partner countries.

In particular:

  • The priority fields (and attributes) on p.16 of the paper should be in the indicator. As we want a manageable list of fields in the indicator, it may mean removing some (for instance, the added value of a project website, or conditions, is much less clear than the aid type).
  • Given the importance of the language barrier, and the fact that the 2.01 version alleviates this problem, publishing in 2.01 should be recognized.
  • Similarly, the publication of content in the partner country’s official language should become an added-value item. A methodology had been developed for earlier versions of the IATI Annual Report, if I recall correctly.

Thanks

#4

We agree with adding “Aid Type” and “Disbursement Channel” to Value Added elements. Josh, the “core” elements are mandatory in the standard so we couldn’t add them there.

We also agree with the inclusion of recipient language. It is not there at the moment because we are battling with the methodology. Yohanna you are correct it was in the first draft of the indicator but issues were raised with it. Do you score all activities or only those where the donor’s language is different from the recipient’s? (If Donor A is English speaking and only does business in English speaking countries they score 100% while Donor B is English speaking and manages to provide translations for half of their non-English speaking recipients and scores less.) If we need therefore to record information about donor languages as well as recipients this is challenging for MDBs, multilaterals and others.

#5

On recipient language, the issue is simple from the partner country’s perspective: is the information in a language that can be understood in the recipient country? The simplest approach would be to match the languages used in key text fields (e.g. title, description) with the official language(s) of the recipient country.

The publisher’s language is not relevant to this - we’re not trying to assess the publisher’s effort, but the availability of the information. It wouldn’t be the only indicator element where publishers can get 100% with limited or no effort.

Regarding the use of 2.01, have there been discussions on how this could be integrated in the indicator? For instance, could the Version field in the Core dimension be tweaked to add a multiplier that would give 100% for using 2.01, 50% for specifying a 1.0*version, and 0 if the version isn’t specified at all?

#6

‘Commitment’ is an ambiguous concept. Denmark reports ‘Commitment’ in the strictest book-keeping sense, which makes sense for the majority of the ODA-frames in Denmark’s, annual Finance Act, but some frames are managed based on Disbursement-basis only. In these cases, no book-keeping entries can provide data on the vaguer concept of underlying ‘grants’ og ‘pledges’. The current method of this indicator could be interpreted as an incentive to either suppress some of the data – that might, after all, also have limited interest among recipients, or apply a less clear definition of ‘Commitment’ – e.g. by simulating annual commitments equal to annual disbursement-budgets on activities that are managed by disbursement. But, in our view, that would not constitute an improvement of our data; just an improvement of out indicator-score.

We are puzzled by the observation, that 5% of activities (more than one year old) still have no disbursements, and will look into that.

We probably will need to contact IATI tech-team, to get more info on the calculation of this ‘budget’-value. But we will try to recreate the figure first.

Comprehensiveness info on the IATI Dashboard
#7

Regarding "language that can be understood in the recipient country ": in order to keep things feasible for our desk-officers, we took the position that civil servants in partner countries understand either French or English, so we ask our colleagues to encode titles/descriptions in one of these two languages in function of the partner country in question, e.g. in English for activities in South Africa or Rwanda, and in French for activities in DRCongo or Senegal.
Is that an acceptable approach ?

#8

Proposed methodology change: Comprehensiveness -> Financials -> Budget

A change is proposed to the way the assessments are made for the budget component of the financials tab of the comprehensiveness page. Currently budgets must be contained within current activities at the publisher’s lowest hierarchy in order to be counted.

Whilst this approach works well for transaction data, we plan to amend the methodology to calculate the percentage of current activities which contain budgets in the hierarchy which contains the largest number of budgets.

Worked example

Publisher A publishers 10 activities across three hierarchies, with data as follows:

Hierarchy 1:
Number of activities: 3
Current activities: 3
Number of current activities containing budgets: 1

Hierarchy 2:
Number of activities: 4
Current activities: 3
Number of current activities containing budgets: 2

Hierarchy 3:
Number of activities: 3
Current activities: 3
Number of current activities containing budgets: 1

In this scenario:

  • The current assessment would be 33% - i.e. 1 current activity containing a budget at hierarchy 3, out of 3 current activities at hierarchy 3 (the lowest hierarchy level reported by this publisher)
  • The proposed assessment would be 66% - i.e. 2 current activities containing budgets, out of 3 current activities at hierarchy 2 (the hierarchy level level with the greatest number of budgets)

In the event of a tie between the same number of budgets reported at two or more hierarchy levels, the lowest hierarchy will be preferred.

The copy for the narrative and table introduction sections would also be amended to this effect.

#9

If I read the Dashboard accurately, this change would affect 2 publishers? I’m sorry but, is it worth the additional complication?

#10

Thanks for your thoughts, I did some work on this yesterday and this is a reasonably [straightforward change] (https://github.com/IATI/IATI-Dashboard/commit/24b8b592e026c8baf30104174d9e4b4867a13b1e), which will allow the calculations to more accurately publishers with hierarchy-based reporting.

This change is now available to view on the dev version of the [Dashboard Comprehensiveness Financials page] (dev.dashboard.iatistandard.org/comprehensiveness_financials.html). Based on my anaylsis of the CSV download, three publishers now have increased budget scores: Malaria Consortium, WaterAid and United States.

#11

I would just add here that although there are a small number of publishers currently using hierarchies we are expecting that number to increase over time. I know that quite a few other publishers are already looking at using hierarchies to better model their operational approach to programming so it should help if we can address this issue in advance

#12

I think we should consider changing the methodology for other elements as well (not only financials) in order to accommodate multiple hierarchy publishers. Some information are collected and published at lower level according to the organization’s reporting structure. As per the current methodology the publisher is given less points because the information is available only at one level. This may send a wrong message while a publisher is publishing complete information at one level, especially for results and location details. For example, one parent activity can have multiple child activities and those child activities can have multiple sub-locations and results indicators. When we publish at lower level it precisely capture the information at the correct level of the particular activity. This way the organization may publish complete information in the IATI data files. When the dashboard calculates the average for total activities (hierarchy 1 + hierarchy 2) the points gets reduced to almost half and sends the wrong message to any viewer. If the organization is required to duplicate the information at the higher level as a remedy to achieve full score, in my opinion, it would unnecessarily make the data files large and add no value.

Since IATI allows publishers to use multiple hierarchies in their data files it should also accommodate the same everywhere to fairly show the statistics of the information published. In my opinion, if a publisher publish an information only at lower level, then the calculation should be based on total number of activities of that hierarchy level.

2 Likes
#13

On the recipient language, it is stated that:
“Only activities containing only one recipient-country are assessed. Activity must contain title and description elements containing at least one of the official languages spoken for the defined recipient-country/@code.”

How does the scoring work? Is the scoring is only recognizing when both the title and description is in one of the recipient country’s official language? Or is it where the title or description includes the recipient country’s language? (e.g. 50 out of 100 countries had the recipient language in the activity)

Also, if the title and descriptions are in the local language, then is it required to change the language code to the corresponding language by activity level or country level?

Thank you.