Sectors: How we broke the standard (and fixed it in d-Portal)

Community Zone Using IATI Data
#1

In Version 1 of the standard sectors were specified at the activity level. In Version 2 it became possible to also report sectors at transaction level.

BUT

A rule was established

Sector MUST EITHER be reported here OR at transaction level for ALL transactions

iati-activity/transaction/sector is now used by 91 publishers including UN CERF, ILO, Spanish Ministry of Foreign Affairs, UNFPA, UNICEF, USA and WHO

This also applies to countries (used by 32 publishers).

At DI we first discovered the problem when we tried to analyse forward-looking budgets by sector. Without trawling through transactions this is no longer (easily) possible.

Similarly, if you filter a search on d-Portal or the Datastore you will get incomplete answers as neither of these systems copes with transaction-level sectors.

Until now

The following logic has now been applied in d-Portal:

  • If no Sectors are reported at activity level they are harvested from transactions
  • If multiple sectors are present a percentage split is calculated from Commitments (or Disbursements and Expenditures if no Commitments are present)

The first iteration of this modification is live now

2 Likes
#2

@bill_anderson via OpenAg (@reidmporter) analysis we had a thread (with contributions from @Herman & @TimDavies) around this rule, which got into the 2.03 list, but didn’t seem to get included in Standards Day, and subsequent discussions. How do we move that forward?

#3

[NB I would add this comment to this thread, but it seems that one is now closed for comments]

@stevieflow: What are you looking to move forward here? The third of @timdavies’ proposals (the <tag> element) was both included in the Standards day and has so far been adopted into v2.03, right?

Personally, I think the second suggestion in the second proposal is a good one, too (make sector restrictions per-vocabulary). I think it’s an oversight that sector restrictions aren’t currently vocabulary-dependent. But (as far as I can tell) this wasn’t worked up into a 2.03 proposal, so I guess that’s why it didn’t move forward.

#4

It’s great that d-portal has been upgraded to reflect this aspect of the 2.0x standard!

But I think the story here is actually that the standard was upgraded, but d-portal was not. I.e. I think a more accurate title for this post would be “That time we upgraded the standard but not d-portal (and then upgraded d-portal to be compliant.)”

When upgrades are made to the standard, it would be great if in-house tools like d-portal led the way in being compliant with the new version. After all (as I understand it) IATI have ultimate control over both things! But in this case, it seems publishers adopted the new version long before d-portal did.

#5

@james.coe points out that this still appears to be an issue for the IATI Datastore. I’ve created a ticket for that here: https://github.com/IATI/IATI-Datastore/issues/284

#6

Hi @andylolz

I dont think Providing sector/country codes at activity and transaction level - enable both? was to do with the tag element. This was an observation that the “rule”, as @bill_anderson states:

Sector MUST EITHER be reported here OR at transaction level for ALL transactions

… seemed rather ambitious and/or impractical for many. There were two use cases We observered:

1 - For a publisher to suddenly drop any sector codes at activity level, in favour of transactional sectors - could throw data users (as now illustrated by d-portal)
2 - ALL transactions should have a sector code. ALL of them? Our observation was that it may make sense for Commitments and Disbursements, but Expenditure? We felt that this wasn’t really researched to fully justify, hence a call for a relaxation of this rule

I’ve no idea why or how this dont make this to standards day at TAG, hence the query.

#7

@andylolz - sorry, I can see that @TimDavies proposes in https://discuss.codeforiati.org/t/providing-sector-country-codes-at-activity-and-transaction-level-enable-both/577/3:

(3) Propose an alternative element for the kinds of coding we are looking for

But - I dont think that satisfies the original observation around this “rule” of sector codes being EITHER activity or transactions, and if the latter then ALL must have a sector.

Since that, the TAG happened, and the tag element (!) has been refined and developed (via Non-statistical secondary sectors (excluded 2.03)) - so it’s actually @TimDavies’ first observation that is more relevant:

(1) Require sectors/countries/regions at the activity level, but make them optional at the transaction level

and

(2) Adjust the restrictions to only apply to OECD DAC Codes, or just to be per-vocabulary

#8

Re. allowing sectors at transaction AND activity level… I think we then get into issues around percentage split, which I believe were discussed as part of the original Non-statistical secondary sectors (excluded 2.03) proposal (and resulted in the proposal being amended.)

(1) Require sectors/countries/regions at the activity level, but make them optional at the transaction level

So… I like this proposal, but I don’t think it constitutes a relaxation of the rule – it’s just a different rule. I say that because currently, sector is not required at activity level. This proposal does require sector at activity level. So I’m pretty sure (though happy to be corrected) that it would require an integer upgrade.

(2) Adjust the restrictions to only apply to OECD DAC Codes, or just to be per-vocabulary

As I mentioned above, I like this proposal. But I think it solves a different problem to the one you’re describing, @stevieflow.

Just to get more of an understanding of the intention of your suggestion, can you explain why it would be difficult for publishers to provide a sector code for expenditures? Or are you speculating that this information wouldn’t be useful to users? I’m not sure I follow.

#9

As pointed out above, this remains broken on the Datastore. I sent a fix for it a few weeks ago: