Size .... the 47,68 Mb file size limit for dashboarding

Community Zone
#1

Discovered the hard way that the IATI dashboards can’t handle xml files larger than 50 million bytes. This is a hardcoded maximum as explained by the IATI support team.

If publishers follow the desired quest in coverage and comprehensiveness sooner or later any file size limit will turn into an obstacle. What’s large today as limit will be tight tomorrow.

Wonder if such size limit stands in ‘modern times’ and in the vision path of IATI. Looking forward to views on this…

In order to have our publishing statistics properly reflected on IATI dashboard and on the Grand Bargain dashboard, we are now advised/forced to split our data set.

Splitting data sets seems an easy fix, it is however cumbersome and not without risks. How to avoid overlap (double activity publication) and assure no gaps. Also one needs to monitor and plan a new split before running into size problems again.

Isn’t it better to adjust the coding and allow for large data files?

1 Like
#2

Hi Leo,
This limitation clearly is a technical limitation. I wonder why this limitation is still there. It is nowadays certainly technical possible to process larger XML files. See this XML file size article from @David_Megginson in 2005!

It seems worth investing in removing this technical limitation. It is the difference between solving this limitation once centrally or forcing many IATI publishers to artificially split XML files. This is not only a waste of time and money but also introduces the risk of duplicate data.

A robust datastore should make it easy for data users to retrieve e.g. country based or sector based IATI data, without worrying about how XML files are split. So also from the functional point of view, there is i.m.o. no reason to split IATI XML file [eg by country or by sector].

#3

(Emphasis mine.) I agree with the above. But I don’t think I’m out on a limb when I say: we don’t currently have a robust datastore. Until it’s improved and provided as a reliable service, I think the filesize limit needs to stay.

Ideally, data users wouldn’t need to interact with XML unless they want to. But at the moment, for bulk data querying, I don’t think it’s avoidable.

1 Like
#4

Wasn’t the original reasoning around something for data users on low bandwidth? The documentation isn’t explicit on that, though.

And yes - if services around IATI can enable these users to access data on a relevant basis, then the shape and size of the original XML wouldn’t be so optimal – unless people really want to go and download individual basis.

Maybe the data use task force can look into this -> @JohnAdams (how do people want their data served?)

2 Likes
#5

Agree. Definitely a topic for the DUTF. It is on the agenda. Nonetheless, also the registry software such as the IATI dashboard should be able to handle large files.

2 Likes