Parsing IATI data from JSON using Ruby

SJohns · July 13, 2016, 10:30am

Hello, I’m trying to parse IATI data from JSON format using Ruby rather than Python. Has anyone done this? I’ve got halfway using JSON that I downloaded via OIPA but I’m getting an unexpected token error. Anyone know what that means? This didn’t work (found on stack overflow):

JSON.parse somehow did not like UTF-8-encoded files. Encode the files to ASCII (= ISO 8859-1)

Any help appreciated!

Kasper · April 6, 2016, 6:58pm

I’m pretty sure that @siemvaessen or @VincentVW could help you with that :)…

siemvaessen · April 7, 2016, 10:16am

Hi Sara,

We don’t do Ruby, so can’t help you out there.

When you say ‘halfway using JSON & the unexpected token error’ => do you mean an OIPA unexpected token error or a Ruby error?

Siem

VincentVW · April 7, 2016, 2:48pm

DevTracker is an Sinatra (Ruby) app that runs on the OIPA API, maybe there’s suggestions to be found there; https://github.com/DFID/devtracker-from-api/search?utf8=✓&q=json.parse

I’ll also mention @r_clements here (developer on DevTracker).

r_clements · April 7, 2016, 3:13pm

Hi Sara, you can find some (parsing examples) [https://github.com/DFID/devtracker-from-api/blob/master/helpers/country_helpers.rb] in the helpers directory of the devtracker-from-api code.

I have userd the country_helpers example as I think that that one is pretty intuative but most of the files in the helpers directory would contain at least a few OIPA api calls.

r_clements · April 7, 2016, 3:15pm

[parsing examples] (https://github.com/DFID/devtracker-from-api/blob/master/helpers/country_helpers.rb)

SJohns · April 8, 2016, 1:22pm

Hi @siemvaessen it’s a Ruby error.

SJohns · April 8, 2016, 1:42pm

Hi Ross and Vincent @r_clements @VincentVW, thanks for the link, exactly what I was looking for!

Sarah

markbrough · April 16, 2016, 11:27am

Hey Sarah - I just remembered that I also wrote a little Ruby/Rails application to pull IATI data from the Registry and display it. It’s about 5 years old and the code might not be that nice but this is the main bit that gets data from the Registry and parses it (from XML) into a database – perhaps it is helpful:

github.com

markbrough/IATI-Data/blob/master/app/controllers/iatiregistry_controller.rb

class IatiregistryController < ApplicationController
  def getPackage
    @package = Package.find(params[:id])
	  pu = {}
	  pu[:retrieved] = 1
	  pu[:retrieved_date] = DateTime.now
	  @package.update_attributes(pu)

    # check if this package is already in db
    @thetext ='';
    @activity_exists = Activity.find_by_package_id(@package.packageid)
    if @activity_exists
	@thetext += "<p>The package <b>" + @package.name + "</b> (" + @package.title + ") has already been downloaded.</p>"
    else
	@thetext += "<p>Downloading new package <b>" + @package.name + "</b> (" + @package.title + ").</p>"
    url = @package.resources_url
	require 'net/http'
	require 'rexml/document'

	# get the XML data as a string

This file has been truncated. show original

There’s a very brief write-up on the aidinfo labs site:
https://web.archive.org/web/20110927111130/http://www.aidinfolabs.org/archives/416

Incidentally, that link was via the internet archive - such a shame that the aidinfo labs site has been taken down, there’s a load of useful information in there; can someone from DI try and get it back up again, even just as a historical record?

dalepotter · April 20, 2016, 10:31am

Thanks for flagging this @markbrough - I’ll look into this and report back shortly.