update world_bank scraper

[cjyetman]

Caution: causes many changes/moves in the data CSV

[cjyetman]

Hi, jumping in here to point out that most of the changes are due to the fact that the dataset is now sorted by wb and not by country anymore. Isn't sorting by country again an easy way to reduce the number of changes here? Sorry if I missed sth obvious about this

Yes, but my goal here was to update the scraper with minimal changes, not to update the data (which is typically done all together in a separate process). Just pointing it out for @vincentarelbundock's awareness.

[NilsEnevoldsen]

It does make it hard to see what, if anything, changed as a result.

[vincentarelbundock]

Thanks all. I added one line of code with an arrange to make the diff easier to see. The changes look pretty minimal.

You can merge whenever you want.

[vincentarelbundock]

Do we want curly ' ?

[cjyetman]

I think it is probably not proper, right? But that's what it is in the original data. Should I make the scraper "fix" it?

[vincentarelbundock]

Yeah, I think so. My gut feeling is that converting to nice curly should be the typesetter's job, not a data-level thing.

[cjyetman]

🤔

packageVersion("countrycode")
#> [1] '1.5.0'
countrycode::countrycode("CIV", "iso3c", "country.name")
#> [1] "Côte d’Ivoire"

[cjyetman]

It's hard to see the difference with GitHub's formatting, but both ASCII single quote and UTF curly quote work, and UTF curly quote is what's currently in countrycode::codelist, so I'm assuming leaving it as curly quote is ok, maybe even ideal.

library(countrycode)
packageVersion("countrycode")
#> [1] '1.6.0'

as.hexmode(utf8ToInt("'"))
#> [1] "27"
as.hexmode(utf8ToInt("’"))
#> [1] "2019"

countrycode("Côte d\U27Ivoire", "country.name", "country.name")
#> [1] "Côte d’Ivoire"
countrycode("Côte d\U2019Ivoire", "country.name", "country.name")
#> [1] "Côte d’Ivoire"

stringi::stri_escape_unicode(countrycode("CI", "iso2c", "country.name"))
#> [1] "C\\u00f4te d\\u2019Ivoire"

[cjyetman]

@vincentarelbundock I was under the impression that all of these "getter" scripts got run together in a Docker during some part of the process you do, so I included the changed CSV just to facilitate reviewing the consequences of the changes I made, but with the intention of removing the changed CSV before merging. Looking a bit deeper at things now, it looks like maybe that doesn't happen anymore or the process has changed? Should I included the modified CSV as well, if it changes, when making a change to a getter script?

[vincentarelbundock]

@vincentarelbundock I was under the impression that all of these "getter" scripts got run together in a Docker during some part of the process you do, so I included the changed CSV just to facilitate reviewing the consequences of the changes I made, but with the intention of removing the changed CSV before merging. Looking a bit deeper at things now, it looks like maybe that doesn't happen anymore or the process has changed? Should I included the modified CSV as well, if it changes, when making a change to a getter script?

Yeah, it's been a while, but if I remember correctly, my previous setup was ridiculously over-(and badly-) engineered. So I simplified everything. get_*() saves a CSV file that we keep in the repo. Then, build.R reads and merges all the CSV files.

I always run it on my local machine; never on Docker. Maybe I should do that, but things have worked ok thus far...