Update old wikidata items #971
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
... and updated processing of wikidata_names.json so as to retaing update/fetch time of the translations and drop+refetch those which are older than `wikidata_max_age` but not more than `wikidata_update_limit` items
Full logs: https://github.com/onthegomap/planetiler/actions/runs/10321914925 |
msbarry
reviewed
Aug 7, 2024
There was a problem hiding this comment.
Looks good overall, a couple minor improvements. At first I was a bit concerned this would just drop the first N translations by ID whereas it seems like we want it to drop them somewhat randomly - but it looks like since we write the translation file however it's ordered in a hashmap that it will be randomly sorted to begin with so dropping the first N will be random also 👍
planetiler-core/src/main/java/com/onthegomap/planetiler/util/Wikidata.java
Show resolved
Hide resolved
planetiler-core/src/main/java/com/onthegomap/planetiler/Planetiler.java
Outdated
Show resolved
Hide resolved
planetiler-core/src/main/java/com/onthegomap/planetiler/util/Wikidata.java
Outdated
Show resolved
Hide resolved
planetiler-core/src/test/java/com/onthegomap/planetiler/util/WikidataTest.java
Outdated
Show resolved
Hide resolved
planetiler-core/src/main/java/com/onthegomap/planetiler/Planetiler.java
Outdated
Show resolved
Hide resolved
planetiler-core/src/main/java/com/onthegomap/planetiler/Planetiler.java
Outdated
Show resolved
Hide resolved
e.g. all entries older than `wikidata_max_age` will be fetched again
and load(Path path, Duration maxAge, int updateLimit) made private
and clean-up of load() changes from previous commit
msbarry
reviewed
Aug 8, 2024
planetiler-core/src/main/java/com/onthegomap/planetiler/Planetiler.java
Outdated
Show resolved
Hide resolved
so that without command-line parameters we are fully backward-compatible
|
msbarry
approved these changes
Aug 10, 2024
|
Looks good! Thanks for adding this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When Planetiler is used with
--fetch-wikidatait works roughly like this:wikidata_names.jsondoes not exist, hence all the translations are fetched from WikiDatawikidata_names.jsonexists, so translations are loaded from it and only translations for new OSM elements are fetched from WikiDataGood thing is, that this lowers the load on WikiData and speeds-up tileset generation.
Problem: If some translation changed for existing OSM element, old value from
wikidata_names.jsonis used. If we want to get updates, we can deletewikidata_names.jsonand fetch all translations once again.This PR tries to partially address the problem without the need for deletion (or manual tweaking of)
wikidata_names.json:wikidata_max_age(with default value 0, e.g. "disabled")wikidata_update_limit(with default value 0, e.g. "disabled")wikidata_update_limititems which are older thanwikidata_max_ageWith the defaults Planetiler works as before.
When called with for example
--wikidata-max-age=P30D --wikidata-update-limit=100000, it should then work roughly as follows:wikidata_names.jsondoes not exist, hence all the translations are fetched from WikiDatawikidata_names.jsonexists, so translations are loaded from it and only translations for new OSM elements are fetched from WikiDatawikidata_max_age=P30Dall translations are now considered outdatedwikidata_update_limit=100000(which is roughly 5% of existing translations for full Planet) only up-to 100'000 translations are dropped and fetched from WikiData againOther combination might be
--wikidata-max-age=P30Dand--wikidata-update-limit=0which would keep using the cached translations for a month but the run after a month will drop all (now outdated) translations and fetch them again from WikiData.