Bring Your Own Data

[Bonkles]

Description

Over the past couple of years we've fielded requests to have folks add their own GPX trace data, geojson data, or vector tile data to the map. This issue will likely be wide-ranging and I encourage folks in our user community to comment with specific use cases that you are interested in seeing come to fruition.

Currently, dragging and dropping Geojson onto the map will display the geojson in a non-editable layer. You can see it, but you can't do anything with it. If the user brought some .geojson onto the map, we might make a wizard-style importer that says:

1. Are you interested in bringing this data into the map?
2. What kind of data is it?
3. What OSM tags would you like to attach to the data?

We may want to include some sort of assertion that the data being brought is licensed correctly and/or owned by the uploader.

[devdattaT]

You could also have templates for the data, such that tags get carried over. Say the input geojson has properties with keys of highway and access. Then the way which gets created, automatically gets those tags, and values.

[devdattaT]

Our Use case: I work for an organisation that has a large fleet of vehicles which send GPS pings every 5 seconds. We internally use OSM for routing, and we have a process which identifies missing roads.

We want to share this back to OSM, but can't share our pings, or these missing roads directly.
What we will like to do, is to have a team of mapmakers, who use these internal sources, and edit the main OSM API, using a feature such as the one described in the OP.

[jakewarren]

we might make a wizard-style importer

Just my two cents but I would prefer that importing operates more or less identical to the existing workflow for other RapiD sources such as Facebook Roads or Microsoft Buildings. Or in other words, RapiD would provide assistance for the user to quickly manually import specific entities since they would likely also be manually conflating the data. For large scale automated imports I feel that a tool like JOSM is more appropriate but that is just my opinion. This type of feature would be incredibly helpful for small scale manual imports.

I have created a crude mock-up to help illustrate better:

In this mockup the user has selected an object from a GeoJSON file. If the user clicks the 'Use this feature' button, a node/way is then created using the tags from the GeoJSON file. The changeset is then reviewed and submitted via the normal workflow.

[tordans]

In the spirit of sharing UseCases and Workflows, here are a few UseCases that I don‘t have an easy answer to … which RapID could help with and make a lot easier.

Building Footprints/Address

Berlin and Brandenburg both have OpenData about Buildings (including building levels) and Addresses. They are not suited for an import, since they only represent what the Cities know about the buildings officially. But they are a great starting point. It‘s the same with addresses, they need reviewing to be placed correctly (or not at all in cases where there are no houses and no construction on the ground).

AFAIK this is basically what the ESRI cooperation handles; but I never figured out how to get this data into this pipeline to use the core RapID features to handle them.

A „bring your own data“-RapID workflow could looks like this: We pre-process the OpenData in QGIS to filter it down to only missing data. We host a GeoJSON with the current version of that processing. We have a RapID URL with that hosted GeoJSON to start mapping. RapID would ideally allow some process to easily copy the external Data in OSM. A more fancy approach would be to prepare the data with https://github.com/ngageoint/hootenanny (at least I think this is a tool for this use case) to improve shapes and position of existing buildings., but I never looked into this.

General UseCase: Increase Trust in OSM Data

To share a motivation that is driving a lot of work with external data sources:

We work with administrations in Berlin and other Regions in Germany to help them use processed OSM data for their decision making. An important topic is, to build trust in OSM data in terms of „up to date ness“ and „completeness“. One proxy for those is to take different OpenData sets and compare them with OSM data to create tasks to check the Data. We use this for a number of UseCases. But the Workflow is complex and disconnected.

• Zebra Crossings: We took two different open datasets, created a delta to OSM with buffer in QGIS, hosted the GeoJSON and created a MapRoulette Challenge. Each task was a zebra crossing that the city claims exists. Results where: Some where missing and added in OSM. Some where mistagged and corrected in OSM. Others where very badly positioned in official data, so OSM based on areal images was more precise. Other could not be found on areal images or Mapillary at all.

Detailed links (MapRoulette, Data, Processing) via https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-Zebrastreifen and https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-Zebrastreifen-Strassenbefahrung

• Parklets: There is OpenData on Parklets in Berlin. We use the same process to add them. Since there is no Preset for Parklets, yet, we have a task list in MapRoulette that Mappers should follow when editing in ID/JOSM which is not a great way to explain what and how to map.

Detailed links (MapRoulette, Data, Processing) via https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-Parklets

• Driveways: For our parking processing we process driveways (which translate to no parking for a few meters). Berlin has old OpenData on those. We use the same process described above.

We did not document this one, yet. Maproulette is at https://maproulette.org/browse/challenges/28466

• Road side parking: Same as above but for Places where parking is only allowed in small areas next to the road. Again, the broader topic is to use the open data available and check it on the ground or with Mapillary to only import what we can tell is still present and still correct. In the end we can claim to have validated OSM data against the best official data available and checked that data in the process.

Detailed links (MapRoulette, Data, Processing) via https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-parkbuchten

• Drinking water: Same story but different Group and Data: The goal is to make user all public drinking water places are in OSM and are well tagged.

https://atiptap.org/projekte/wasserwende/openstreetmap/, https://maproulette.org/browse/challenges/28303

Help with data preparation

A big issue with all those topics is preparing the external Data to create tasks for OSM. For most of the simpler datasets this can be done with QGIS with a few steps:

take data > take the center point > make a buffer > create OSM data > take the center point > make a buffer > find intersections of both > filter those without intersection > recreate point data of the buffer > polish the data properties (I like adding a custom task property with dynamic Mapillary Lookup that I use in MapRoulette as a Task Description).

There is no tooling out there, that makes this easier AFAIK. There is also a lack of direct tutorials on this stuff. So my nudge is, that improving external data handling in RapID might also be about improving the pre processing or creating some educational content (screenshare-sessions) on how to prepare data well.

General Idea: MapRoulette integration

Most of the Examples above use MapRoulette to store the status of Tasks and an overall Status of the Project. I like the fact that MR allows user to use the editor they like but still work on the same Tasklist. And that I get an overall status of the progress. So I wonder if a feature to handle external data in ID could be focused on integrating MR. At some point, I hope more editor – especially on the ground editors, see maproulette/maproulette3#1737 – will pick up this theme and provide a data validation / adding experience that is even closer to the task in question (like: I am standing next to it…).

General Idea: Merge external and OSM Data

A low hanging fruit would be to make it easier to copy a set of tags that are provided within the external data to copy into an OSM object. That could be a „osm_tags“ property of a geoJSON (multilingual or array). Or just making the RapId UI a DIV instead of the table of readonly input fields.

More fancy would be to provide a UI to do some custom mapping, but that becomes too complex very fast… Even more fancy would be a way to select OSM + External Objects and have a merge UI/Action. (UIs that merge Address-Data could come to mind for this.)

General Idea: List next task nearby

One thing I missing when working with external Data as an overlay is something that will guide me to the next piece of data that needs adding. MapRoulette has a „next task nearby“ feature, but that external. External Data in ID can only be navigated visually (right?), so whenever the data is not dense (but spread around the city) it is hard to find the next „task“.

---

Hope that provides some inspiration.

[Gustavo22Soaresh]

One thing I missing when working with external Data as an overlay is something that will guide me to the next piece of data that needs adding. MapRoulette has a „next task nearby“ feature, but that external. External Data in ID can only be navigated visually (right?), so whenever the data is not dense (but spread around the city) it is hard to find the next „task“.

I believe that for this case it would be better if you didn't have to reload the ID or RapiD completely with each new task and also it would be nice to support Maprolette Cooperative Challenges

[skunkman56]

This would be a very helpful option, to specify another Open Data API to pull from and assist with adding authoritative data

[tsmock]

The JOSM MapWithAI plugin has the ability to get data and add data from additional sources.

We should probably set down and put together a format for that -- right now, the plugin reads https://github.com/JOSM/MapWithAI/blob/pages/json/sources.json for all sources, and I don't know if the json format makes sense for Rapid, and it currently isn't set up for data sources that haven't set up mappings for their keys to OSM keys.

[cbeddow]

One very simple piece of this I want to suggest focus on is what @tordans mentions, an "osm_tags" property.

For example, if the user opens a local geojson in Rapid, it could have any format, and the geometry is displayed. however, if the Rapid editor senses it has "osm_tags", it should then enable the use/ignore buttons and be suggesting any tags in that. Could maybe do a tag lint check, but mostly just trust the input as-is. So format would look like:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "amenity": "restaurant",
        "name": "Casa d'Oro",
        "cuisine": "Italian",
        "addr:city": "Modena"
      },
      "geometry": {
        "coordinates": [
          10.45683300592367,
          46.06594839555103
        ],
        "type": "Point"
      }
    }
  ]
}

A PMTile, a vector tile endpoint, CSV or other filetypes could have the same expectation. Any key/value in the properties will be converted to OSM tags. If it does not exist in the presets... that's another issue. We could argue that at least 1 of the key/value pairs should be matching the presets or it throws an error, or could throw an error with an override option.

If a tile layer is used then the changeset should reference that tile layer as a source, and could otherwise reference the geojson filename as a source (which may be not useful anyway, but helps establish some possible trail of source evidence in case reverts or license checks are needed).

[baditaflorin]

I would love to suggest adding this so we can make the IDEditor as a simple way where other websites can point to this as an official way to add informations into OpenStreetMap in a simple way
openstreetmap/iD#10247

[k-yle]

Hey @bhousel, following on from openstreetmap/iD#10407 (comment), if RapiD were to support custom data, would it only support creating new features, or would it also allow editing/moving/deleting features?

If the plan is to only support creating features, I think this could be implemented already (?) since there's no ambiguity around converting a geojson file into osm features

[tordans]

if RapiD were to support custom data, would it only support creating new features, or would it also allow editing/moving/deleting features?

@k-yle I think a good way to go into more details is to write down the usage scenario behind those features.
Most Szenarios I have in mind fit the use case of a MapRoulette + Fix-Tag Featureset where users can quickly work down a list of tasks but the whole thing is user driven and guided; every step can be checked.

Thinking about a different example:
If I where to take lokal Berlin Building OpenData and either filter them down to only missing buildings (only create) or even do some https://github.com/ngageoint/hootenanny (or something similar) processing to update existing geometries (create+modify). I would still want a way to quickly check and accept/reject each change, to validate the data, right? So ideally this dataset would be placed into the quick-check-import-or-reject-layer that Rapid uses for the regular rapid datasets.

If this separate layer where not there and the loaded data where merged into the loaded OSM data, how would I see which objects are modified, what the modifications (tags, geometries; similar to OSMCha?), and would there be a way to list them? I never saw how JOSM does this kind of thing but I image the changes would be more visible there?

I just read about this planned import of some official data of one state in Germany https://wiki.openstreetmap.org/wiki/Organised_Editing/Activities/Entwicklungsagentur_Rheinland-Pfalz#Geplant (and I want to learn more about what they are doing) but it sounds like they did all this checking and validating beforehand and now have a "finished" file that they only plan to upload(?). IMO all the hard parts to check and merge the data happened elsewhere. But it sounds like this would be the usecase for the feature from openstreetmap/iD#10407, right? It's just, that I am not sure that this is really a win for Rapid/iD, especially given how parts of the community tends to talk about tools that are not JOSM and work in this area.

[bhousel]

Hey @bhousel, following on from openstreetmap/iD#10407 (comment), if RapiD were to support custom data, would it only support creating new features, or would it also allow editing/moving/deleting features?

We're open to anything, as long as people find it useful.
I guess ingesting an osmchange file would cover all those situations. As far as I know, people mostly use JOSM to work with these currently.

[k-yle]

@k-yle I think a good way to go into more details is to write down the usage scenario behind those features.

@tordans for the NZ address import, this is the process (using RapiD):

• the user opens our fork of RapiD
• they load a "task", which is a geojson file.
• An example geojson file might contain: 2 features to be edited, 1 node to be created, and 3 to be deleted
• The user continually presses Zoom to Next (shortcut G) to open the next feature in the geojson file
• Once the next geojson feature is selected, they can accept the tag-fix or ignore it. Same process for creating or deleting a feature

For my use-case, it would amazing if RapiD allowed the user to import their own geojson (or osmChange) file, and showed a prompt like this for each feature.

This use case consists of 3 distinct features:

1. importing your own data as if it were a ESRI datset (Bring Your Own Data #585)
2. supporting tags changes and deleting features (discussed in Bring Your Own Data #585)
3. adding a panel or shortcut so users can jump to the next closest feature in the dataset (no issue)