[RFC] Splitting zones.json into multiple files

[madsnedergaard]

Description

Currently, config/zones.json holds a lot of information about a zone.

In most use cases, we only need to import something about a zone and not everything.

Secondly, we it's currently more difficult to automate the review process because of this (we can't auto-apply labels as easily).

Motivation for changing this

• Could make it easier for contributors to update capacities
• Would make it easier for us to add specific labels based on file changes (without adding/creating a new bot that deep searches in changed files)
• Could make it simpler for us/people to review PRs (hypothesis, not sure how much it matters)

The Proposal: Split into multiple files

I think splitting the file into multiple files would help with the above. Here's how it could be done (let me know if I missed something):

config/
	zones.json
		- name
		- timezone
		- bounding_boxes
		- flag_file_name
		- subZoneNames
	capacities.json
		- capacity
		- contributors
	parsers.json
		- parsers
		- contributors
		- delays

Downsides

• Would require merging contributors from two files
• ???

Tasks

• Split files
• Split schemas
• Update wiki and READMEs
• Setup labeler bot to add capacity label

---

Dunno if this should be an issue, but what do you think @Kongkille, @FelixDQ and anyone else?

[Kongkille]

I think this makes a lot of sense, @FelixDQ spent some time also on these config files so I'm curious whether there are any downsides to consider.

[FelixDQ]

I think this makes a lot of sense. zones.json is definitely getting too big. One place where this gets a bit tricky is exchanges.json which also has capacities and parsers. Could we split the files in a way that makes sense regarding those also?

[skovhus]

I’m unsure about the actual value of splitting this. In terms of tooling, we would likely require a custom bot anyway... In terms of contribution, then I see some value that information for one zone is colocated in one file (currently zones.json).

If we want to split, then we could considering if one file per zone would be better.

On a related note: wouldn’t it make sense if the references (currently in DATA_SOURCES.md would be located in zones.json or the country zone files). We can always auto generate DATA_SOURCES on merge if we like to have this in a markdown file.

[FelixDQ]

Yes, I think the idea of the .md file was to make it easy for non technical people to find the data source. So autogenerating it would also solve this problem. The only concern would again be the size of zones.json

[Kongkille]

I really like the idea of having the editing of data sources in a JSON file as well

[gwpicard]

@skovhus Would it make sense to turn this into an issue? I think it would be great to have references/sources specified for each zone within the zones.json file, and it would also make aspects of data reporting easier to automate.

[skovhus]

Yes, if you find it valuable, then I think we should do it. Note that there might be some parts of the data source file that refers to the other .json files (like exchanges.json).

[veqtrus]

If we end up doing this I would suggest using camelCase keys. Ping me if you would like me to update the schemata.

I think a contributors array per zone in zones.json alone would be enough.

[skovhus]

If we end up doing this I would suggest using camelCase keys.

Good point!

Edit: I think a single contributors array in zones.json would be enough.

So contributors array for all zones? In that case, we could just auto-extract this information from git.

The value from having it pr zone is being able to credit the contributor directly on a specific zone.

[veqtrus]

Edit: I think a single contributors array in zones.json would be enough.

So contributors array for all zones? In that case, we could just auto-extract this information from git.

The value from having it pr zone is being able to credit the contributor directly on a specific zone.

Sorry, I realised what I wrote wasn't clear. I meant one array per zone, but only in zones.json, not separately in capacities.json and parsers.json, unless it would be shown separately on the website.

[VIKTORVAV99]

If we do decide to split these (which it think we should) we should try to do it in a way so we include as little unnecessary data in the frontend as possible.

As far as I know the only keys from zones.json used in the frontend are: contributors, flag_file_name, bounding_boxes, timezones and disclaimer.

For exchanges.json only the lonlat and rotation keys are used in the frontend.

The other keys are either not used at all for the frontend or they are passed through it via the history file, like the capacities for both zones and exchanges so we should try and keep these in separate files to ensure we don't bloat the frontend unnecessarily.

Another option would be to include all of these in the state and/or history information (most already are) for each country and skip using the zones.json in the frontend at all and just keep the exchanges.json in the frontend. Then we can split them however we see fit.

Both of these options can be combined to some degree as well.

[VIKTORVAV99]

I created a different RFC at #4396 for historical capacities that directly relates to this discussion.
Basically I want to split out the capacities into a new file, see the RFC for more details.

[VIKTORVAV99]

zones.json and exchanges.json was split #4435 and the old zones.json and exchange.json will be removed in #4655.