Expansion of data types

[pierresegonne]

Hey all, we need your inputs!

🔭🔭 We are taking a look ahead into the future 🔭🔭.

We are looking into how to keep improving the app. Our goal is to become the reference source of electricity data, with the vision that by organising the world's electricity data, we'll drive the transition towards decarbonised grids.
With that in mind, our data coverage must be exhaustive and accurate if we realistically want to achieve our mission.

This is why we're asking your inputs here:

What are in your mind the data types that electricityMap is currently missing? And why?

Feel free to suggest anything relevant, no matter how far off it might be to from the data we currently have. Examples could be: supporting more precise production modes, more exhaustive price data, merit order curves, marginal data etc etc

Looking forward to hearing your thoughts!

[brandongalbraith]

Curtailment data when available from a data provider. Combined with pricing data, this can be used to determine where storage would be most beneficially deployed.

[pierresegonne]

Hey @brandongalbraith, do you know any data sources that openly share that kind of data? 🧐

[VIKTORVAV99]

This has been discussed somewhat in #2368 already and I believe they have a few sources that publish curtailment data in there.

Might be a good place to start.

[brandongalbraith]

@pierresegonne As @VIKTORVAV99 mentions, there are a few sources that currently publish curtailment data (and I'm happy to do the leg work to aggregate those sources if necessary). With that said, my ask would be: what would be the most ideal format for curtailment data for electricitymap to consume? I can then go to various data providers to specifically ask for it in that format.

[pierresegonne]

Super nice!
The perfect scenario would be to have curtailment data per zone, per production mode in MW :)

[brandongalbraith]

Did a bit of research on this, and I retract putting this idea forth at this time. Unfortunately, curtailment data I've found is not as granular as generation mix data (usually provided as excel or pdf files on a monthly basis), and isn't likely to be too beneficial for ingestion by EM.

I'd be willing to revisit this when curtailment data is treated as a first class data citizen by data providers similarly to generation mix data.

[gwpicard]

I think price data would be really nice, namely:

• day-ahead prices
• intraday prices

[VIKTORVAV99]

I think the estimation models, especially forecast/real-time models, would benefit from this kind of data:

• Scheduled and forced production unavailability (both aggregated and per production unit if available).
• Scheduled and forced exchange unavailability.
• Scheduled generation (either per source, unit or total).
• Total load both live and day ahead.
• Real-time and forecasted exchange capacities (different from maximum capacity).

Note
Most if not all of this data is already available on ENTSO-E

And while we already have installed capacities for production modes and exchanges it would be nice to parse this data instead so there is less work spent on doing chores.

EDIT:

It would also be nice if we split up the aggregated production modes so we can get higher granularity, like offshore and onshore wind separate or waste and biomass separate. This would work for pretty much every production mode we currently have except maybe nuclear and geothermal.

[pierresegonne]

Good suggestions thanks :)
FYI we already parse forecasted production / consumption / exchanges

[VIKTORVAV99]

Not a new data type but a improvement/expansion of a existing one.
But I have given it some thought and I think this would be a good structure for the production object that the backend should accept so it can calculate aggregate production and co2 intensities on the backend instead of aggregating production in the parsers:

type production = {
  biomass?: number | {
    biogas?: number;
    biomass?: number;
    waste?: number
  };
  coal?: number | {
    hard?: number;
    lignite?: number;
    peat?: number;
    oil_shale?: number
  };
  gas?: number | {
    fossil?: number;
    coal_derived?: number
  };
  geothermal?: number;
  hydro?: number | {
    run_of_river?: number;
    reservoir?: number
  };
  nuclear?: number | {
    conventional?: number;
    smr?: number
  };
  oil?: number;
  solar?: number | {
    pv?: number;
    thermal?: number
  };
  wind?: number | {
    offshore?: number;
    onshore?: number
  };
  ocean?: number | {
    tidal_stream?: number;
    wave?: number
  };
  unknown?: number;
};

^{TypeScript type.}

This would make all the production properties optional so we easily can add new ones in the future without any impact (if #4086 is merged there should be no unnecessary whitespace or clutter either), will make it easier to write parsers as they only need to return the relevant modes for that region.
The structure also allows all top level keys to accept a number or a object that splits the current aggregated modes to higher granularity as a value. (The current implementation only accept a number so allowing the keys to still accept numbers will make it backwards compatible and should be treated the same as today.)

The benefit of doing it this way is that we can gradually update all parsers that have more granular data in a non blocking way with gradual improvement parser after parser.

Note
Production modes and names are not final and only suggestions, there could be several more added if needed.

[pierresegonne]

That's a really good suggestion!
Adopting something like this might have broader consequences on how the backend works so we'll have to weigh in the pros and cons before trying out the idea. I'll keep the idea in the back of my head, and will see if I find time to make a prototype or something like that with it.

[VIKTORVAV99]

Now I don't know how the backend works specifically but since the state files already returns all the necessary fields (most importantly the co2intensity field per production mode).

So with that information already being processed by the frontend I would assume all that's needed is a step in the backend that aggregates the data in each field, checks if that mode has a dedicated co2intensity (if not fall back to the default) and then calculate the average co2 intensity per category and then run the backend as is today.

Obviously we would need some other changes for the co2intesnity files but I think that would be the easiest implementation at least.