Add External Source (and External Event) Attribute Validation

[pranav-super]

• Tickets addressed: MPS-133
• Review: By commit
• Merge strategy: Merge (no squash)

Description

This pull request introduces the feature of external source and event attribute validation. External sources, when uploaded, now bear attributes that describe information about them beyond the basic key, duration, or other standard fields. More miscellaneous, type-specific data, like a source's originator, author, or sequence_number (these are arbitrary examples), or something similar for events, can now be specified. This now means that source's take the following, general format:

{
  "source": {
    "key": "sample_source.json",
    "source_type": "SourceTypeA",
    "valid_at": "2024-001T00:00:00Z",
    "period": {
      "start_time": "2024-001T00:00:00Z",
      "end_time": "2024-007T00:00:00Z"
    },
    "attributes": {
      "serial": 1314321,
      "type": "Class C",
      // other fields...
    }
  },
  "events": [
    {
      "key": "EventTypeA:1/1",
      "event_type": "EventTypeA",
      "start_time": "2024-001T00:00:00Z",
      "duration": "02:00:00",
      "attributes": {
        "projectUser": "Jerry",
        "origin": "Canberra"
        // other fields...
      }
    },
    ...
  ]
}

To constrain these fields, we also introduce attribute schemas for source and event types. Users can submit these when explicitly creating a source or event type here:

The general format for these attribute schemas follows that of a standard JSON Schema. For example, an attribute schema for an event type may look as follows:

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "title": "EventTypeA",
  "description": "Schema for the attributes of EventTypeA.",
  "type": "object",
  "properties": {
    "series": {
      "type": "object",
      "properties": {
        "type": { "type": "string" },
        "make": {"type": "string"},
        "iteration": {"type": "number"}
      }
    }
  }
}

This states that EventTypeA requires (and exclusively allows only) a single field, called series, which is an object with subfields type (a string), make (a string), and iteration (a number).

NOTE: under deliberation is whether a stricter 'metaschema' should be imposed on top of these attributes to better constrain them, beyond simply checking that they are valid JSON Schema. While leaving out such a metaschema more aptly uses the full functionality of JSON Schema, the lack of a metaschema could allow extra junk fields, or the exclusion of fields, like properties.

Verification

These changes were validated with an updated set of gateway tests in e2e-tests.

Documentation

The main documentation that would change is that around the creation of source types, event types, and derivation groups.

Future work

With regards to the schema functionality specifically, the only additional feature that may need to be built and discussed further as an add-on to this is a way to constrain which event types are allowable for a given source type. The need for this and how it should be implemented, namely in the frontend, have yet to be discussed.