Cleanup sensitive data from KEB db for all deprovisioned clusters

[szwedm]

Items from operations and runtime states must be removed when the instance is fully deprovisioned.
There must be a time, when the items are kept for investigation (defined retention period)

[ukff]

Check impact of cleanup on current metrics and if needed adjust implementation to handle it
It must be done before we will run cleanup.

https://github.com/orgs/kyma-project/projects/38/views/1?pane=issue&itemId=53104649

[piotrmiskiewicz]

Solution:

New entity, where we store all necessary non-sensitive data after deprovisioning.

Subtask:

• define model and a storage layer
• add a logic which creates the archived instnace at the end of deprovidioning process and delete operations/runtimestates (with feature flag) PR: Archiving and cleaning in the deprovisioning process #498
• implement getting archived instances in the runtimes endpoint for deprovisioned instance PR: Fetch instance archived runtimes endpoint #527
• create a job which will be run on a migration time (feature flag) and archive all deprovisioned instances
• documentation
• enable archiving and cleaning, run the migration job

[ebensom]

The behavior of existing {prefix}_operations_*_total statistics count metrics should be preserved, as they are used as high-level SLIs to track success ratio of provision/deprovision operations (and update in the near future). So the new instances_archived model IMO should be able to reflect the provisioning state (failed or succeeded), the count of deprovisioning and update operations per state, so that the stats collector could use these.

[ebensom]

The track-record of operations for a deprovisioned instance could also be useful for post-mortem troubleshooting. How about keeping all the operations records after deprovisioning, just cleaning up the sensitive part (present in data and provisioning_parameters columns)?

[piotrmiskiewicz]

We are planning to use increase prometheus function. It looks we can use it instead of using absolute values.

[piotrmiskiewicz]

The release scenario:

1. Enable Archiving on DEV, verify if the deprovisioning works fine (no errors, KCP CLI shows such deprovisioned runtime - -S deprovisioned)
2. Enable Cleaning on DEV, verify if the deprovisioning works fine (no errors, KCP CLI shows such deprovisioned runtime - -S deprovisioned)
3. Run the archiver job on DEV, check if archived (by the job) runtimes are visible by KCP CLI
4. Enable archiving on STAGE
5. Enable cleaning on STAGE
6. Run the archiver on STAGE
7. Wait few days to check if there are any problems on DEV/STAGE
8. Enable archiving on PROD
9. Enable cleaning on PROD
10. Run archiver on PROD

Enabling archiving and cleaning in the deprovisioning process is defined in a yaml files, see: https://github.tools.sap/kyma/management-plane-config/pull/5068/files

[piotrmiskiewicz]

fullfill change requirements, see: https://pages.github.tools.sap/kyma/docusaurus-docs/kyma/ops/change_process/

[piotrmiskiewicz]

waiting for: https://github.tools.sap/kyma/backlog/issues/5258

[piotrmiskiewicz]

STAGE (before running):

select count(*) from operations;
 count
-------
 94107
(1 row)

broker=> select count(*) from operations where instance_id not in (select instance_id from instances);
 count
-------
 63625
(1 row)


broker=> select count(*) from runtime_states;
 count
--------
 124967
(1 row)

after running archiver (but before enabling archiving and cleaning):

broker=> select count(*) from operations;
 count
-------
 30549
(1 row)

broker=> select count(*) from operations where instance_id not in (select instance_id from instances);
 count
-------
    59
(1 row)

broker=> select count(*) from runtime_states;
 count
-------
 27604
(1 row)