Where are logs stored?

[Pballer]

I have a self-hosted Prefect server v3 running with Postgresql DB. I'm curious how logging works.

1. Are the logs stored in Postgresql DB?

2. If yes, is there any cleanup of old logs? For example, could I purge logs over 1 year old?

3. My ECS Fargate task also sends the logs to AWS CloudWatch. It is possible to only store the logs in AWS CW and retrieve them from there?

[zzstoatzz]

hi @Pballer

1. with the open-source server, yes logs are stored in the database.

2. there is no automatic clean-up of old logs in the open source server, but you could implement a periodic job to clean up old logs

3. technically it would be possible, but not without changing how the prefect server works. you could just aggressively clear logs or avoid using the prefect logger in the first place

[Pballer]

@zzstoatzz thank you for the answers.

Just so I'm clear, there is no way to configure Prefect dashboard to pull logs from CloudWatch? That would require a feature request?

[zzstoatzz]

That would require a feature request?

Yes!

[mikelogaciuk]

For example in my company, we clean the logs on daily basis at night using the external job on Kubernetes:

DELETE FROM log WHERE created < (CURRENT_DATE -14);
DELETE FROM flow_run WHERE created < (CURRENT_DATE -14);
DELETE FROM flow_run_state WHERE created < (CURRENT_DATE -14);
DELETE FROM task_run WHERE created < (CURRENT_DATE -14);
DELETE FROM task_run_state WHERE created < (CURRENT_DATE -14);
DELETE FROM events WHERE created < (CURRENT_DATE -14);
DELETE FROM event_resources WHERE created < (CURRENT_DATE -14);
DELETE FROM artifact WHERE created < (CURRENT_DATE -14);

You also have to run VACUUM in order to restore the storage (for each table):

VACUUM FULL log; (...)