Media store can leave many empty directories, leading to large disk space usage

[Twi1ightSparkle]

Description

On my server, the directory media_store/remote_content currently has 567 746 empty sub directories (takes about 2.5 GB of disk space)

Happy to share logs in a non-public channel or give access to my server for debugging.

Some homeservers seems more affected than others.

Steps to reproduce

• uh, have a Synapse running for six months

Version information

• Homeserver: twilightsparkle.dev

If not matrix.org:

• Version: 1.15.0

• Install method: pip

• Platform: Ubuntu Server 18.04.4 LTS (4.15.0-106-generic x86_64)

VPS (running on KVM I believe)

[clokep]

This is probably due to the media pruning script not trying to find empty directories to delete. Unfortunately, this is harder to fix than it seems due to race conditions with uploads. (As background the OS takes 4 KiB or something for each empty directory, so 4 KiB * 567,746 = 2.2 GiB or so.)

It should safe to manually delete directories that are empty if you're not uploading any media / have Synapse stopped.

[clokep]

It could also be beneficial to have the media store create less directories in the future.

[richvdh]

another cause for this is that we create directories for remote media before we check that the remote host is reachable (and then never have cause to delete them). This one should be relatively easy to fix - we can just (try to) delete the directory after a failed download.

[skepticalwaves]

is a manual operation like

find /matrix/synapse/storage/media-store/ -empty -type d -delete

Safe to run to clean up?

[Twi1ightSparkle]

is a manual operation like

find /matrix/synapse/storage/media-store/ -empty -type d -delete

Safe to run to clean up?

I did something similar to this back when I had this issue. Nothing exploded. But this was a long time ago so can't say how it would work today.
When I did it, I had Synapse stopped, and I made a backup of the media directory first just in case

[richvdh]

in theory you could race with synapse creating a new directory, which could then cause a media request to fail. But that's not exactly the end of the world.

[skepticalwaves]

I'll use it the next time I'm down for maintenance. Thanks.

[clokep]

It looks like #10924 fixes this for URL cache directories, but those are a bit different since they are time based.

[marius851000]

Another issue that this, in addition to leading to usage of disk space, is that it can make a filesystem run out of inode (like what happened on my homeserver).