Increase performance of 'dump'

[baurmatt]

Output of restic version

restic 0.9.6 compiled with go1.12.12 on linux/amd64

(Ubuntu 20.04)

What should restic do differently? Which functionality do you think we should add?

There is a major performance difference between using restore and dump. It would be great to get the same performance from both commands.

Restoring the same 50GB file from a object storage showed the following performance difference:

restore: 122 MB/s (Restore time of 50GB: ~7min)
dump: 17 MB/s (Restore time of 50GB: ~50min)

This problem was also discussed in the forum -> https://forum.restic.net/t/performance-differences-between-restic-dump-mount-restore/3878

@MichaelEischer Suggested to open a feature request regarding this topic.

What are you trying to do? What problem would this solve?

I'm trying to restore a LVM snapshot backup from restic repository stored on a object storage:

restic dump latest /vg0-data--snapshot.img > /dev/vg0/data

Others have reported on the forum that they're trying to restore large database backups directly into the database.

Did restic help you today? Did it make you happy in any way?

I discovered that I can do LVM snapshot backups by piping the snapshots directly to restic by leveraging the --stdin and --stdin-filename parameter. Thanks! 👍

[MichaelEischer]

In case anyone wants to help, here are some thoughts of how the performance could be improved:

At least for large files it should be relatively straightforward to improve the performance of the dump command: internal/dump/common.GetNodeData currently iterates over all blocks of a file sequentially. The simplest option would be to just add a bit of code which runs multiple LoadBlob calls in parallel (and makes sure to not mix up the blob order).

A more advanced option would be to keep a cache of limited size to allow reusing blobs if possible. Ideally, the cache would only cache blobs which will be reused in the future, I'm just not sure whether its worth the complexity that would introduce.

[mlew980]

I decided to locally implement your first suggestion and attempted to restore a 4.1 GB file.

restic restore
time to restore: 4m34.161s
md5 checksum: c9ea43abe651810576e6f1c197025f1f

original restic dump
time to restore: 31m15.299s
md5 checksum: c9ea43abe651810576e6f1c197025f1f

parallel restic dump
time to restore (using 8 workers): 6m47.095s
md5 checksum: c9ea43abe651810576e6f1c197025f1f

I don't normally code using Golang so my code might be a bit rough, please don't hesitate to point out any errors:

https://github.com/mlew980/restic/blob/master/internal/dump/common.go

[MichaelEischer]

@mlew980 Could you open a PR for you change? That would make it easier to discuss the code.

From a high-level perspective it is possible that completedJobs grows without bounds. The simplest solution to that is probably to use a buffered job input queue and fill that with e.g. up to 16 jobs relative to the blob that should be written next. Then the output collection loop can wait for loaded blobs, and if blob to write next was loaded, then it can refill the job queue.