Support direct I/O

[UltraCoderRU]

Output of restic version

restic 0.9.6 compiled with go1.12.15 on linux/amd64

What should restic do differently? Which functionality do you think we should add?

When backing up large directories on Linux, restic (or, better say, kernel) actively fills up RAM with disk cache. If target directory size is greater than free RAM size, kernel will begin to move other programs virtual memory pages to swap partition/file. So simple backup task can affect other services and applications performance. I/O cache system in Linux is good in general, but sometimes it is better to avoid caching in particular application. For example, if application reads large amounts of data, that will be not needed in the future (as restic does).

Avoiding disk cache system called direct I/O. The way to enable it depends on the operating system and the support of it depends on the file system. For Linux, one can use O_DIRECT flag in open() system calls. For Windows, there are flags called FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH in CreateFileA. But it is not sufficient to use these flags, because direct I/O requires read/write operations to be aligned to logical sector size. I found two Go-libraries for direct I/O: brk0v/directio and ncw/directio. May be you could use one of them in restic. Btw, I don't know if it will work for metadata reading.

Another way to avoid disk caching, which is often suggested, is POSIX fadvise() system call. But here is an explanation, why it will not work.

I think it would be great to add direct I/O support in restic, if it's even possible. And thank you for a good project! :)

[rawtaz]

@UltraCoderRU Really appreciated that you wrote such a good explanation of the problem you're suggesting a solution for, thanks so much!

[greatroar]

I did some research into this, and I think we have to distinguish between direct input for backup, direct output for the local backend and direct output for the restorer.

Backup. The actual Read calls are done in the chunker. We'd have to buffer input in a custom Reader before feeding it to the chunker. We'd also have to handle the tail end of a file on Linux, where reading must be done in multiples of the block size, in addition to the aligment requirements pointed out in the issue (Mac and BSD don't have hard requirements, but may perform suboptimally when data is not aligned). Also, direct I/O turns off readahead, so it might cause a slowdown.

ncw/directio could help, but it doesn't do buffering and it seems to get the block size subtly wrong. It assumes 4kB is universal, but according to the Linux manpage open(2), we should do an ioctl BLKSSZGET operaton to find out.

alignment restrictions vary by filesystem and kernel version and might be absent entirely. (...) [T]here is currently no filesystem-independent interface for an application to discover these restrictions for a given file or filesystem.

brk0v/directio offers a Writer only.

Local backend. This may be easier, as the changes are localized. We'd still have to buffer, though, as output isn't naturally performed in block-sized chunks. brk0v/directio does this, but it too gets the buffer size wrong, assuming that 512B is universal. ncw/directio doesn't do any buffering.

Restorer. This is much harder than the local backend, since it writes variable-size chunks in the order they come in over the network. I don't see an easy way of buffering here.

[greatroar]

Worth a read: Linus on direct I/O. In short, let's not do this. I suggest closing this issue.

[ifedorenko]

Worth a read: Linus on direct I/O. In short, let's not do this. I suggest closing this issue.

The problem the user wants to solve is restic backup and restore fill operating system page cache with data that is read/written only once. Which affects performance of other applications running on the system. Whether restic uses direct I/O or finds some other means to solve this problem, is secondary I think.

[greatroar]

Well, the issue title is "Support direct I/O" and the conclusion is "it would be great to add direct I/O support in restic", so I took that to be the gist of it. I don't deny that the cache flushing is a problem.

Anyway, the question is, what should be done about that problem? Direct I/O is very tricky and non-portable. posix_fadvise doesn't have the option we need (on Linux) and FADV_DONTNEED is a remedy worse than the disease (#666, #670). Mmapped I/O?

[MichaelEischer]

restic reads the file content only once, which should be enough to let the page cache decide to not prioritize keeping these pages in memory. I don't think there's anything we can do about the directory metadata used by the filesystem which will end up getting loaded.

So let's close this issue, unless someone has a good idea which would improve the status quo.