restic 0.17.0

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

release notes

Changelog for restic 0.17.0 (2024-07-26)
=======================================
The following sections list the changes in restic 0.17.0 relevant to restic users. The changes are ordered by importance.
Summary

Fix #3600: Handle unreadable xattrs in folders above backup source
Fix #4209: Fix slow SFTP upload performance
Fix #4503: Correct hardlink handling in stats command
Fix #4568: Prevent forget --keep-tags <invalid> from deleting all snapshots
Fix #4615: Make find not sometimes ignore directories
Fix #4656: Properly report ID of newly added keys
Fix #4703: Shutdown cleanly when receiving SIGTERM
Fix #4709: Correct --no-lock handling of ls and tag commands
Fix #4760: Fix possible error on concurrent cache cleanup
Fix #4850: Handle UTF-16 password files in key command correctly
Fix #4902: Update snapshot summary on rewrite
Chg #956: Return exit code 10 and 11 for non-existing and locked repository
Chg #4540: Require at least ARMv6 for ARM binaries
Chg #4602: Deprecate legacy index format and s3legacy repository layout
Chg #4627: Redesign backend error handling to improve reliability
Chg #4707: Disable S3 anonymous authentication by default
Chg #4744: Include full key ID in JSON output of key list
Enh #662: Optionally skip snapshot creation if nothing changed
Enh #693: Include snapshot size in snapshots output
Enh #805: Add bitrot detection to diff command
Enh #828: Improve features of the repair packs command
Enh #1786: Support repositories with empty password
Enh #2348: Add --delete option to restore command
Enh #3067: Add extended options to configure Windows Shadow Copy Service
Enh #3406: Improve dump performance for large files
Enh #3806: Optimize and make prune command resumable
Enh #4006: (alpha) Store deviceID only for hardlinks
Enh #4048: Add support for FUSE-T with mount on macOS
Enh #4251: Support reading backup from a command's standard output
Enh #4287: Support connection to rest-server using unix socket
Enh #4354: Significantly reduce prune memory usage
Enh #4437: Make check command create non-existent cache directory
Enh #4472: Support AWS Assume Role for S3 backend
Enh #4547: Add --json option to version command
Enh #4549: Add --ncdu option to ls command
Enh #4573: Support rewriting host and time metadata in snapshots
Enh #4583: Ignore s3.storage-class archive tiers for metadata
Enh #4590: Speed up mount command's error detection
Enh #4601: Add support for feature flags
Enh #4611: Back up more file metadata on Windows
Enh #4664: Make ls use message_type field in JSON output
Enh #4676: Make key command's actions separate sub-commands
Enh #4678: Add --target option to the dump command
Enh #4708: Back up and restore SecurityDescriptors on Windows
Enh #4733: Allow specifying --host via environment variable
Enh #4737: Include snapshot ID in reason field of forget JSON output
Enh #4764: Support forgetting all snapshots
Enh #4768: Allow specifying custom User-Agent for outgoing requests
Enh #4781: Add restore options to read include/exclude patterns from files
Enh #4807: Support Extended Attributes on Windows NTFS
Enh #4817: Make overwrite behavior of restore customizable
Enh #4839: Add dry-run support to restore command

Details


Bugfix #3600: Handle unreadable xattrs in folders above backup source
When backup sources are specified using absolute paths, backup also includes information about the parent folders of the backup sources in the snapshot.
If the extended attributes for some of these folders could not be read due to missing permissions, this caused the backup to fail. This has now been fixed.
#3600 #4668 https://forum.restic.net/t/parent-directories-above-the-snapshot-source-path-fatal-error-permission-denied/7216


Bugfix #4209: Fix slow SFTP upload performance
Since restic 0.12.1, the upload speed of the sftp backend to a remote server has regressed significantly. This has now been fixed.
#4209 #4782


Bugfix #4503: Correct hardlink handling in stats command
If files on different devices had the same inode ID, the stats command did not correctly calculate the snapshot size. This has now been fixed.
#4503 #4006 https://forum.restic.net/t/possible-bug-in-stats/6461/8


Bugfix #4568: Prevent forget --keep-tags <invalid> from deleting all snapshots
Running forget --keep-tags <invalid>, where <invalid> is a tag that does not exist in the repository, would remove all snapshots. This is especially problematic if the tag name contains a typo.
The forget command now fails with an error if all snapshots in a snapshot group would be deleted. This prevents the above example from deleting all snapshots.
It is possible to temporarily disable the new check by setting the environment variable RESTIC_FEATURES=safe-forget-keep-tags=false. Note that this feature flag will be removed in the next minor restic version.
#4568 #4764


Bugfix #4615: Make find not sometimes ignore directories
In some cases, the find command ignored empty or moved directories. This has now been fixed.
#4615


Bugfix #4656: Properly report ID of newly added keys
restic key add now reports the ID of the newly added key. This simplifies selecting a specific key using the --key-hint key option.
#4656 #4657


Bugfix #4703: Shutdown cleanly when receiving SIGTERM
Previously, when restic received the SIGTERM signal it would terminate immediately, skipping cleanup and potentially causing issues like stale locks being left behind. This primarily effected containerized restic invocations that use SIGTERM, but could also be triggered via a simple killall restic.
This has now been fixed, such that restic shuts down cleanly when receiving the SIGTERM signal.
#4703


Bugfix #4709: Correct --no-lock handling of ls and tag commands
The ls command never locked the repository. This has now been fixed, with the old behavior still being supported using ls --no-lock. The latter invocation also works with older restic versions.
The tag command erroneously accepted the --no-lock command. This command now always requires an exclusive lock.
#4709


Bugfix #4760: Fix possible error on concurrent cache cleanup
If multiple restic processes concurrently cleaned up no longer existing files from the cache, this could cause some of the processes to fail with an no such file or directory error. This has now been fixed.
#4760 #4761


Bugfix #4850: Handle UTF-16 password files in key command correctly
Previously, key add and key passwd did not properly decode UTF-16 encoded passwords read from a password file. This has now been fixed to correctly match the encoding when opening a repository.
#4850 #4851


Bugfix #4902: Update snapshot summary on rewrite
Restic previously did not recalculate the total number of files and bytes processed when files were excluded from a snapshot by the rewrite command. This has now been fixed.
#4902 #4905


Change #956: Return exit code 10 and 11 for non-existing and locked repository
If a repository does not exist or cannot be locked, restic previously always returned exit code 1. This made it difficult to distinguish these cases from other errors.
Restic now returns exit code 10 if the repository does not exist, and exit code 11 if the repository could be not locked due to a conflicting lock.
#956 #4884


Change #4540: Require at least ARMv6 for ARM binaries
The official release binaries of restic now require at least ARMv6 support for ARM platforms.
#4540 #4542


Change #4602: Deprecate legacy index format and s3legacy repository layout
Support for the legacy index format used by restic before version 0.2.0 has been deprecated and will be removed in the next minor restic version. You can use restic repair index to update the index to the current format.
It is possible to temporarily reenable support for the legacy index format by setting the environment variable RESTIC_FEATURES=deprecate-legacy-index=false. Note that this feature flag will be removed in the next minor restic version.
Support for the s3legacy repository layout used for the S3 backend before restic 0.7.0 has been deprecated and will be removed in the next minor restic version. You can migrate your S3 repository to the current layout using RESTIC_FEATURES=deprecate-s3-legacy-layout=false restic migrate s3_layout.
It is possible to temporarily reenable support for the s3legacy layout by setting the environment variable RESTIC_FEATURES=deprecate-s3-legacy-layout=false. Note that this feature flag will be removed in the next minor restic version.
#4602 #4724 #4743


Change #4627: Redesign backend error handling to improve reliability
Restic now downloads pack files in large chunks instead of using a streaming download. This prevents failures due to interrupted streams. The restore command now also retries downloading individual blobs that could not be retrieved.
HTTP requests that are stuck for more than two minutes while uploading or downloading are now forcibly interrupted. This ensures that stuck requests are retried after a short timeout.
Attempts to access a missing or truncated file will no longer be retried. This avoids unnecessary retries in those cases. All other backend requests are retried for up to 15 minutes. This ensures that temporarily interrupted network connections can be tolerated.
If a download yields a corrupt file or blob, then the download will be retried once.
Most parts of the new backend error handling can temporarily be disabled by setting the environment variable RESTIC_FEATURES=backend-error-redesign=false. Note that this feature flag will be removed in the next minor restic version.
#4627 #4193 #4515 #1523 #4605 #4792 #4520 #4800 #4784 #4844


Change #4707: Disable S3 anonymous authentication by default
When using the S3 backend with anonymous authentication, it continuously tried to retrieve new authentication credentials, causing bad performance.
Now, to use anonymous authentication, it is necessary to pass the extended option -o s3.unsafe-anonymous-auth=true to restic.
It is possible to temporarily revert to the old behavior by setting the environment variable RESTIC_FEATURES=explicit-s3-anonymous-auth=false. Note that this feature flag will be removed in the next minor restic version.
#4707 #4908


Change #4744: Include full key ID in JSON output of key list
The JSON output of the key list command has changed to include the full key ID instead of just a shortened version of the ID, as the latter can be ambiguous in some rare cases. To derive the short ID, please truncate the full ID down to eight characters.
#4744 #4745


Enhancement #662: Optionally skip snapshot creation if nothing changed
The backup command always created a snapshot even if nothing in the backup set changed compared to the parent snapshot.
Restic now supports the --skip-if-unchanged option for the backup command, which omits creating a snapshot if the new snapshot's content would be identical to that of the parent snapshot.
#662 #4816


Enhancement #693: Include snapshot size in snapshots output
The snapshots command now prints the size for snapshots created using this or a future restic version. To achieve this, the backup command now stores the backup summary statistics in the snapshot.
The text output of the snapshots command only shows the snapshot size. The other statistics are only included in the JSON output. To inspect these statistics use restic snapshots --json or restic cat snapshot <snapshotID>.
#693 #4705 #4913


Enhancement #805: Add bitrot detection to diff command
The output of the diff command now includes the modifier ? for files to indicate bitrot in backed up files. The ? will appear whenever there is a difference in content while the metadata is exactly the same.
Since files with unchanged metadata are normally not read again when creating a backup, the detection is only effective when the right-hand side of the diff has been created with backup --force.
#805 #4526


Enhancement #828: Improve features of the repair packs command
The repair packs command has been improved to also be able to process truncated pack files. The check and check --read-data command will provide instructions on using the command if necessary to repair a repository. See the guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html for further instructions.
#828 #4644 #4882


Enhancement #1786: Support repositories with empty password
Restic previously required a password to create or operate on repositories. Using the new option --insecure-no-password it is now possible to disable this requirement. Restic will not prompt for a password when using this option.
For security reasons, the option must always be specified when operating on repositories with an empty password, and specifying --insecure-no-password while also passing a password to restic via a CLI option or environment variable results in an error.
The init and copy commands add the related --from-insecure-no-password option, which applies to the source repository. The key add and key passwd commands add the --new-insecure-no-password option to add or set an empty password.
#1786 #4326 #4698 #4808


Enhancement #2348: Add --delete option to restore command
The restore command now supports a --delete option that allows removing files and directories from the target directory that do not exist in the snapshot. This option also allows files in the snapshot to replace non-empty directories having the same name.
To check that only expected files are deleted, add the --dry-run --verbose=2 options.
#2348 #4881


Enhancement #3067: Add extended options to configure Windows Shadow Copy Service
Previous, restic always used a 120 seconds timeout and unconditionally created VSS snapshots for all volume mount points on disk. This behavior can now be fine-tuned by the following new extended options (available only on Windows):

-o vss.timeout: Time that VSS can spend creating snapshot before timing out (default: 120s) - -o vss.exclude-all-mount-points: Exclude mountpoints from snapshotting on all volumes (default: false) - -o vss.exclude-volumes: Semicolon separated list of volumes to exclude from snapshotting - -o vss.provider: VSS provider identifier which will be used for snapshotting

For example, change VSS timeout to five minutes and disable snapshotting of mount points on all volumes:
Restic backup --use-fs-snapshot -o vss.timeout=5m -o vss.exclude-all-mount-points=true
Exclude drive d:, mount point c:\mnt and a specific volume from snapshotting:
Restic backup --use-fs-snapshot -o vss.exclude-volumes="d:;c:\mnt;\?\Volume{e2e0315d-9066-4f97-8343-eb5659b35762}"
Uses 'Microsoft Software Shadow Copy provider 1.0' instead of the default provider:
Restic backup --use-fs-snapshot -o vss.provider={b5946137-7b9f-4925-af80-51abd60b20d5}
#3067


Enhancement #3406: Improve dump performance for large files
The dump command now retrieves the data chunks for a file in parallel. This improves the download performance by up to as many times as the configured number of parallel backend connections.
#3406 #4796


Enhancement #3806: Optimize and make prune command resumable
Previously, if the prune command was interrupted, a later prune run would start repacking pack files from the start, as prune did not update the index while repacking.
The prune command now supports resuming interrupted prune runs. The update of the repository index has also been optimized to use less memory and only rewrite parts of the index that have changed.
#3806 #4812


Enhancement #4006: (alpha) Store deviceID only for hardlinks
Set RESTIC_FEATURES=device-id-for-hardlinks to enable this alpha feature. The feature flag will be removed after repository format version 3 becomes available or be replaced with a different solution.
When creating backups from a filesystem snapshot, for example created using BTRFS subvolumes, the deviceID of the filesystem changes compared to previous snapshots. This prevented restic from deduplicating the directory metadata of a snapshot.
When this alpha feature is enabled, the deviceID is only stored for hardlinks, which significantly reduces the metadata duplication for most backups.
#4006


Enhancement #4048: Add support for FUSE-T with mount on macOS
The restic mount command now supports creating FUSE mounts using FUSE-T on macOS.
#4048 #4825


Enhancement #4251: Support reading backup from a command's standard output
The backup command now supports the --stdin-from-command option. When using this option, the arguments to backup are interpreted as a command instead of paths to back up. backup then executes the given command and stores the standard output from it in the backup, similar to the what the --stdin option does. This also enables restic to verify that the command completes with exit code zero. A non-zero exit code causes the backup to fail.
Note that the --stdin option does not have to be specified at the same time, and that the --stdin-filename option also applies to --stdin-from-command.
Example: restic backup --stdin-from-command --stdin-filename dump.sql mysqldump [...]
#4251 #4410


Enhancement #4287: Support connection to rest-server using unix socket
Restic now supports using a unix socket to connect to a rest-server version 0.13.0 or later. This allows running restic as follows:
rest-server --listen unix:/tmp/rest.socket --data /path/to/data &
restic -r rest:http+unix:///tmp/rest.socket:/my_backup_repo/ [...]

#4287 #4655


Enhancement #4354: Significantly reduce prune memory usage
The prune command has been optimized to use up to 60% less memory. The memory usage should now be roughly similar to creating a backup.
#4354 #4812


Enhancement #4437: Make check command create non-existent cache directory
Previously, if a custom cache directory was specified for the check command, but the directory did not exist, check continued with the cache disabled.
The check command now attempts to create the cache directory before initializing the cache.
#4437 #4805 #4883


Enhancement #4472: Support AWS Assume Role for S3 backend
Previously only credentials discovered via the Minio discovery methods were used to authenticate.
However, there are many circumstances where the discovered credentials have lower permissions and need to assume a specific role. This is now possible using the following new environment variables:

RESTIC_AWS_ASSUME_ROLE_ARN - RESTIC_AWS_ASSUME_ROLE_SESSION_NAME - RESTIC_AWS_ASSUME_ROLE_EXTERNAL_ID - RESTIC_AWS_ASSUME_ROLE_REGION (defaults to us-east-1) - RESTIC_AWS_ASSUME_ROLE_POLICY - RESTIC_AWS_ASSUME_ROLE_STS_ENDPOINT

#4472 #4474


Enhancement #4547: Add --json option to version command
Restic now supports outputting restic version along with the Go version, OS and architecture used to build restic in JSON format using version --json.
#4547 #4553


Enhancement #4549: Add --ncdu option to ls command
NCDU (NCurses Disk Usage) is a tool to analyse disk usage of directories. It has an option to save a directory tree and analyse it later.
The ls command now supports outputting snapshot information in the NCDU format using the --ncdu option. Example usage: restic ls latest --ncdu | ncdu -f -
#4549 #4550 #4911


Enhancement #4573: Support rewriting host and time metadata in snapshots
The rewrite command now supports rewriting the host and/or time metadata of a snapshot using the new --new-host and --new-time options.
#4573


Enhancement #4583: Ignore s3.storage-class archive tiers for metadata
Restic used to store all files on S3 using the specified s3.storage-class.
Now, restic will only use non-archive storage tiers for metadata, to avoid problems when accessing a repository. To restore any data, it is still necessary to manually warm up the required data beforehand.
NOTE: There is no official cold storage support in restic, use this option at your own risk.
#4583 #4584


Enhancement #4590: Speed up mount command's error detection
The mount command now checks for the existence of the mountpoint before opening the repository, leading to quicker error detection.
#4590


Enhancement #4601: Add support for feature flags
Restic now supports feature flags that can be used to enable and disable experimental features. The flags can be set using the environment variable RESTIC_FEATURES. To get a list of currently supported feature flags, use the features command.
#4601 #4666


Enhancement #4611: Back up more file metadata on Windows
Previously, restic did not back up all common Windows-specific metadata.
Restic now stores file creation time and file attributes like the hidden, read-only and encrypted flags when backing up files and folders on Windows.
#4611


Enhancement #4664: Make ls use message_type field in JSON output
The ls command was the only restic command that used the struct_type field in its JSON output format to specify the message type.
The JSON output of the ls command now also includes the message_type field, which is consistent with other commands. The struct_type field is still included, but now deprecated.
#4664


Enhancement #4676: Make key command's actions separate sub-commands
Each of the add, list, remove and passwd actions provided by the key command is now a separate sub-command and have its own documentation which can be invoked using restic key <add|list|remove|passwd> --help.
#4676 #4685


Enhancement #4678: Add --target option to the dump command
Restic dump always printed to the standard output. It now supports specifying a --target file to write its output to.
#4678 #4682 #4692


Enhancement #4708: Back up and restore SecurityDescriptors on Windows
Restic now backs up and restores SecurityDescriptors for files and folders on Windows which includes owner, group, discretionary access control list (DACL) and system access control list (SACL).
This requires the user to be a member of backup operators or the application must be run as admin. If that is not the case, only the current user's owner, group and DACL will be backed up, and during restore only the DACL of the backed up file will be restored, with the current user's owner and group being set on the restored file.
#4708


Enhancement #4733: Allow specifying --host via environment variable
Restic commands that operate on snapshots, such as restic backup and restic snapshots, support the --host option to specify the hostname for grouping snapshots.
Such commands now also support specifying the hostname via the environment variable RESTIC_HOST. Note that --host still takes precedence over the environment variable.
#4733 #4734


Enhancement #4737: Include snapshot ID in reason field of forget JSON output
The JSON output of the forget command now includes id and short_id of snapshots in the reason field.
#4737


Enhancement #4764: Support forgetting all snapshots
The forget command now supports the --unsafe-allow-remove-all option, which removes all snapshots in the repository.
This option must always be combined with a snapshot filter (by host, path or tag). For example, the command forget --tag example --unsafe-allow-remove-all removes all snapshots with the tag "example".
#4764


Enhancement #4768: Allow specifying custom User-Agent for outgoing requests
Restic now supports setting a custom User-Agent for outgoing HTTP requests using the global option --http-user-agent or the RESTIC_HTTP_USER_AGENT environment variable.
#4768 #4810


Enhancement #4781: Add restore options to read include/exclude patterns from files
Restic now supports reading include and exclude patterns from files using the --include-file, --exclude-file, --iinclude-file and --iexclude-file options of the restore command.
#4781 #4811


Enhancement #4807: Support Extended Attributes on Windows NTFS
Restic now backs up and restores Extended Attributes for files and folders on Windows NTFS.
#4807


Enhancement #4817: Make overwrite behavior of restore customizable
The restore command now supports an --overwrite option to configure whether already existing files are overwritten. The overwrite behavior can be configured using the following option values:

--overwrite always (default): Always overwrites already existing files. The restore command will verify the existing file content and only restore mismatching parts to minimize downloads. Updates the metadata of all files. - --overwrite if-changed: Like always, but speeds up the file content check by assuming that files with matching size and modification time (mtime) are already up to date. In case of a mismatch, the full file content is verified like with always. Updates the metadata of all files. - --overwrite if-newer: Like always, but only overwrites existing files when the file in the snapshot has a newer modification time (mtime) than the existing file. - --overwrite never: Never overwrites existing files.

#4817 #200 #407 #2662 #4837 #4838 #4864 #4921


Enhancement #4839: Add dry-run support to restore command
The restore command now supports the --dry-run option to perform a dry run. Pass the --verbose=2 option to see which files would remain unchanged, and which would be updated or freshly restored.
#4839

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.