restic 0.17.3

Changelog for restic 0.17.3 (2024-11-08)

The following sections list the changes in restic 0.17.3 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #4971: Fix unusable mount on macOS Sonoma
• Fix #5003: Fix metadata errors during backup of removable disks on Windows
• Fix #5101: Do not retry load/list operation if SFTP connection is broken
• Fix #5107: Fix metadata error on Windows for backups using VSS
• Enh #5096: Allow prune --dry-run without lock

Details

• Bugfix #4971: Fix unusable mount on macOS Sonoma

  On macOS Sonoma when using FUSE-T, it was not possible to access files in a mounted repository. This issue is now resolved.

  #4971 #5048

• Bugfix #5003: Fix metadata errors during backup of removable disks on Windows

  Since restic 0.17.0, backing up removable disks on Windows could report errors with retrieving metadata like shown below.

  error: incomplete metadata for d:\filename: get named security info failed with: Access is denied.
  

  This has now been fixed.

  #5003 #5123 https://forum.restic.net/t/backing-up-a-folder-from-a-veracrypt-volume-brings-up-errors-since-restic-v17-0/8444

• Bugfix #5101: Do not retry load/list operation if SFTP connection is broken

  When using restic with the SFTP backend, backend operations that load a file or list files were retried even if the SFTP connection was broken. This has now been fixed.

  #5101 https://forum.restic.net/t/restic-hanging-on-backup/8559

• Bugfix #5107: Fix metadata error on Windows for backups using VSS

  Since restic 0.17.2, when creating a backup on Windows using --use-fs-snapshot, restic would report an error like the following:

  error: incomplete metadata for C:\: get EA failed while opening file handle for path \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopyXX\, with: The process cannot access the file because it is being used by another process.
  

  This has now been fixed by correctly handling paths that refer to volume shadow copy snapshots.

  #5107 #5110 #5112

• Enhancement #5096: Allow prune --dry-run without lock

  The prune --dry-run --no-lock now allows performing a dry-run without locking the repository. Note that if the repository is modified concurrently, prune may return inaccurate statistics or errors.

  #5096

restic 0.17.2

Changelog for restic 0.17.2 (2024-10-27)

The following sections list the changes in restic 0.17.2 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #4004: Support container-level SAS/SAT tokens for Azure backend
• Fix #5047: Resolve potential error during concurrent cache cleanup
• Fix #5050: Return error if tag fails to lock repository
• Fix #5057: Exclude irregular files from backups
• Fix #5063: Correctly backup extended metadata when using VSS on Windows

Details

• Bugfix #4004: Support container-level SAS/SAT tokens for Azure backend

  Restic previously expected SAS/SAT tokens to be generated at the account level, which prevented tokens created at the container level from being used to initialize a repository. This caused an error when attempting to initialize a repository with container-level tokens.

  Restic now supports both account-level and container-level SAS/SAT tokens for initializing a repository.

  #4004 #5093

• Bugfix #5047: Resolve potential error during concurrent cache cleanup

  When multiple restic processes ran concurrently, they could compete to remove obsolete snapshots from the local backend cache, sometimes leading to a "no such file or directory" error. Restic now suppresses this error to prevent issues during cache cleanup.

  #5047

• Bugfix #5050: Return error if tag fails to lock repository

  Since restic 0.17.0, the tag command did not return an error when it failed to open or lock the repository. This issue has now been fixed.

  #5050 #5056

• Bugfix #5057: Exclude irregular files from backups

  Since restic 0.17.1, files with the type irregular could mistakenly be included in snapshots, especially when backing up special file types on Windows that restic cannot process. This issue has now been fixed.

  Previously, this bug caused the check command to report errors like the following one:

    tree 12345678[...]: node "example.zip" with invalid type "irregular"
  

  To repair affected snapshots, upgrade to restic 0.17.2 and run:

  restic repair snapshots --forget
  

  This will remove the irregular files from the snapshots (creating a new snapshot ID for each of the affected snapshots).

  #5057 https://forum.restic.net/t/errors-found-by-check-1-invalid-type-irregular-2-ciphertext-verification-failed/8447/2

• Bugfix #5063: Correctly backup extended metadata when using VSS on Windows

  On Windows, when creating a backup with the --use-fs-snapshot option, restic read extended metadata from the original filesystem path instead of from the snapshot. This could result in errors if files were removed during the backup process.

  This issue has now been resolved.

  #5063 #5097 #5099

restic 0.17.1

Changelog for restic 0.17.1 (2024-09-05)

The following sections list the changes in restic 0.17.1 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #2004: Correctly handle volume names in backup command on Windows
• Fix #4945: Include missing backup error text with --json
• Fix #4953: Correctly handle long paths on older Windows versions
• Fix #4957: Fix delayed cancellation of certain commands
• Fix #4958: Don't ignore metadata-setting errors during restore
• Fix #4969: Correctly restore timestamp for files with resource forks on macOS
• Fix #4975: Prevent backup --stdin-from-command from panicking
• Fix #4980: Skip extended attribute processing on unsupported Windows volumes
• Fix #5004: Fix spurious "A Required Privilege Is Not Held by the Client" error
• Fix #5005: Fix rare failures to retry locking a repository
• Fix #5018: Improve HTTP/2 support for REST backend
• Chg #4953: Also back up files with incomplete metadata
• Enh #4795: Display progress bar for restore --verify
• Enh #4934: Automatically clear removed snapshots from cache
• Enh #4944: Print JSON-formatted errors during restore --json
• Enh #4959: Return exit code 12 for "bad password" errors
• Enh #4970: Make timeout for stuck requests customizable

Details

• Bugfix #2004: Correctly handle volume names in backup command on Windows

  On Windows, when the specified backup target only included the volume name without a trailing slash, for example, C:, then restoring the resulting snapshot would result in an error. Note that using C:\ as backup target worked correctly.

  Specifying volume names is now handled correctly. To restore snapshots created before this bugfix, use the : syntax. For example, to restore a snapshot with ID 12345678 that backed up C:, use the following command:

  restic restore 12345678:/C/C:./ --target output/folder
  

  #2004 #5028

• Bugfix #4945: Include missing backup error text with --json

  Previously, when running a backup with the --json option, restic failed to include the actual error message in the output, resulting in "error": {} being displayed.

  This has now been fixed, and restic now includes the error text in JSON output.

  #4945 #4946

• Bugfix #4953: Correctly handle long paths on older Windows versions

  On older Windows versions, like Windows Server 2012, restic 0.17.0 failed to back up files with long paths. This problem has now been resolved.

  #4953 #4954

• Bugfix #4957: Fix delayed cancellation of certain commands

  Since restic 0.17.0, some commands did not immediately respond to cancellation via Ctrl-C (SIGINT) and continued running for a short period. The most affected commands were diff,find, ls, stats and rewrite. This is now resolved.

  #4957 #4960

• Bugfix #4958: Don't ignore metadata-setting errors during restore

  Previously, restic used to ignore errors when setting timestamps, attributes, or file modes during a restore. It now reports those errors, except for permission related errors when running without root privileges.

  #4958

• Bugfix #4969: Correctly restore timestamp for files with resource forks on macOS

  On macOS, timestamps were not restored for files with resource forks. This has now been fixed.

  #4969 #5006

• Bugfix #4975: Prevent backup --stdin-from-command from panicking

  Restic would previously crash if --stdin-from-command was specified without providing a command. This issue has now been fixed.

  #4975 #4976

• Bugfix #4980: Skip extended attribute processing on unsupported Windows volumes

  With restic 0.17.0, backups of certain Windows paths, such as network drives, failed due to errors while fetching extended attributes.

  Restic now skips extended attribute processing for volumes where they are not supported.

  #4955 #4950 #4980 #4998

• Bugfix #5004: Fix spurious "A Required Privilege Is Not Held by the Client" error

  On Windows, creating a backup could sometimes trigger the following error:

  error: nodeFromFileInfo [...]: get named security info failed with: a required privilege is not held by the client.
  

  This has now been fixed.

  #5004 #5019

• Bugfix #5005: Fix rare failures to retry locking a repository

  Restic 0.17.0 could in rare cases fail to retry locking a repository if one of the lock files failed to load, resulting in the error:

  unable to create lock in backend: circuit breaker open for file <lock/1234567890>
  

  This issue has now been addressed. The error handling now properly retries the locking operation. In addition, restic waits a few seconds between locking retries to increase chances of successful locking.

  #5005 #5011 #5012

• Bugfix #5018: Improve HTTP/2 support for REST backend

  If rest-server tried to gracefully shut down an HTTP/2 connection still in use by the client, it could result in the following error:

  http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
  

  This issue has now been resolved.

  #5018 https://forum.restic.net/t/receiving-http2-goaway-messages-with-windows-restic-v0-17-0/8367

• Change #4953: Also back up files with incomplete metadata

  If restic failed to read extended metadata for a file or folder during a backup, then the file or folder was not included in the resulting snapshot. Instead, a warning message was printed along with returning exit code 3 once the backup was finished.

  Now, restic also includes items for which the extended metadata could not be read in a snapshot. The warning message has been updated to:

  incomplete metadata for /path/to/file: <details about error>
  

  #4953 #4977

• Enhancement #4795: Display progress bar for restore --verify

  When the restore command is run with --verify, it now displays a progress bar while the verification step is running. The progress bar is not shown when the --json flag is specified.

  #4795 #4989

• Enhancement #4934: Automatically clear removed snapshots from cache

  Previously, restic only removed snapshots from the cache on the host where the forget command was executed. On other hosts that use the same repository, the old snapshots remained in the cache.

  Restic now automatically clears old snapshots from the local cache of the current host.

  #4934 #4981

• Enhancement #4944: Print JSON-formatted errors during restore --json

  Restic used to print any restore errors directly to the console as freeform text messages, even when using the --json option.

  Now, when --json is specified, restic prints them as JSON formatted messages.

  #4944 #4946

• Enhancement #4959: Return exit code 12 for "bad password" errors

  Restic now returns exit code 12 when it cannot open the repository due to an incorrect password.

  #4959

• Enhancement #4970: Make timeout for stuck requests customizable

  Restic monitors connections to the backend to detect stuck requests. If a request does not return any data within five minutes, restic assumes the request is stuck and retries it. However, for large repositories this timeout might be insufficient to collect a list of all files, causing the following error:

  `List(data) returned error,...

restic 0.17.0

Changelog for restic 0.17.0 (2024-07-26)

The following sections list the changes in restic 0.17.0 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #3600: Handle unreadable xattrs in folders above backup source
• Fix #4209: Fix slow SFTP upload performance
• Fix #4503: Correct hardlink handling in stats command
• Fix #4568: Prevent forget --keep-tags <invalid> from deleting all snapshots
• Fix #4615: Make find not sometimes ignore directories
• Fix #4656: Properly report ID of newly added keys
• Fix #4703: Shutdown cleanly when receiving SIGTERM
• Fix #4709: Correct --no-lock handling of ls and tag commands
• Fix #4760: Fix possible error on concurrent cache cleanup
• Fix #4850: Handle UTF-16 password files in key command correctly
• Fix #4902: Update snapshot summary on rewrite
• Chg #956: Return exit code 10 and 11 for non-existing and locked repository
• Chg #4540: Require at least ARMv6 for ARM binaries
• Chg #4602: Deprecate legacy index format and s3legacy repository layout
• Chg #4627: Redesign backend error handling to improve reliability
• Chg #4707: Disable S3 anonymous authentication by default
• Chg #4744: Include full key ID in JSON output of key list
• Enh #662: Optionally skip snapshot creation if nothing changed
• Enh #693: Include snapshot size in snapshots output
• Enh #805: Add bitrot detection to diff command
• Enh #828: Improve features of the repair packs command
• Enh #1786: Support repositories with empty password
• Enh #2348: Add --delete option to restore command
• Enh #3067: Add extended options to configure Windows Shadow Copy Service
• Enh #3406: Improve dump performance for large files
• Enh #3806: Optimize and make prune command resumable
• Enh #4006: (alpha) Store deviceID only for hardlinks
• Enh #4048: Add support for FUSE-T with mount on macOS
• Enh #4251: Support reading backup from a command's standard output
• Enh #4287: Support connection to rest-server using unix socket
• Enh #4354: Significantly reduce prune memory usage
• Enh #4437: Make check command create non-existent cache directory
• Enh #4472: Support AWS Assume Role for S3 backend
• Enh #4547: Add --json option to version command
• Enh #4549: Add --ncdu option to ls command
• Enh #4573: Support rewriting host and time metadata in snapshots
• Enh #4583: Ignore s3.storage-class archive tiers for metadata
• Enh #4590: Speed up mount command's error detection
• Enh #4601: Add support for feature flags
• Enh #4611: Back up more file metadata on Windows
• Enh #4664: Make ls use message_type field in JSON output
• Enh #4676: Make key command's actions separate sub-commands
• Enh #4678: Add --target option to the dump command
• Enh #4708: Back up and restore SecurityDescriptors on Windows
• Enh #4733: Allow specifying --host via environment variable
• Enh #4737: Include snapshot ID in reason field of forget JSON output
• Enh #4764: Support forgetting all snapshots
• Enh #4768: Allow specifying custom User-Agent for outgoing requests
• Enh #4781: Add restore options to read include/exclude patterns from files
• Enh #4807: Support Extended Attributes on Windows NTFS
• Enh #4817: Make overwrite behavior of restore customizable
• Enh #4839: Add dry-run support to restore command

Details

• Bugfix #3600: Handle unreadable xattrs in folders above backup source

  When backup sources are specified using absolute paths, backup also includes information about the parent folders of the backup sources in the snapshot.

  If the extended attributes for some of these folders could not be read due to missing permissions, this caused the backup to fail. This has now been fixed.

  #3600 #4668 https://forum.restic.net/t/parent-directories-above-the-snapshot-source-path-fatal-error-permission-denied/7216

• Bugfix #4209: Fix slow SFTP upload performance

  Since restic 0.12.1, the upload speed of the sftp backend to a remote server has regressed significantly. This has now been fixed.

  #4209 #4782

• Bugfix #4503: Correct hardlink handling in stats command

  If files on different devices had the same inode ID, the stats command did not correctly calculate the snapshot size. This has now been fixed.

  #4503 #4006 https://forum.restic.net/t/possible-bug-in-stats/6461/8

• Bugfix #4568: Prevent forget --keep-tags <invalid> from deleting all snapshots

  Running forget --keep-tags <invalid>, where <invalid> is a tag that does not exist in the repository, would remove all snapshots. This is especially problematic if the tag name contains a typo.

  The forget command now fails with an error if all snapshots in a snapshot group would be deleted. This prevents the above example from deleting all snapshots.

  It is possible to temporarily disable the new check by setting the environment variable RESTIC_FEATURES=safe-forget-keep-tags=false. Note that this feature flag will be removed in the next minor restic version.

  #4568 #4764

• Bugfix #4615: Make find not sometimes ignore directories

  In some cases, the find command ignored empty or moved directories. This has now been fixed.

  #4615

• Bugfix #4656: Properly report ID of newly added keys

  restic key add now reports the ID of the newly added key. This simplifies selecting a specific key using the --key-hint key option.

  #4656 #4657

• Bugfix #4703: Shutdown cleanly when receiving SIGTERM

  Previously, when restic received the SIGTERM signal it would terminate immediately, skipping cleanup and potentially causing issues like stale locks being left behind. This primarily effected containerized restic invocations that use SIGTERM, but could also be triggered via a simple killall restic.

  This has now been fixed, such that restic shuts down cleanly when receiving the SIGTERM signal.

  #4703

• Bugfix #4709: Correct --no-lock handling of ls and tag commands

  The ls command never locked the repository. This has now been fixed, with the old behavior still being supported using ls --no-lock. The latter invocation also works with older restic versions.

  The tag command erroneously accepted the --no-lock command. This command now always requires an exclusive lock.

  #4709

• Bugfix #4760: Fix possible error on concurrent cache cleanup

  If multiple restic processes concurrently cleaned up no longer existing files from the cache, this could cause some of the processes to fail with an no such file or directory error. This has now been fixed.

  #4760 #4761...

restic 0.16.5

Changelog for restic 0.16.5 (2024-07-01)

The following sections list the changes in restic 0.16.5 relevant to restic users. The changes are ordered by importance.

Summary

• Enh #4799: Add option to force use of Azure CLI credential
• Enh #4873: Update dependencies

Details

• Enhancement #4799: Add option to force use of Azure CLI credential

  A new environment variable AZURE_FORCE_CLI_CREDENTIAL=true allows forcing the use of Azure CLI credential, ignoring other credentials like managed identity.

  #4799

• Enhancement #4873: Update dependencies

  A few potentially vulnerable dependencies were updated.

  #4873 #4878

restic 0.16.4

This release works around and improves detection of a bug in the compression library used by restic. The resulting issue only happens when using restic 0.16.3 and the max compression level (the default auto and off compression levels are not affected), and when the source files being backed up have specific data in them to trigger the bug. If you use max compression, you can use restic check --read-data to make sure you're not affected.

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.16.4 (2024-02-04)

The following sections list the changes in restic 0.16.4 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #4677: Downgrade zstd library to fix rare data corruption at max. compression
• Enh #4529: Add extra verification of data integrity before upload

Details

• Bugfix #4677: Downgrade zstd library to fix rare data corruption at max. compression

  In restic 0.16.3, backups where the compression level was set to max (using --compression max) could in rare and very specific circumstances result in data corruption due to a bug in the library used for compressing data. Restic 0.16.1 and 0.16.2 were not affected.

  Restic now uses the previous version of the library used to compress data, the same version used by restic 0.16.2. Please note that the auto compression level (which restic uses by default) was never affected, and even if you used max compression, chances of being affected by this issue are small.

  To check a repository for any corruption, run restic check --read-data. This will download and verify the whole repository and can be used at any time to completely verify the integrity of a repository. If the check command detects anomalies, follow the suggested steps.

  #4677 #4679

• Enhancement #4529: Add extra verification of data integrity before upload

  Hardware issues, or a bug in restic or its dependencies, could previously cause corruption in the files restic created and stored in the repository. Detecting such corruption previously required explicitly running the check --read-data or check --read-data-subset commands.

  To further ensure data integrity, even in the case of hardware issues or software bugs, restic now performs additional verification of the files about to be uploaded to the repository.

  These extra checks will increase CPU usage during backups. They can therefore, if absolutely necessary, be disabled using the --no-extra-verify global option. Please note that this should be combined with more active checking using the previously mentioned check commands.

  #4529 #4681

restic 0.16.3

This release fixes a couple of bugs on Windows and in the restore command. It also works around an unlikely yet possible situation with rclone which could potentially result in data loss.

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.16.3 (2024-01-14)

The following sections list the changes in restic 0.16.3 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #4560: Improve errors for irregular files on Windows
• Fix #4574: Support backup of deduplicated files on Windows again
• Fix #4612: Improve error handling for rclone backend
• Fix #4624: Correct restore progress information if an error occurs
• Fix #4626: Improve reliability of restoring large files

Details

• Bugfix #4560: Improve errors for irregular files on Windows

  Since Go 1.21, most filesystem reparse points on Windows are considered to be irregular files. This caused restic to show an error: invalid node type "" error message for those files.

  This error message has now been improved and includes the relevant file path: error: nodeFromFileInfo path/to/file: unsupported file type "irregular". As irregular files are not required to behave like regular files, it is not possible to provide a generic way to back up those files.

  #4560 #4620 https://forum.restic.net/t/windows-backup-error-invalid-node-type/6875

• Bugfix #4574: Support backup of deduplicated files on Windows again

  With the official release builds of restic 0.16.1 and 0.16.2, it was not possible to back up files that were deduplicated by the corresponding Windows Server feature. This also applied to restic versions built using Go 1.21.0-1.21.4.

  The Go version used to build restic has now been updated to fix this.

  #4574 #4621

• Bugfix #4612: Improve error handling for rclone backend

  Since restic 0.16.0, if rclone encountered an error while listing files, this could in rare circumstances cause restic to assume that there are no files. Although unlikely, this situation could result in data loss if it were to happen right when the prune command is listing existing snapshots.

  Error handling has now been improved to detect and work around this case.

  #4612 #4618

• Bugfix #4624: Correct restore progress information if an error occurs

  If an error occurred while restoring a snapshot, this could cause the restore progress bar to show incorrect information. In addition, if a data file could not be loaded completely, then errors would also be reported for some already restored files.

  Error reporting of the restore command has now been made more accurate.

  #4624 https://forum.restic.net/t/errors-restoring-with-restic-on-windows-server-s3/6943

• Bugfix #4626: Improve reliability of restoring large files

  In some cases restic failed to restore large files that frequently contain the same file chunk. In combination with certain backends, this could result in network connection timeouts that caused incomplete restores.

  Restic now includes special handling for such file chunks to ensure reliable restores.

  #4626 https://forum.restic.net/t/errors-restoring-with-restic-on-windows-server-s3/6943

restic 0.16.2

restic 0.16.2 restores support for ARMv5 which was accidentally not included in the previous one, and also fixes missing documentation on Read the Docs.

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.16.2 (2023-10-29)

The following sections list the changes in restic 0.16.2 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #4540: Restore ARMv5 support for ARM binaries
• Fix #4545: Repair documentation build on Read the Docs

Details

• Bugfix #4540: Restore ARMv5 support for ARM binaries

  The official release binaries for restic 0.16.1 were accidentally built to require ARMv7. The build process is now updated to restore support for ARMv5.

  Please note that restic 0.17.0 will drop support for ARMv5 and require at least ARMv6.

  #4540

• Bugfix #4545: Repair documentation build on Read the Docs

  For restic 0.16.1, no documentation was available at https://restic.readthedocs.io/ .

  The documentation build process is now updated to work again.

  #4545

restic 0.16.1

restic 0.16.1 has been released! It contains a few usage and cosmetic improvements, but also one bugfix for a very unlikely but still possible data corruption issue when using the max compression level. Please read the changelog for more details and information on how to verify your repository if you used max compression. Please also note that the auto compression level (which is the default) was never affected.

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.16.1 (2023-10-24)

The following sections list the changes in restic 0.16.1 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #4513: Make key list command honor --no-lock
• Fix #4516: Do not try to load password on command line autocomplete
• Fix #4523: Update zstd library to fix possible data corruption at max. compression
• Chg #4532: Update dependencies and require Go 1.19 or newer
• Enh #229: Show progress bar while loading the index
• Enh #4128: Automatically set GOMAXPROCS in resource-constrained containers
• Enh #4480: Allow setting REST password and username via environment variables
• Enh #4511: Include inode numbers in JSON output for find and ls commands
• Enh #4519: Add config option to set SFTP command arguments

Details

• Bugfix #4513: Make key list command honor --no-lock

  The key list command now supports the --no-lock options. This allows determining which keys a repo can be accessed by without the need for having write access (e.g., read-only sftp access, filesystem snapshot).

  #4513 #4514

• Bugfix #4516: Do not try to load password on command line autocomplete

  The command line autocompletion previously tried to load the repository password. This could cause the autocompletion not to work. Now, this step gets skipped.

  #4516 #4526

• Bugfix #4523: Update zstd library to fix possible data corruption at max. compression

  In restic 0.16.0, backups where the compression level was set to max (using --compression max) could in rare and very specific circumstances result in data corruption due to a bug in the library used for compressing data.

  Restic now uses the latest version of the library used to compress data, which includes a fix for this issue. Please note that the auto compression level (which restic uses by default) was never affected, and even if you used max compression, chances of being affected by this issue were very small.

  To check a repository for any corruption, run restic check --read-data. This will download and verify the whole repository and can be used at any time to completely verify the integrity of a repository. If the check command detects anomalies, follow the suggested steps.

  To simplify any needed repository repair and minimize data loss, there is also a new and experimental repair packs command that salvages all valid data from the affected pack files (see restic help repair packs for more information).

  #4523 #4530

• Change #4532: Update dependencies and require Go 1.19 or newer

  We have updated all dependencies. Since some libraries require newer Go standard library features, support for Go 1.18 has been dropped, which means that restic now requires at least Go 1.19 to build.

  #4532 #4533

• Enhancement #229: Show progress bar while loading the index

  Restic did not provide any feedback while loading index files. Now, there is a progress bar that shows the index loading progress.

  #229 #4419

• Enhancement #4128: Automatically set GOMAXPROCS in resource-constrained containers

  When running restic in a Linux container with CPU-usage limits, restic now automatically adjusts GOMAXPROCS. This helps to reduce the memory consumption on hosts with many CPU cores.

  #4128 #4485 #4531

• Enhancement #4480: Allow setting REST password and username via environment variables

  Previously, it was only possible to specify the REST-server username and password in the repository URL, or by using the --repository-file option. This meant it was not possible to use authentication in contexts where the repository URL is stored in publicly accessible way.

  Restic now allows setting the username and password using the RESTIC_REST_USERNAME and RESTIC_REST_PASSWORD variables.

  #4480

• Enhancement #4511: Include inode numbers in JSON output for find and ls commands

  Restic used to omit the inode numbers in the JSON messages emitted for nodes by the ls command as well as for matches by the find command. It now includes those values whenever they are available.

  #4511

• Enhancement #4519: Add config option to set SFTP command arguments

  When using the sftp backend, scenarios where a custom identity file was needed for the SSH connection, required the full command to be specified: -o sftp.command='ssh user@host:port -i /ssh/my_private_key -s sftp'

  Now, the -o sftp.args=... option can be passed to restic to specify custom arguments for the SSH command executed by the SFTP backend. This simplifies the above example to -o sftp.args='-i /ssh/my_private_key'.

  #4241 #4519

restic 0.16.0

restic 0.16.0 has been released!

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.16.0 (2023-07-31)

The following sections list the changes in restic 0.16.0 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #2565: Support "unlimited" in forget --keep-* options
• Fix #3311: Support non-UTF8 paths as symlink target
• Fix #4199: Avoid lock refresh issues on slow network connections
• Fix #4274: Improve lock refresh handling after standby
• Fix #4319: Correctly clean up status bar output of the backup command
• Fix #4333: generate and init no longer silently ignore unexpected arguments
• Fix #4400: Ignore missing folders in rest backend
• Chg #4176: Fix JSON message type of scan_finished for the backup command
• Chg #4201: Require Go 1.20 for Solaris builds
• Enh #426: Show progress bar during restore
• Enh #719: Add --retry-lock option
• Enh #1495: Sort snapshots by timestamp in restic find
• Enh #1759: Add repair index and repair snapshots commands
• Enh #1926: Allow certificate paths to be passed through environment variables
• Enh #2359: Provide multi-platform Docker images
• Enh #2468: Add support for non-global Azure clouds
• Enh #2679: Reduce file fragmentation for local backend
• Enh #3328: Reduce memory usage by up to 25%
• Enh #3397: Improve accuracy of ETA displayed during backup
• Enh #3624: Keep oldest snapshot when there are not enough snapshots
• Enh #3698: Add support for Managed / Workload Identity to azure backend
• Enh #3871: Support <snapshot>:<subfolder> syntax to select subfolders
• Enh #3941: Support --group-by for backup parent selection
• Enh #4130: Cancel current command if cache becomes unusable
• Enh #4159: Add --human-readable option to ls and find commands
• Enh #4188: Include restic version in snapshot metadata
• Enh #4220: Add jq binary to Docker image
• Enh #4226: Allow specifying region of new buckets in the gs backend
• Enh #4375: Add support for extended attributes on symlinks

Details

• Bugfix #2565: Support "unlimited" in forget --keep-* options

  Restic would previously forget snapshots that should have been kept when a negative value was passed to the --keep-* options. Negative values are now forbidden. To keep all snapshots, the special value unlimited is now supported. For example, --keep-monthly unlimited will keep all monthly snapshots.

  #2565 #4234

• Bugfix #3311: Support non-UTF8 paths as symlink target

  Earlier restic versions did not correctly backup and restore symlinks that contain a non-UTF8 target. Note that this only affected systems that still use a non-Unicode encoding for filesystem paths.

  The repository format is now extended to add support for such symlinks. Please note that snapshots must have been created with at least restic version 0.16.0 for restore to correctly handle non-UTF8 symlink targets when restoring them.

  #3311 #3802

• Bugfix #4199: Avoid lock refresh issues on slow network connections

  On network connections with a low upload speed, backups and other operations could fail with the error message Fatal: failed to refresh lock in time.

  This has now been fixed by reworking the lock refresh handling.

  #4199 #4304

• Bugfix #4274: Improve lock refresh handling after standby

  If the restic process was stopped or the host running restic entered standby during a long running operation such as a backup, this previously resulted in the operation failing with Fatal: failed to refresh lock in time.

  This has now been fixed such that restic first checks whether it is safe to continue the current operation and only throws an error if not.

  #4274 #4374

• Bugfix #4319: Correctly clean up status bar output of the backup command

  Due to a regression in restic 0.15.2, the status bar of the backup command could leave some output behind. This happened if filenames were printed that are wider than the current terminal width. This has now been fixed.

  #4319 #4318

• Bugfix #4333: generate and init no longer silently ignore unexpected arguments

  #4333

• Bugfix #4400: Ignore missing folders in rest backend

  If a repository accessed via the REST backend was missing folders, then restic would fail with an error while trying to list the data in the repository. This has been now fixed.

  #235 #4400

• Change #4176: Fix JSON message type of scan_finished for the backup command

  Restic incorrectly set the message_type of the scan_finished message to status instead of verbose_status. This has now been corrected so that the messages report the correct type.

  #4176

• Change #4201: Require Go 1.20 for Solaris builds

  Building restic on Solaris now requires Go 1.20, as the library used to access Azure uses the mmap syscall, which is only available on Solaris starting from Go 1.20. All other platforms however continue to build with Go 1.18.

  #4201

• Enhancement #426: Show progress bar during restore

  The restore command now shows a progress report while restoring files.

  Example: [0:42] 5.76% 23 files 12.98 MiB, total 3456 files 23.54 GiB

  JSON output is now also supported.

  #426 #3413 #3627 #3991 #4314 https://forum.restic.net/t/progress-bar-for-restore/5210

• Enhancement #719: Add --retry-lock option

  This option allows specifying a duration for which restic will wait if the repository is already locked.

  #719 #2214 #4107

• Enhancement #1495: Sort snapshots by timestamp in restic find

  The find command used to print snapshots in an arbitrary order. Restic now prints snapshots sorted by timestamp.

  #1495 #4409

• Enhancement #1759: Add repair index and repair snapshots commands

  The rebuild-index command has been renamed to repair index. The old name will still work, but is deprecated.

  When a snapshot was damaged, the only option up to now was to completely forget the snapshot, even if only some unimportant files in it were damaged and other files were still fine.

  Restic now has a repair snapshots command, which can salvage any non-damaged files and parts of files in the snapshots by removing damaged directories and missing file contents. Please note that the damaged data may still be lost and see the "Troubleshooting" section in the documentation for more details.

  #1759 [#1714](https://github.com/restic/r...