Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bcachefs, self healing reads as "poor man's scrub". #780

Open
Nihon-Ryori opened this issue Nov 13, 2024 · 2 comments
Open

Bcachefs, self healing reads as "poor man's scrub". #780

Nihon-Ryori opened this issue Nov 13, 2024 · 2 comments

Comments

@Nihon-Ryori
Copy link

Nihon-Ryori commented Nov 13, 2024

The following untested bash script copies all files of a partition to /dev/null. In the event that bcachefs uses e.g. dupe>=2 or a raid with redundancy, the files are compared with their checksums during the read process by bcachefs' own self-healing and files for which the checksum is not correct are replaced by files for which the checksum of the file is correct.


# [Nihon-Ryori](https://github.com/Nihon-Ryori)
# Open Source Code. Feel free to use the Code for bcachefs project or what ever.
# 2024-11-12, ver 002
# This version should do a Bcachefs, self healing reads as "poor man's scrub" for files, no directory support but now with progress bar

# Untested Code. Use the code at your own risk. Please use the code only if you would write it yourself. The code is only intended for test use with a hard drive of a test system that does not contain any data that is still required.

#!/bin/bash

# Set source and destination directories
SOURCE_DIR="/dev/sdb1"  # Replace with actual drive letter/label
DEST_DIR="/dev/null"     # Or replace with other drive letter/label

# Initialize counters
total_files=0
processed_files=0

# Check if source and destination drives exist
if ! mountpoint -q "$SOURCE_DIR"; then
    echo "Error: Source drive $SOURCE_DIR does not exist."
    exit 1
fi

if ! mountpoint -q "$DEST_DIR"; then
    echo "Error: Destination drive $DEST_DIR does not exist."
    exit 1
fi

# Start copying process
echo "Starting file copy from $SOURCE_DIR to $DEST_DIR"

# Use find to iterate through all files on the source drive
find "$SOURCE_DIR" -type f | while read -r file; do
    # Increment counters
    ((total_files++))
    
    # Copy each file to the destination drive
    cp "$file" "$DEST_DIR" || { echo "Error copying $file"; continue; }
    
    # Update processed files counter
    ((processed_files++))
    
    # Provide progress feedback
    echo "Copied: $file"
    
    # Update progress percentage every 100 files
    if ((processed_files % 100 == 0)); then
        echo "Progress: $(printf "%.1f%%" ((${processed_files} / ${total_files}) * 100)) Complete"
    fi
    
done

echo "poor man's scrub completed."

Untested Code. Use the code at your own risk. Please use the code only if you would write it yourself. The code is only intended for test use with a hard drive of a test system that does not contain any data that is still required.

Open Source Code. Feel free to use the Code for bcachefs project or what ever.

@Nihon-Ryori
Copy link
Author

Nihon-Ryori commented Nov 13, 2024

The follow version should check directory's and files and not only files:

# [Nihon-Ryori](https://github.com/Nihon-Ryori)
# Open Source Code. Feel free to use the Code for bcachefs project or what ever.
# 2024-11-12, ver 003
# This version should do a Bcachefs, self healing reads as "poor man's scrub" for files, directorys and now with progress bar

# Untested Code. Use the code at your own risk. Please use the code only if you would write it yourself. The code is only intended for test use with a hard drive of a test system that does not contain any data that is still required.

#!/bin/bash

# Set source and destination directories
SOURCE_DIR="/dev/sdb1"  # Replace with actual drive letter/label
DEST_DIR="/dev/null"     # Or replace with other drive letter/label

# Initialize counters
total_items=0
processed_items=0

# Check if source directory exists
if ! [ -d "$SOURCE_DIR" ]; then
    echo "Error: Source directory $SOURCE_DIR does not exist."
    exit 1
fi

# Start copying process
echo "Starting copy from $SOURCE_DIR to $DEST_DIR"

# Use find to iterate through all items on the source directory
find "$SOURCE_DIR" | while read -r item; do
    # Increment counters
    ((total_items++))
    
    # Copy each item to the destination directory
    cp -R "$item" "$DEST_DIR" || { echo "Error copying $item"; continue; }
    
    # Update processed items counter
    ((processed_items++))
    
    # Provide progress feedback
    echo "Copied: $item"
    
    # Update progress percentage every 100 items
    if ((processed_items % 100 == 0)); then
        echo "Progress: $(printf "%.1f%%" ((${processed_items} / ${total_items}) * 100)) Complete"
    fi
    
done

echo "poor man's scrub completed."

@Nihon-Ryori
Copy link
Author

Nihon-Ryori commented Nov 15, 2024

The follow version should check directory's and files and not only files:

poor_mans_scrub.sh

# [Nihon-Ryori](https://github.com/Nihon-Ryori)
# Open Source Code. Feel free to use the Code for bcachefs project or what ever.
# 2024-11-15, ver 004
# This version should do a Bcachefs, self healing reads as "poor man's scrub" for files, directory's and now with progress bar, The script close on the end, after pressing enter.

# Untested Code. Use the code at your own risk and only if you would write it yourself. The code is only intended for test use with a test system that does not contain any data that is still required.

#!/bin/bash

# Set source and destination directories
SOURCE_DIR="/dev/sdb1"  # Replace with actual drive letter/label
DEST_DIR="/dev/null"     # Or replace with other drive letter/label

# Initialize counters
total_items=0
processed_items=0

# Check if source directory exists
if ! [ -d "$SOURCE_DIR" ]; then
    echo "Error: Source directory $SOURCE_DIR does not exist."
    exit 1
fi

# Start copying process
echo "Starting copy from $SOURCE_DIR to $DEST_DIR"

# Use find to iterate through all items on the source directory
find "$SOURCE_DIR" | while read -r item; do
    # Increment counters
    ((total_items++))
    
    # Copy each item to the destination directory
    cp -R "$item" "$DEST_DIR" || { echo "Error copying $item"; continue; }
    
    # Update processed items counter
    ((processed_items++))
    
    # Provide progress feedback
    echo "Copied: $item"
    
    # Update progress percentage every 100 items
    if ((processed_items % 100 == 0)); then
        echo "Progress: $(printf "%.1f%%" ((${processed_items} / ${total_items}) * 100)) Complete"
    fi
    
done

echo "poor man's scrub completed."

echo "Press Enter to end the script"; read -r

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant