HARP: Unsupervised Histopathological Artifact Restoration

This is the official code repository for Unsupervised Histopathological Artifact Restoration Pipeline (HARP). Reliability of histopathological analysis is often challenged by artifacts introduced during sample preparation and imaging, ranging from staining inconsistencies to physical obstructions. While there's been a recent emergence in the domain of artifact restoration in histopathological images, all of them rely on supervised methodologies, requiring considerable manual input or supervision on artifacts for effective WSI restoration. So to address this, we introduce HARP, a novel, fully unsupervised histopathological artifact restoration pipeline, that integrates artifact detection, localization, and restoration into one pipeline.

Table Of Contents

1. Installation
2. Model Checkpoints and Dataset
3. Usage
4. Results
5. Citations
6. Acknowledgement

Installation

To set up HARP, start by cloning the repository and navigate to the environment.yaml file to install the conda environment named harp. Activate the Conda environment.

git clone https://github.com/MECLabTUDA/HARP.git
cd HARP
conda env create -f environment.yaml
conda activate harp

To install via PyPi please run:

conda create -n harp python=3.11
conda activate harp
pip install HARPipe
pip install 'git+https://github.com/facebookresearch/segment-anything.git'

Model Checkpoints and Dataset

Download the checkpoints of the respective models from the provided sources. Once downloaded, update the paths of the models in the config/config_restoration_model.json file to reflect their new locations.

• Artifact Detection: FastFlow model
• Artifact Segmentation: ViT-H SAM model
• Artifact Restoration: Our restoration model

To train all the above models, we have have extracted patches from the BCSS dataset of size 600x600 with 0.24 mpp, which we downsize to 256x256. This has resulted in 11075 training, 1031 validation and 1000 test patches. For evaluation, we have leveraged FrOoDo to synthetically apply 10 different artifacts on randomly chosen 100 test patches.The dataset have been made available and can be downloaded from the source stated below:

• Dataset used for training and evaluation: Dataset

Usage

1. To train the restoration model on your own dataset, use the following training script. Ensure to update the data_root path for both the train and validation split of the dataset accordingly.

python train_restoration.py -p train -c config/config_restoration_model.json

2. To train the Fastflow model for artifact detection, refer to the Anomalib repo.

3. To run the HARP pipeline, use the script below. Update the input and output path in the config/config_harp.json. Also make sure to update the model checkpoint paths as shown in Model Checkpoints.

python harp_pipeline.py -c config/config_harp.json

Usage as PyPI Package

from harp.harp_pipeline import HARP
from harp.pipeline.harp_dataset import HARPDataset

harp = HARP("path/to/config_harp.json")
dataset = HARPDataset(harp.config["input_folder_path"], harp.config["image_size"])

for image in dataset:
        restored_image, restored_mask = harp.harp_pipeline(image)

Results

Qualitative Results

This video shows the steps in the HARP Pipeline including our novel restoration method.

HARP.Paper.Video.Compressed.mp4

User Study

We conducted a user study with four pathologists on 50 image pairs as a visual turning test to evaluate the produced image quality. One of the images from the pair is a normal image from the training distribution, and the other is an image from the test distribution augmented with an artifact and then restored with HARP. All our participants found it impossible to tell the real difference between images, as our results suggest, at best, there is a weak positive correlation by chance.

Clinician	DE1	DE2	DE3	DE4
MCC	-0.071	-0.159	0.239	0.296
FR	27/50	30/50	20/50	18/50

Downstream Performance

We evaluated the segmentation performance of the downstream model using clean images, artifact images, and restored images with HARP. The performance of the artifact images significantly decreases compared to the originally clean images. HARP is able to recover the artifact images and effectively reduces the performance drop introduced by the artifacts by 48%, which makes the downstream model more robust and reliable for the daily clinical workflow.

Performance (DICE %)	Tumor	Stroma	Lymphocyte-rich	Necrosis	Average
Clean	86.1	83.8	81.8	74.2	81.5
Artifacts	77.7	77.9	76.2	64.9	74.2
HARP (Ours)	82.2	82.0	78.5	69.3	78.0

Citations

If you are using HARP for your article, please cite the following paper:

@inproceedings{fuchs2024harp,
  title={HARP: Unsupervised Histopathology Artifact Restoration},
  author={Fuchs, Moritz and Sivakumar, Ssharvien Kumar R and Sch{\"o}ber, Mirko and Woltering, Niklas and Eich, Marie-Lisa and Schweizer, Leonille and Mukhopadhyay, Anirban},
  booktitle={Medical Imaging with Deep Learning},
  year={2024}
}

Acknowledgement

Thanks for the following projects and theoretical works that we have either used or inspired from:

• The architecture of our conditional model from Janspiry's Palette implementation
• Segment Anything (SAM)
• The implementation of FastFlow by Anomalib