Skip to content

Latest commit

 

History

History
143 lines (105 loc) · 5.24 KB

README.md

File metadata and controls

143 lines (105 loc) · 5.24 KB
Raw

DTP (Disentangle then Parse)

This repository implements a PyTorch re-implementation of the research paper: "Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement".

overview

Dataset

Nightcity-fine: Access the dataset via Google Drive.

Cityscapes: Access the dataset via cityscapes-dataset.com.

Environment Setup

Set up your environment with these steps:

conda create -n dtp python=3.10
conda activate dtp
conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
# Alternatively: pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu116
pip install tensorboard
pip install -U openmim
mim install mmcv-full
pip install -v -e .
# Alternatively: python setup.py develop

Preparation

Download and Organize

  1. Decompress the nightcity-fine.zip dataset and relocate it to ./data/nightcity-fine.

  2. Download Cityscapes

    1. gtFine_trainvaltest.zip (241MB) [md5]

    2. gtCoarse.zip (1.3GB) [md5]

    3. leftImg8bit_trainvaltest.zip (11GB) [md5]

    4. git clone https://github.com/mcordts/cityscapesScripts.git

pip install cityscapesscripts

    5. Extract all the above zip files and git repo into the ./data folder

    6. vim cityscapesScripts/cityscapesscripts/preparation/createTrainIdLabelImgs.py 

Add the next line of code after `import os`
os.envieron['CITYSCAPES_DATASET'] = "../../../"

python cityscapesScripts/cityscapesscripts/preparation/createTrainIdLabelImgs.py

mkdir cityscapes

mv gtFine cityscapes && mv leftImg8bit cityscapes

  3. Download the checkpoint from Google Drive and place it in ./checkpoints.

Your directory structure should resemble:

.
├── checkpoints
│   ├── night
│   ├── night+day
|   └── simmim_pretrain__swin_base__img192_window6__800ep.pth
├── custom
├── custom-tools
│   ├── dist_test.sh
│   ├── dist_train.sh
│   ├── test.py
│   └── train.py
├── data
│   ├── cityscapes
│   │   ├── gtFine
│   │   └── leftImg8bit
│   └── nightcity-fine
│       ├── train
│       └── val
├── mmseg
├── readme.md
├── requirements.txt
├── setup.cfg
└── setup.py

Testing

Execute tests using:

python custom-tools/test.py checkpoints/night/cfg.py checkpoints/night/night.pth --eval mIoU --aug-test

Training

  1. Download pre-training weight from Google Drive.
  2. Convert it to MMSeg format using: 
    python custom-tools/swin2mmseg.py </path/to/pretrain> checkpoints/simmim_pretrain__swin_base__img192_window6__800ep.pth
  3. Start training with: 
    python custom-tools/train.py </path/to/your/config>
# </path/to/your/config>：our config: checkpoints/night/cfg.py or checkpoints/night+day/cfg.py

Results

The table below summarizes our findings:

logs train dataset validation dataset mIoU
checkpoints/night/eval_multi_scale_20230801_162237.json nightcity-fine nightcity-fine 64.2
checkpoints/night+day/eval_multi_scale_20230809_170141.json nightcity-fine + cityscapes nightcity-fine 64.9

Acknowledgements

This dataset is refined based on the dataset of NightCity by Xin Tan et al. and NightLab by Xueqing Deng et al..

This project is based on the mmsegmentation.

Pretraining checkpoint comes from the SimMIM.

The annotation process was completed using LabelMe.

Citation

If you find this code or data useful, please cite our paper

@InProceedings{Wei_2023_ICCV,
    author    = {Wei, Zhixiang and Chen, Lin and Tu, Tao and Ling, Pengyang and Chen, Huaian and Jin, Yi},
    title     = {Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21593-21603}
}