Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
ECCV 2024
street_gaussians.mp4
Clone this repository
git clone https://github.com/zju3dv/street_gaussians.git
Set up the python environment
# Set conda environment
conda create -n street-gaussian python=3.8
conda activate street-gaussian
# Install torch (corresponding to your CUDA version)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
# Install requirements
pip install -r requirements.txt
# Install submodules
pip install ./submodules/diff-gaussian-rasterization
pip install ./submodules/simple-knn
pip install ./submodules/simple-waymo-open-dataset-reader
python script/test_gaussian_rasterization.py
Prepare Waymo Open Dataset.
We provide the example scenes here. You can directly download the data and skip the following steps for a quick start.
Download the training and validation set of Waymo Open Dataset.
We provide the split file following EmerNeRF. You can refer to this document for download details.
Download the tracking predictions on validation set, We provide the processed results here.
Preprocess the example scenes
python script/waymo/waymo_converter.py --root_dir TRAINING_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/demo.txt --segment_file script/waymo/waymo_splits/segment_list_train.txt
Preprocess the experiment scenes
python script/waymo/waymo_converter.py --root_dir VALIDATION_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/val_dynamic.txt --segment_file script/waymo/waymo_splits/segment_list_val.txt
--track_file TRACKER_PATH
Generating LiDAR depth
python script/waymo/generate_lidar_depth.py --datadir DATA_DIR
Generating sky mask
Install GroundingDINO following this repo and download SAM checkpoint from this link.
python script/waymo/generate_sky_mask.py --datadir DATA_DIR --sam_checkpoint SAM_CKPT
Prepare Custom Dataset.
TODOpython train.py --config configs/xxxx.yaml
Training on example scenes
bash script/waymo/train_waymo_expample.sh
Training on experiment scenes
bash script/waymo/train_waymo_exp.sh
python render.py --config configs/xxxx.yaml mode {evaluate, trajectory}
Rendering on example scenes
bash script/waymo/render_waymo_expample.sh
Rendering on experiment scenes
bash script/waymo/render_waymo_exp.sh
You can convert the scene at one certain frame into the format that can be viewed in SIBR_viewers.
python make_ply.py --config configs/xxxx.yaml viewer.frame_id {frame_idx} mode evaluate
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{yan2024street,
title={Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting},
author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng},
booktitle={ECCV},
year={2024}
}