Skip to content

[ECCV 2024] Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

License

Notifications You must be signed in to change notification settings

zju3dv/street_gaussians

Repository files navigation

Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
ECCV 2024

street_gaussians.mp4

Installation

Clone this repository
git clone https://github.com/zju3dv/street_gaussians.git
Set up the python environment
# Set conda environment
conda create -n street-gaussian python=3.8
conda activate street-gaussian

# Install torch (corresponding to your CUDA version)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

# Install requirements
pip install -r requirements.txt

# Install submodules
pip install ./submodules/diff-gaussian-rasterization
pip install ./submodules/simple-knn
pip install ./submodules/simple-waymo-open-dataset-reader
python script/test_gaussian_rasterization.py
Prepare Waymo Open Dataset.

We provide the example scenes here. You can directly download the data and skip the following steps for a quick start.

Download the training and validation set of Waymo Open Dataset.

We provide the split file following EmerNeRF. You can refer to this document for download details.

Preprocess the data

Download the tracking predictions on validation set, We provide the processed results here.

Preprocess the example scenes

python script/waymo/waymo_converter.py --root_dir TRAINING_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/demo.txt --segment_file script/waymo/waymo_splits/segment_list_train.txt

Preprocess the experiment scenes

python script/waymo/waymo_converter.py --root_dir VALIDATION_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/val_dynamic.txt --segment_file script/waymo/waymo_splits/segment_list_val.txt
--track_file TRACKER_PATH

Generating LiDAR depth

python script/waymo/generate_lidar_depth.py --datadir DATA_DIR

Generating sky mask

Install GroundingDINO following this repo and download SAM checkpoint from this link.

python script/waymo/generate_sky_mask.py --datadir DATA_DIR --sam_checkpoint SAM_CKPT
Prepare Custom Dataset. TODO

Training

python train.py --config configs/xxxx.yaml

Training on example scenes

bash script/waymo/train_waymo_expample.sh

Training on experiment scenes

bash script/waymo/train_waymo_exp.sh

Rendering

python render.py --config configs/xxxx.yaml mode {evaluate, trajectory}

Rendering on example scenes

bash script/waymo/render_waymo_expample.sh

Rendering on experiment scenes

bash script/waymo/render_waymo_exp.sh

Visualization

You can convert the scene at one certain frame into the format that can be viewed in SIBR_viewers.

python make_ply.py --config configs/xxxx.yaml viewer.frame_id {frame_idx} mode evaluate

Pipeline

pipeline

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{yan2024street,
    title={Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting}, 
    author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng},
    booktitle={ECCV},
    year={2024}
}

About

[ECCV 2024] Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published