This is a fork of the unofficial implementation of Street Gaussians, which enables Street Gaussians on AMD GPUs.
It modifies the original codebase in the following ways:
- Makes Street Gaussians work with a more recent version of gsplat, which requires major changes to the
way the densification process.
- Check out gsplat migration here.
- Allows Street Gaussians to be run on both AMD and NVIDIA GPUs, mainly thanks to the gsplat upgrade,
which means we can run with AMD-enabled gsplat.
- Adds some new options to allow experimentation with some new techniques and to output more debugging
images and metrics.
- Refactors the code of the models so that it is easier to understand how the loss is computed, in particular
what computation is done on the scene graph (the union of the background and object models) and what is
done on the individual models.
- Enhances the rendering tool to render new videos from a trained scene:
- Interpolation between training poses to generate smooth videos at a higher framerate than the training input.
- Some simple editing operations, specified using config files, that manipulate the paths of the dynamic objects
to produce a new variant of the scenario.
-
Download a scene from Waymo dataset
- Go here and login with your google account and complete the registration form.
- Download waymo_open_dataset_v_1_4_0 -> individual_files -> validation -> segment_1906113358876584689 and add it to the data directory.
-
Build Docker images: build the main and the data pre-processing docker images:
docker build -t street-gaussians .
docker build -t street-gaussians-data-proc -f Dockerfile.data_proc .
-
Data preprocessing: inside the data pre-processing docker container:
First extract the data from the Waymo dataset using the Waymo tool (script from Street Gaussians).
python scripts/pythons/extract_waymo.py --waymo_root ./data --out_root ./data --split validation
Street Gaussians provides us with a data processing script that calls these scripts:
- segs_generate.sh: Semantic segmentation of images using Mark2Former, which will be used for creating sky masks.
- masks_generate.sh: create dynamic object masks, which will be used in COLMAP to avoid feature extraction in those regions.
- run_colmap.sh: Run the COLMAP pipeline to generate an initial point cloud.
- points_cloud_generate.sh: Get a point cloud also from lidar data and combine it with the COLMAP point cloud.
- object_pts_generate.sh: Load the point cloud and extract regions corresponding to detected objects into separate clouds.
In the original dataset, the images for each camera are stored in different subfolders.
bash -e scripts/shells/data_process.sh ./data/validation/1906113358876584689_1359_560_1379_560/
Exit the data pre-processing container.
-
Run training: inside the main docker container:
bash scripts/shells/train/train.sh ./data/validation/1906113358876584689_1359_560_1379_560/
Check the parameters that you can modify:
sgn-train -h
sgn-train street-gaussians-ns -h
sgn-train street-gaussians-ns colmap-data-parser-config -h
NOTE: the default parameters are set in sgn_config.py script and differ somewhat from the original codebase.
-
Run rendering: inside the base container:
bash scripts/shells/render.sh [config_file]
Check the perameters that you can modify:
sgn-render -h
-
Perform scene editing: inside the base container:
bash scripts/shells/render.sh [config_file] [edit_config_file]
-
Run evaluation: inside the base container:
bash scripts/shells/eval.sh [config_file]
Check the perameters that you can modify:
sgn-eval -h
If you find this code useful, please be so kind to cite
@inproceedings{yan2024street,
title={Street Gaussians for Modeling Dynamic Urban Scenes},
author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng},
booktitle={ECCV},
year={2024}
}