You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
2024/11/18 16:17: Update model and demo data download link through HuggingFace, Personally I found `wget` from HuggingFace link is much faster than Zenodo.
12
-
13
-
2024/09/26 16:24: All codes already uploaded and tested. You can to try training directly by downloading (through [HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow)/[Zenodo](https://zenodo.org/records/13744999)) demo data or pretrained weight for evaluation.
<details> <summary>🎁 <b>One repository, All methods!</b> OpenSceneFlow integrates the following excellent works </summary>
14
28
15
-
Pre-trained weights for models are available in [Zenodo](https://zenodo.org/records/13744999)/[HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow) link. Check usage in [2. Evaluation](#2-evaluation) or [3. Visualization](#3-visualization).
16
-
17
-
Task: __Self-Supervised__ Scene Flow Estimation in Autonomous Driving. No human-label needed. Real-time inference (15-20Hz in RTX3090).
18
-
19
-
We directly follow our previous work [code structure](https://github.com/KTH-RPL/DeFlow), so you may want to start from the easier one with supervised learning first: Try [DeFlow](https://github.com/KTH-RPL/DeFlow). Then you will find this is simple to you (things about how to train under self-supervised). Here are **Scripts** quick view in this repo:
29
+
-[ ][NSFP](https://arxiv.org/abs/2111.01253): NeurIPS 2021, faster 3x than original version because of [our CUDA speed up](assets/cuda/README.md), same (slightly better) performance. Done coding, public after review.
30
+
-[ ][FastNSF](https://arxiv.org/abs/2304.09121): ICCV 2023. Done coding, public after review.
31
+
-[ ][Flow4D](https://arxiv.org/abs/2407.07995): Under Review. Done coding, public after review.
32
+
-[ ] ... more on the way
20
33
21
-
-`dataprocess/extract_*.py` : pre-process data before training to speed up the whole training time.
22
-
[Dataset we included now: Argoverse 2 and Waymo. more on the way: Nuscenes, custom data.]
23
-
24
-
-`process.py`: process data with save dufomap, cluster labels inside file. Only needed once for training.
34
+
</details>
25
35
26
-
-`train.py`: Train the model and get model checkpoints. Pls remember to check the config.
36
+
## Citation
27
37
28
-
-`eval.py` : Evaluate the model on the validation/test set. And also output the zip file to upload to online leaderboard.
38
+
If you find *OpenSceneFlow* useful to your research, please cite our work as encouragement. (੭ˊ꒳ˋ)੭✧
29
39
30
-
-`save.py` : Will save result into h5py file, using [tool/visualization.py] to show results with interactive window.
40
+
```
41
+
@inproceedings{zhang2024seflow,
42
+
author={Zhang, Qingwen and Yang, Yi and Li, Peizheng and Andersson, Olov and Jensfelt, Patric},
43
+
title={{SeFlow}: A Self-Supervised Scene Flow Method in Autonomous Driving},
44
+
booktitle={European Conference on Computer Vision (ECCV)},
45
+
year={2024},
46
+
pages={353–369},
47
+
organization={Springer},
48
+
doi={10.1007/978-3-031-73232-4_20},
49
+
}
50
+
@inproceedings{zhang2024deflow,
51
+
author={Zhang, Qingwen and Yang, Yi and Fang, Heng and Geng, Ruoyu and Jensfelt, Patric},
52
+
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
53
+
title={{DeFlow}: Decoder of Scene Flow Network in Autonomous Driving},
54
+
year={2024},
55
+
pages={2105-2111},
56
+
doi={10.1109/ICRA57147.2024.10610278}
57
+
}
58
+
```
31
59
32
-
<details> <summary>🎁 <b>One repository, All methods!</b> </summary>
33
-
<!-- <br> -->
34
-
You can try following methods in our code without any effort to make your own benchmark.
-[x][ZeroFlow](https://arxiv.org/abs/2305.10424): ICLR 2024, their pre-trained weight can covert into our format easily through [the script](tools/zerof2ours.py).
40
-
-[ ][NSFP](https://arxiv.org/abs/2111.01253): NeurIPS 2021, faster 3x than original version because of [our CUDA speed up](assets/cuda/README.md), same (slightly better) performance. Done coding, public after review.
41
-
-[ ][FastNSF](https://arxiv.org/abs/2304.09121): ICCV 2023. Done coding, public after review.
42
-
<!-- - [ ] [Flow4D](https://arxiv.org/abs/2407.07995): 1st supervise network in the new leaderboard. Done coding, public after review. -->
43
-
-[ ] ... more on the way
62
+
📜 Changelog:
44
63
45
-
</details>
64
+
- 🤗 2024/11/18 16:17: Update model and demo data download link through HuggingFace, Personally I found `wget` from HuggingFace link is much faster than Zenodo.
65
+
- 2024/09/26 16:24: All codes already uploaded and tested. You can to try training directly by downloading (through [HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow)/[Zenodo](https://zenodo.org/records/13744999)) demo data or pretrained weight for evaluation.
💡: Want to learn how to add your own network in this structure? Check [Contribute](assets/README.md#contribute) section and know more about the code. Fee free to pull request!
48
68
49
-
## 0. Setup
69
+
## 0. Installation
50
70
51
-
**Environment**: Same to [DeFlow](https://github.com/KTH-RPL/DeFlow). And even lighter here with extracting mmcv module we needed into cuda assets.
CUDA package (need install nvcc compiler), the compile time is around 1-5 minutes:
59
79
```bash
60
-
mamba activate seflow
80
+
mamba activate opensf
61
81
# CUDA already install in python environment. I also tested others version like 11.3, 11.4, 11.7, 11.8 all works
62
82
cd assets/cuda/mmcv && python ./setup.py install &&cd ../../..
63
83
cd assets/cuda/chamfer3D && python ./setup.py install &&cd ../../..
64
84
```
65
85
66
-
Or you always can choose [Docker](https://en.wikipedia.org/wiki/Docker_(software)) which isolated environment and free yourself from installation, you can pull it by.
67
-
If you have different arch, please build it by yourself `cd SeFlow && docker build -t zhangkin/seflow` by going through [build-docker-image](https://github.com/KTH-RPL/DeFlow/blob/main/assets/README.md/#build-docker-image) section.
86
+
<!--Or you always can choose [Docker](https://en.wikipedia.org/wiki/Docker_(software)) which isolated environment and free yourself from installation, you can pull it by.
87
+
If you have different arch, please build it by yourself `cd OpenSceneFlow && docker build -t zhangkin/opensf` by going through [build-docker-image](assets/README.md/#build-docker-image) section.
68
88
```bash
69
89
# option 1: pull from docker hub
70
90
docker pull zhangkin/seflow
71
91
72
92
# run container
73
93
docker run -it --gpus all -v /dev/shm:/dev/shm -v /home/kin/data:/home/kin/data --name seflow zhangkin/seflow /bin/zsh
74
-
```
94
+
```-->
75
95
76
-
## 1. Run & Train
77
96
78
-
Note: Prepare raw data and process train data only needed run once for the task. No need repeat the data process steps till you delete all data. We use [wandb](https://wandb.ai/) to log the training process, and you may want to change all `entity="kth-rpl"` to your own entity.
97
+
## 1. Data Preparation
79
98
80
-
### Data Preparation
81
-
82
-
Check [dataprocess/README.md](dataprocess/README.md#argoverse-20) for downloading tips for the raw Argoverse 2 dataset. Or maybe you want to have the **mini processed dataset** to try the code quickly, We directly provide one scene inside `train` and `val`. It already converted to `.h5` format and processed with the label data.
83
-
You can download it from [Zenodo](https://zenodo.org/records/13744999/files/demo_data.zip)/[HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/demo_data.zip) and extract it to the data folder. And then you can skip following steps and directly run the [training script](#train-the-model).
99
+
Check [dataprocess/README.md](dataprocess/README.md#argoverse-20) for downloading tips for the raw Argoverse 2 dataset.
100
+
Or maybe you want to have the **mini processed dataset** to try the code quickly, We directly provide one scene inside `train` and `val`.
101
+
It already converted to `.h5` format and processed with the label data.
102
+
You can download it from [Zenodo](https://zenodo.org/records/13744999/files/demo_data.zip)/[HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/demo_data.zip) and extract it to the data folder.
103
+
Then you can directly use demo data to run the [training script](#2-quick-start).
Checking more information (step for downloading raw data, storage size, #frame etc) in [dataprocess/README.md](dataprocess/README.md). Extract all data to unified `.h5` format.
93
-
[Runtime: Normally need 45 mins finished run following commands totally in setup mentioned in our paper]
python dataprocess/extract_av2.py --av2_type sensor --data_mode val --mask_dir /home/kin/data/av2/3d_scene_flow
97
-
python dataprocess/extract_av2.py --av2_type sensor --data_mode test --mask_dir /home/kin/data/av2/3d_scene_flow
98
-
```
112
+
### SeFlow
99
113
100
-
#### Process train data
101
-
102
-
Process train data for self-supervised learning. Only training data needs this step. [Runtime: Normally need 15 hours for my desktop, 3 hours for the cluster with five available nodes parallel running.]
114
+
Train SeFlow needed to specify the loss function, we set the config of our best model in the leaderboard. [Runtime: Around 11 hours in 4x A100 GPUs.]
Or you can directly download the pre-trained weight from [Zenodo](https://zenodo.org/records/13744999/files/seflow_best.ckpt)/[HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/seflow_best.zip) and skip the training step.
125
+
### DeFlow
117
126
118
-
### Other Benchmark Models
127
+
Train DeFlow with the leaderboard submit config. [Runtime: Around 6-8 hours in 4x A100 GPUs.] Please change `batch_size&lr` accoordingly if you don't have enough GPU memory. (e.g. `batch_size=6` for 24GB GPU)
119
128
120
-
You can also train the supervised baseline model in our paper with the following command. [Runtime: Around 10 hours in 4x A100 GPUs.]
> You may found the different settings in the paper that is all methods are enlarge learning rate to 2e-4 and decrease the epochs to 20 for faster converge and better performance.
128
-
> However, we kept the setting on lr=2e-6 and 50 epochs in (SeFlow & DeFlow) paper experiments for the fair comparison with ZeroFlow where we directly use their provided weights.
129
-
> We suggest afterward researchers or users to use the setting here (larger lr and smaller epoch) for faster converge and better performance.
You can view Wandb dashboard for the training and evaluation results or upload result to online leaderboard.
134
141
135
142
Since in training, we save all hyper-parameters and model checkpoints, the only thing you need to do is to specify the checkpoint path. Remember to set the data path correctly also.
136
143
137
144
```bash
138
-
# downloaded pre-trained weight, or train by yourself
And the terminal will output the command for you to submit the result to the online leaderboard. You can follow [this section for evalai](https://github.com/KTH-RPL/DeFlow?tab=readme-ov-file#2-evaluation).
153
+
<!--And the terminal will output the command for you to submit the result to the online leaderboard. You can follow [this section for evalai](https://github.com/KTH-RPL/DeFlow?tab=readme-ov-file#2-evaluation).
150
154
151
-
Check all detailed result files (presented in our paper Table 1) in [this discussion](https://github.com/KTH-RPL/DeFlow/discussions/2).
155
+
Check all detailed result files (presented in our paper Table 1) in [this discussion](https://github.com/KTH-RPL/DeFlow/discussions/2).-->
152
156
153
-
## 3. Visualization
157
+
## 4. Visualization
154
158
155
159
We provide a script to visualize the results of the model also. You can specify the checkpoint path and the data path to visualize the results. The step is quickly similar to evaluation.
author={Zhang, Qingwen and Yang, Yi and Li, Peizheng and Andersson, Olov and Jensfelt, Patric},
178
-
title={{SeFlow}: A Self-Supervised Scene Flow Method in Autonomous Driving},
179
-
booktitle={European Conference on Computer Vision (ECCV)},
180
-
year={2024},
181
-
pages={353–369},
182
-
organization={Springer},
183
-
doi={10.1007/978-3-031-73232-4_20},
184
-
}
185
-
@inproceedings{zhang2024deflow,
186
-
author={Zhang, Qingwen and Yang, Yi and Fang, Heng and Geng, Ruoyu and Jensfelt, Patric},
187
-
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
188
-
title={{DeFlow}: Decoder of Scene Flow Network in Autonomous Driving},
189
-
year={2024},
190
-
pages={2105-2111},
191
-
doi={10.1109/ICRA57147.2024.10610278}
192
-
}
193
-
```
194
-
195
-
💞 Thanks to RPL member: [Li Ling](https://www.kth.se/profile/liling) helps revise our SeFlow manuscript. Thanks to [Kyle Vedder](https://kylevedder.github.io), who kindly opened his code (ZeroFlow) including pre-trained weights, and discussed their result with us which helped this work a lot.
177
+
## Acknowledgement
196
178
197
-
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and Prosense (2020-02963) funded by Vinnova.
179
+
These work were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and Prosense (2020-02963) funded by Vinnova.
198
180
The computations were enabled by the supercomputing resource Berzelius provided by National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg Foundation, Sweden.
<!-- *OpenSceneFlow* is designed by [Qingwen Zhang](https://kin-zhang.github.io/). It -->
201
183
184
+
❤️: Evaluation Metric from [BucketedSceneFlowEval](https://github.com/kylevedder/BucketedSceneFlowEval); README reference from [Pointcept](https://github.com/Pointcept/Pointcept); Many thanks to [ZeroFlow](https://github.com/kylevedder/zeroflow) ...
0 commit comments