Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Marching Cubes CUDA implementation generates face indices out of bounds #1751

Closed
kaftanski opened this issue Mar 7, 2024 · 2 comments
Closed

Marching Cubes CUDA implementation generates face indices out of bounds #1751

kaftanski opened this issue Mar 7, 2024 · 2 comments

Comments

@kaftanski
Copy link

🐛 Bugs / Unexpected behaviors

I am getting an index out of bounds in pytorch3d.ops.marching_cubes.py in line:

pytorch3d/pytorch3d/ops/marching_cubes.py

Line 299 in a27755d

faces = inverse_idx[faces]

The issue occurs only in the CUDA implementation, which generates face-indices much larger than possible (e.g. 1112276992 with only 506 verts in the example below).
I am unsure, when this happens, I have not been able to construct a case myself, it occurs rather randomly in my scripts.
There are no issues in the CPU-Implementation.

The stacktrace is the following:

/opt/conda/conda-bld/pytorch_1704987394225/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/core/formatters.py", line 226, in catch_format_error
    r = method(self, *args, **kwargs)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/core/formatters.py", line 711, in __call__
    printer.pretty(obj)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 394, in pretty
    return self.type_pprinters[cls](obj, self, cycle)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 641, in inner
    p.pretty(x)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 394, in pretty
    return self.type_pprinters[cls](obj, self, cycle)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 641, in inner
    p.pretty(x)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 411, in pretty
    return _repr_pprint(obj, self, cycle)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 779, in _repr_pprint
    output = repr(obj)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor.py", line 461, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 677, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 597, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 349, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 137, in __init__
    nonzero_finite_vals = torch.masked_select(
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Instructions To Reproduce the Issue:

Failure case volume: vol.pth.zip

Code to reproduce:

import torch
from pytorch3d.ops.marching_cubes import marching_cubes
vol = torch.load('vol.pth')
verts, faces = marching_cubes(vol, isolevel=0)  # works fine
verts, faces = marching_cubes(vol.cuda(), isolevel=0)  # index out of bounds

My environment (conda list). I installed pytorch3d via pip install "git+https://github.com/facebookresearch/pytorch3d.git".

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
addict                    2.4.0                    pypi_0    pypi
argparse                  1.4.0                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
batchgenerators           0.25                     pypi_0    pypi
blas                      1.0                         mkl  
blinker                   1.7.0                    pypi_0    pypi
blosc                     1.21.3               h6a678d5_0  
bottleneck                1.3.7           py310ha9d4c09_0  
brotli                    1.0.9                h5eee18b_7  
brotli-bin                1.0.9                h5eee18b_7  
brotli-python             1.0.9           py310h6a678d5_7  
brunsli                   0.1                  h2531618_0  
bzip2                     1.0.8                h5eee18b_5  
c-ares                    1.19.1               h5eee18b_0  
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cfitsio                   3.470                h5893167_7  
charls                    2.2.0                h2531618_0  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.7           py310h06a4308_0  
cloudpickle               2.2.1           py310h06a4308_0  
comm                      0.2.1                    pypi_0    pypi
configargparse            1.7                      pypi_0    pypi
contourpy                 1.2.0           py310hdb19cb5_0  
cuda-cudart               12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.3.101                      0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cycler                    0.11.0             pyhd3eb1b0_0  
cyrus-sasl                2.1.28               h52b45da_1  
cytoolz                   0.12.2          py310h5eee18b_0  
dash                      2.16.0                   pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
dask-core                 2023.11.0       py310h06a4308_0  
dbus                      1.13.18              hb2f20db_0  
decorator                 5.1.1                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
expat                     2.5.0                h6a678d5_0  
fastjsonschema            2.19.1                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.13.1          py310h06a4308_0  
flask                     3.0.2                    pypi_0    pypi
fontconfig                2.14.1               h4c34cd2_2  
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.12.1               h4a9f257_0  
fsspec                    2023.10.0       py310h06a4308_0  
future                    1.0.0                    pypi_0    pypi
fvcore                    0.1.5.post20221221          pypi_0    pypi
giflib                    5.2.1                h5eee18b_3  
glib                      2.78.4               h6a678d5_0  
glib-tools                2.78.4               h6a678d5_0  
gmp                       6.2.1                h295c915_3  
gmpy2                     2.1.2           py310heeb90bb_0  
gnutls                    3.6.15               he1e5248_0  
gst-plugins-base          1.14.1               h6a678d5_1  
gstreamer                 1.14.1               h5eee18b_1  
icu                       73.1                 h6a678d5_0  
idna                      3.4             py310h06a4308_0  
igl                       2.2.1           py310h784662a_2    conda-forge
imagecodecs               2021.8.26       py310hecf7e94_1  
imageio                   2.33.1          py310h06a4308_0  
importlib-metadata        7.0.1           py310h06a4308_0  
intel-openmp              2023.1.0         hdb19cb5_46306  
iopath                    0.1.10                   pypi_0    pypi
ipython                   8.22.2                   pypi_0    pypi
ipywidgets                8.1.2                    pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jinja2                    3.1.3           py310h06a4308_0  
joblib                    1.2.0           py310h06a4308_0  
jpeg                      9e                   h5eee18b_1  
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter-core              5.7.1                    pypi_0    pypi
jupyterlab-widgets        3.0.10                   pypi_0    pypi
jxrlib                    1.1                  h7b6447c_2  
kiwisolver                1.4.4           py310h6a678d5_0  
krb5                      1.20.1               h143b758_1  
lame                      3.100                h7b6447c_0  
lazy_loader               0.3             py310h06a4308_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libaec                    1.0.4                he6710b0_1  
libblas                   3.9.0           1_h86c2bf4_netlib    conda-forge
libbrotlicommon           1.0.9                h5eee18b_7  
libbrotlidec              1.0.9                h5eee18b_7  
libbrotlienc              1.0.9                h5eee18b_7  
libcblas                  3.9.0           5_h92ddd45_netlib    conda-forge
libclang                  14.0.6          default_hc6dbbc7_1  
libclang13                14.0.6          default_he11475f_1  
libcublas                 12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufile                 1.8.1.2                       0    nvidia
libcups                   2.4.2                h2d74bed_1  
libcurand                 10.3.4.107                    0    nvidia
libcurl                   8.5.0                h251f7ec_0  
libcusolver               11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libdeflate                1.8                  h7f8727e_5  
libedit                   3.1.20230828         h5eee18b_0  
libev                     4.33                 h7f8727e_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgfortran-ng            11.2.0               h00389a5_1  
libgfortran5              11.2.0               h1234567_1  
libglib                   2.78.4               hdc74915_0  
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.4                h5eee18b_0  
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
libllvm14                 14.0.6               hdb19cb5_3  
libnghttp2                1.57.0               h2d74bed_0  
libnpp                    12.0.2.50                     0    nvidia
libnvjitlink              12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libpng                    1.6.39               h5eee18b_0  
libpq                     12.17                hdbd6064_0  
libssh2                   1.10.0               hdbd6064_2  
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libtasn1                  4.19.0               h5eee18b_0  
libtiff                   4.4.0                hecacb30_2  
libunistring              0.9.10               h27cfd23_0  
libuuid                   1.41.5               h5eee18b_0  
libwebp                   1.2.4                h11a3e52_1  
libwebp-base              1.2.4                h5eee18b_1  
libxcb                    1.15                 h7f8727e_0  
libxkbcommon              1.0.1                h5eee18b_1  
libxml2                   2.10.4               hf1b16e4_1  
libzopfli                 1.0.3                he6710b0_0  
linecache2                1.0.0                    pypi_0    pypi
llvm-openmp               14.0.6               h9e868ea_0  
locket                    1.0.0           py310h06a4308_0  
lz4-c                     1.9.4                h6a678d5_0  
markupsafe                2.1.3           py310h5eee18b_0  
matplotlib                3.8.0           py310h06a4308_0  
matplotlib-base           3.8.0           py310h1128e8f_0  
matplotlib-inline         0.1.6                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344  
mkl-service               2.4.0           py310h5eee18b_1  
mkl_fft                   1.3.8           py310h5eee18b_0  
mkl_random                1.2.4           py310hdb19cb5_0  
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.2                hb69a4c5_1  
mpmath                    1.3.0           py310h06a4308_0  
munkres                   1.1.4                      py_0  
mysql                     5.7.24               h721c034_2  
nbformat                  5.9.2                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
nest-asyncio              1.6.0                    pypi_0    pypi
nettle                    3.7.3                hbbd107a_1  
networkx                  3.1             py310h06a4308_0  
numexpr                   2.8.7           py310h85018f9_0  
numpy                     1.26.4          py310h5f9d8c6_0  
numpy-base                1.26.4          py310hb5e798b_0  
open3d                    0.18.0                   pypi_0    pypi
opencv-python             4.9.0.80                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openjpeg                  2.4.0                h3ad879b_0  
openssl                   3.2.1                hd590300_0    conda-forge
packaging                 23.1            py310h06a4308_0  
pandas                    2.2.1           py310h6a678d5_0  
parso                     0.8.3                    pypi_0    pypi
partd                     1.4.1           py310h06a4308_0  
pcre2                     10.42                hebb0a14_0  
pexpect                   4.9.0                    pypi_0    pypi
pillow                    9.4.0           py310h6a678d5_0  
pip                       23.3.1          py310h06a4308_0  
platformdirs              4.2.0                    pypi_0    pypi
plotly                    5.19.0                   pypi_0    pypi
ply                       3.11            py310h06a4308_0  
plyfile                   1.0.3                    pypi_0    pypi
portalocker               2.8.2                    pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyamg                     5.0.1                    pypi_0    pypi
pycpd                     2.0.0                    pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.0.9           py310h06a4308_0  
pyqt                      5.15.10         py310h6a678d5_0  
pyqt5-sip                 12.13.0         py310h5eee18b_0  
pyquaternion              0.9.9                    pypi_0    pypi
pysocks                   1.7.1           py310h06a4308_0  
python                    3.10.13              h955ad1f_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-tzdata             2023.3             pyhd3eb1b0_0  
python_abi                3.10                    2_cp310    conda-forge
pytorch                   2.2.0           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
pytorch-cuda              12.1                 ha16c6d3_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytorch3d                 0.7.6                    pypi_0    pypi
pytz                      2023.3.post1    py310h06a4308_0  
pywavelets                1.5.0           py310ha9d4c09_0  
pyyaml                    6.0.1           py310h5eee18b_0  
qt-main                   5.15.2              h53bd1ea_10  
readline                  8.2                  h5eee18b_0  
referencing               0.33.0                   pypi_0    pypi
requests                  2.31.0          py310h06a4308_1  
retrying                  1.3.4                    pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
scikit-image              0.20.0          py310h6a678d5_0  
scikit-learn              1.3.0           py310h1128e8f_1  
scipy                     1.11.4          py310h5f9d8c6_0  
seaborn                   0.12.2          py310h06a4308_0  
setuptools                68.2.2          py310h06a4308_0  
simpleitk                 2.2.1           py310h3fd9d12_0    simpleitk
sip                       6.7.12          py310h6a678d5_0  
six                       1.16.0             pyhd3eb1b0_1  
snappy                    1.1.10               h6a678d5_1  
sqlite                    3.41.2               h5eee18b_0  
stack-data                0.6.3                    pypi_0    pypi
sympy                     1.12            py310h06a4308_0  
tabulate                  0.9.0                    pypi_0    pypi
tbb                       2021.8.0             hdb19cb5_0  
tenacity                  8.2.3                    pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
thop                      0.1.1-2209072238          pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
tifffile                  2021.7.2           pyhd3eb1b0_2  
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1           py310h06a4308_0  
toolz                     0.12.0          py310h06a4308_0  
torchaudio                2.2.0               py310_cu121    pytorch
torchtriton               2.2.0                     py310    pytorch
torchvision               0.17.0              py310_cu121    pytorch
tornado                   6.3.3           py310h5eee18b_0  
tqdm                      4.66.2                   pypi_0    pypi
traceback2                1.4.0                    pypi_0    pypi
traitlets                 5.14.1                   pypi_0    pypi
trimesh                   4.1.7              pyhd8ed1ab_0    conda-forge
typing_extensions         4.9.0           py310h06a4308_1  
tzdata                    2024a                h04d1e81_0  
unittest2                 1.1.0                    pypi_0    pypi
urllib3                   2.1.0           py310h06a4308_1  
wcwidth                   0.2.13                   pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2          py310h06a4308_0  
widgetsnbextension        4.0.10                   pypi_0    pypi
xz                        5.4.6                h5eee18b_0  
yacs                      0.1.8                    pypi_0    pypi
yaml                      0.2.5                h7b6447c_0  
zfp                       0.5.5                h295c915_6  
zipp                      3.17.0          py310h06a4308_0  
zlib                      1.2.13               h5eee18b_0  
zstd                      1.5.5                hc292b87_0
facebook-github-bot pushed a commit that referenced this issue Mar 7, 2024
@JaapSuter @facebook-github-bot
CUDA marching_cubes fix
7566530 
Summary:
Fix an inclusive vs exclusive scan mix-up that was accidentally introduced when removing the Thrust dependency (`Thrust::exclusive_scan`) and reimplementing it using `at::cumsum` (which does an inclusive scan).

This fixes two Github reported issues:

 * #1731
 * #1751

Reviewed By: bottler

Differential Revision: D54605545

fbshipit-source-id: da9e92f3f8a9a35f7b7191428d0b9a9ca03e0d4d
@JaapSuter
Copy link
Contributor

I believe this to be fixed by 7566530, I ran your repro locally and it succeeded. Do let me know if it's still broken for you. Thanks!

@kaftanski
Copy link
Author

I don't have any more issues after re-installing pytorch3d from master with your fix. Thank you so much for the very fast help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JaapSuter @kaftanski