[Bug] Segmentation fault while importing tvm

[JWXiang-404]

Thanks for participating in the TVM community! We use https://discuss.tvm.ai for any general usage questions and discussions. The issue tracker is used for actionable items such as feature proposals discussion, roadmaps, and bug tracking. You are always welcomed to post on the forum first 😸

Issues that are inactive for a period of time may get closed. We adopt this policy so that we won't lose track of actionable issues that may fall at the bottom of the pile. Feel free to reopen a new one if you feel there is an additional problem that needs attention when an old one gets closed.

Expected behavior

Successfully import tvm in python.

Actual behavior

Segmentation fault (core dumped) when running python -c "import tvm" after compiling TVM successfully follow this doc.

I've tried using faulthandler to locate the error:

#code.py
import faulthandler
faulthandler.enable()
import tvm

After running python -X faulthandler code.py, I got messages below:

Fatal Python error: Segmentation fault

Current thread 0x00007f01d5705280 (most recent call first):
  File "/root/tvm-18/python/tvm/_ffi/_ctypes/packed_func.py", line 235 in __call__
  File "/root/tvm-18/python/tvm/target/tag.py", line 33 in list_tags
  File "/root/tvm-18/python/tvm/target/tag.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/root/tvm-18/python/tvm/target/__init__.py", line 77 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
  File "/root/tvm-18/python/tvm/__init__.py", line 58 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "code.py", line 3 in <module>
Segmentation fault (core dumped)

There seems to be something wrong with the fuction in tvm/python/tvm/_ffi/_ctypes/packed_func.py as mentioned above while executing in my Environment.

Environment

• Ubuntu 20.04.6 LTS
• Docker image of 12.4.1-cudnn-devel-ubuntu20.04
• LLVM-18.1.8 prebuild version for Ubuntu 18.04: (download from here)
• Use Apache TVM v0.18.0 Release version, and config.cmake appending settings are: set(USE_CUDA ON), set(USE_CUBLAS ON), set(USE_CUDNN ON), set(USE_LLVM "\<path-to-llvm-config\> --ignore-libllvm --link-static") and set(HIDE_PRIVATE_SYMBOLS ON)
• Correctly set TVM_HOME and PYTHONPATH, use conda venv and python 3.8.

Steps to reproduce

python -c "import tvm"

Triage

• needs-triage
• core:ffi

[JWXiang-404]

New discovery: This issue may be caused by a version of LLVM that is too high. When the LLVM version used is 16 or above, it will compile code that leads to a Segmentation Fault, but the compilation itself passes; however, when using LLVM 15, everything works as expected.

[cbalint13]

@JWXiang-404 ,

I tried to reproduce steps (official doc) with latest head, and it works fine.
A similar reported issue (library loading) was fixed upstream a few days ago.

$ git log | grep -b3 commit | head -3
0:commit 513c2be0c3b853a3b77de729f0ea75d448ee3c37
48-Author: Renat Idrisov <[email protected]>
117-Date:   Mon Dec 2 14:24:00 2024 -0800

$ python -c "import tvm; print(tvm.__file__)"
/home/cbalint/work/GITHUB/tvm/python/tvm/__init__.py
(tvm-build-venv) [cbalint@yoda tvm]$ python
Python 3.11.10 | packaged by conda-forge | (main, Oct 16 2024, 01:27:36) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import tvm

>>> print(tvm.__file__)
/home/cbalint/work/GITHUB/tvm/python/tvm/__init__.py

>>> print(tvm.__version__)
0.19.dev0

>>> print(tvm.target.codegen.llvm_version_major())
19

If the problem persists with latest head checkout , could give some more detail of your environment ?

[JWXiang-404]

@JWXiang-404 ,

I tried to reproduce steps (official doc) with latest head, and it works fine. A similar reported issue (library loading) was fixed upstream a few days ago.

$ git log | grep -b3 commit | head -3
0:commit 513c2be0c3b853a3b77de729f0ea75d448ee3c37
48-Author: Renat Idrisov <[email protected]>
117-Date:   Mon Dec 2 14:24:00 2024 -0800

$ python -c "import tvm; print(tvm.__file__)"
/home/cbalint/work/GITHUB/tvm/python/tvm/__init__.py
(tvm-build-venv) [cbalint@yoda tvm]$ python
Python 3.11.10 | packaged by conda-forge | (main, Oct 16 2024, 01:27:36) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import tvm

>>> print(tvm.__file__)
/home/cbalint/work/GITHUB/tvm/python/tvm/__init__.py

>>> print(tvm.__version__)
0.19.dev0

>>> print(tvm.target.codegen.llvm_version_major())
19

If the problem persists with latest head checkout , could give some more detail of your environment ?

Thanks a lot!! I'll try.