RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

[knorrrr]

I got an error when running run_test_interframe.py according to README.md.
How to solve it?

/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /opt/conda/conda-bld/pytorch_1646755903507/work/aten/src/ATen/native/TensorShape.cpp:2228.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Testing curious-oath-7:   0%|                                                                     | 0/19 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "scripts/run_test_interframe.py", line 85, in <module>
    metrics, detections_one_offset = run_test_with_visualization(test_loader, ema.ema, dataset=args.dataset, name=wandb.run.name, compile_detections=True,
  File "/home/apollon-22/dagr/dagr/src/dagr/utils/testing.py", line 33, in run_test_with_visualization
    detections, targets = model(data.clone())
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/apollon-22/dagr/dagr/src/dagr/model/networks/dagr.py", line 91, in forward
    outputs = YOLOX.forward(self, x)
  File "/home/apollon-22/dagr/dagr/libs/YOLOX/yolox/models/yolox.py", line 30, in forward
    fpn_outs = self.backbone(x)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/apollon-22/dagr/dagr/src/dagr/model/networks/net.py", line 108, in forward
    image_feat, image_outputs = self.net(data.image)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/apollon-22/dagr/dagr/src/dagr/model/networks/net_img.py", line 125, in forward
    self.module(x)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torchvision/models/resnet.py", line 283, in forward
    return self._forward_impl(x)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torchvision/models/resnet.py", line 278, in _forward_impl
    x = self.fc(x)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/apollon-22/anaconda3/envs/dagr/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

[JranDiana]

After useing “unset LD_LIBRARY_PATH”，the previous error disappeared, but a new error occurred.