FS#68879 - [python-pytorch-opt-cuda] CUDA error: no kernel image is available for execution on the device

Attached to Project: Community Packages
Opened by Harish Rajagopal (rharish) - Monday, 07 December 2020, 10:39 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Saturday, 12 December 2020, 11:33 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
I cannot use PyTorch on my GPU (Nvidia GTX 960M, compute capability 5.0). Any operation leads to the error: `RuntimeError: CUDA error: no kernel image is available for execution on the device`.

Additional info:
* package version(s):
- python-pytorch-opt-cuda: 1.7.1rc2-1
- cuda: 11.1.1-1
- cudnn: 8.0.5.39-1
* config and/or log files etc.
- Traceback:
```
Traceback (most recent call last):
File "/home/rharish/test.py", line 5, in <module>
a + 0
RuntimeError: CUDA error: no kernel image is available for execution on the device
```
* link to upstream bug report, if any:
There's one, but not sure if it's relevant: https://github.com/pytorch/pytorch/issues/31285

Steps to reproduce:
Run the attached code with CUDA.
   test.py (0.1 KiB)
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Saturday, 12 December 2020, 11:33 GMT
Reason for closing:  Won't fix
Comment by Harish Rajagopal (rharish) - Wednesday, 09 December 2020, 11:02 GMT
I've tested TensorFlow, and the same error occurs there too. I guess this is not a PyTorch bug, but a CUDA/cuDNN bug.

Additional info:
* package version(s):
- python-tensorflow-opt-cuda: 2.3.1-7
- tensorflow-opt-cuda: 2.3.1-7
- python-pytorch-opt-cuda: 1.7.1rc2-1
- cuda: 11.1.1-1
- cudnn: 8.0.5.39-1
- nvidia: 455.45.01-6
- nvidia-utils: 455.45.01-1
- linux: 5.9.13.arch1-1
* config and/or log files etc.
- Error message (fails with return code 134):
```
2020-12-09 12:01:08.513584: F ./tensorflow/core/kernels/random_op_gpu.h:230] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device
```

Steps to reproduce:
Run the attached code with CUDA enabled: `python test-tf.py`.
   test-tf.py (0.1 KiB)
Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 12 December 2020, 11:30 GMT
Works perfectly for me on tensorflow 2.3.1, 2.4.0rc4 and the provided test file on pytorch.
Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 12 December 2020, 11:32 GMT
Ah I see, your 960M is apparently too old. cuda 11.1 deprecated support for your compute level. I suppose you could try compiling pytorch and tensorflow for your architecture but you're not going to get official support (from either NVIDIA or Arch Linux) for it.

Loading...