FS#77800 - CUDA update to 12.1 breaks CUDA tools

Attached to Project: Community Packages
Opened by Michael (ZeroBeat) - Thursday, 09 March 2023, 22:21 GMT
Last edited by Toolybird (Toolybird) - Thursday, 23 March 2023, 20:29 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:
Latest CUDA update to version 12.1

$ pacman -Q | grep cuda
cuda 12.1.0-1

doesn't match to the driver
$ pacman -Q | grep nvidia
nvidia 525.89.02-6
nvidia-settings 525.89.02-1
nvidia-utils 525.89.02-2
opencl-nvidia 525.89.02-2

according to
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html
this driver (currently beta) is mandatory:
CUDA 12.1 GA >= 530.30.02

Trying to compile a CUDA application on current driver 525 will end up in an ERROR:

cuLinkAddData(): the provided PTX was compiled with an unsupported toolchain.
* Device #1: Kernel /usr/share/hashcat/OpenCL/shared.cl link failed. Error Log:
ptxas application ptx input, line 9; fatal : Unsupported .version 8.1; current version is '8.0'
* Device #1: Kernel /usr/share/hashcat/OpenCL/shared.cl build failed.

Solution:
Downgrade to 12.0 until driver 530.30.02 is in the repository.

I suggest to add a notice to "latest news" that users should not update to CUDA 12.1 unless 530.30 has been published.




This task depends upon

Closed by  Toolybird (Toolybird)
Thursday, 23 March 2023, 20:29 GMT
Reason for closing:  Fixed
Additional comments about closing:  nvidia 530.41.03-1
Comment by Sven-Hendrik Haase (Svenstaro) - Friday, 10 March 2023, 06:52 GMT
Can't reproduce. All of our dependents compiled fine and don't appear to be complaining.
Comment by Michael (ZeroBeat) - Friday, 10 March 2023, 07:12 GMT
We talked abut this problem already here:
https://bugs.archlinux.org/task/69005

The steps to reproduce it are the same as mentioned in FSX6905:
$ pacman -Q | grep cuda
cuda 12.0.1-1

$ git clone https://github.com/hashcat/hashcat
$ cd hashcat
$ make

everything is compiled fine.

Now do the update to CUDA 12.1 and compile it again:
cuLinkAddData(): the provided PTX was compiled with an unsupported toolchain.
* Device #1: Kernel /usr/share/hashcat/OpenCL/shared.cl link failed. Error Log:
ptxas application ptx input, line 9; fatal : Unsupported .version 8.1; current version is '8.0'
* Device #1: Kernel /usr/share/hashcat/OpenCL/shared.cl build failed.

API of the 525 dirver is 8.0
API of CUDA is 8.1

API of next driver 530 will be 8.1








Comment by Michael (ZeroBeat) - Friday, 10 March 2023, 07:15 GMT
To prevent this in future time I suggest to wait with CUDA (only after update an API change ) e.g 71 to 72, 80 to 81, ...) until the matching driver has been published, too.
Comment by Michael (ZeroBeat) - Friday, 10 March 2023, 07:25 GMT
As mentioned here:
https://bugs.archlinux.org/task/69005

"Comment by Michael (ZeroBeat) - Thursday, 17 December 2020, 15:36 GMT
I wonder why NVIDIA released CUDA 11.2 before release of driver 460.27.4 (with regard to the API change 7.1 -> 7.2)."

And I still wonder (again) why NVIDIA released CUDA 12.1 before release of driver 530 (with regard to the API change 8.0 -> 8.1).

Comment by Michael (ZeroBeat) - Friday, 10 March 2023, 07:28 GMT
BTW:
We can strongly assume that this will happen again on change of API 8.1 -> 8.2 and later on, on 9.0 -> 9.1
Regarding Linux, NVIDA always publish CUDA update before the driver has left beta status.

Comment by Michael (ZeroBeat) - Friday, 10 March 2023, 08:08 GMT
Looks like the CUDA update always come together with the Windows driver update:

GeForce Game Ready Driver
Version: 531.18 WHQL
Release Date: 2023.2.28
Operating System: Windows 10 64-bit, Windows 11
Language: English (US)
File Size: 855.68 MB


but never ever together with the Linux driver update:
Linux x64 (AMD64/EM64T) Display Driver
Version: 525.89.02
Release Date: 2023.2.8
Operating System: Linux 64-bit
Language: English (US)
File Size: 394.93 MB


Comment by Friedrich (F1729) - Sunday, 12 March 2023, 15:46 GMT
Thank you for posting this bug. The solution to downgrade CUDA from 12.1 to 12.0 allowed me to continue using dependent programs.

Current output of nvidia-smi (regardless whether CUDA 12.0 or CUDA 12.1 is installed):
NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0

Is there any reason not to retract the cuda 12.1 update until the compatible nvidia 530 update is available?
Comment by Michael (ZeroBeat) - Sunday, 12 March 2023, 16:51 GMT
Well, it happened (unfortunately). It is a nasty race condition between release of CUDA and release of driver.
Now a short message in latest news should be ok.
Usually it will not take a long time until the final version of the driver will be released.
The next upcoming CUDA/driver updates will be ok - as long as ptx doesn't change.
Luckily this will be announced in the compatibility table.



Comment by Michael (ZeroBeat) - Thursday, 23 March 2023, 15:51 GMT
We can close this report, due to release of 530.41.03
https://www.nvidia.com/download/driverResults.aspx/200481/en-us/

Loading...