Arch Linux

All CUDA-enabled programs are in fault state. Another solution in addition to the suggested one can be downgrading cuda to 11.1.1 until the release of nvidia 460.

This seems like the kind of situation where we should revert with an epoch...

Hang on. I specifically test cuda compatibility every time I upgrade cuda when there's only a beta driver out and I could run tensorflow and pytorch just fine. How do I reproduce your issues? You didn't mention anything specific.

It looks like that only some CUDA functions are effected
https://docs.nvidia.com/cuda/parallel-thread-execution/#changes-in-ptx-isa-version-7-2
and the "basic functions" (e.g. quering a device) are still working.

This is an output of a small CUDA code (only basic functions) to query my device:
$ ./dp
CUDA Device Query...
There are 1 CUDA devices.

CUDA Device #0
Major revision number: 5
Minor revision number: 2
Name: GeForce GTX 970
Total global memory: 4236115968
Total shared memory per block: 49152
Total registers per block: 65536
Warp size: 32
Maximum memory pitch: 2147483647
Maximum threads per block: 1024
Maximum dimension 0 of block: 1024
Maximum dimension 1 of block: 1024
Maximum dimension 2 of block: 64
Maximum dimension 0 of grid: 2147483647
Maximum dimension 1 of grid: 65535
Maximum dimension 2 of grid: 65535
Clock rate: 1228000
Total constant memory: 65536
Texture alignment: 512
Concurrent copy and execution: Yes
Number of multiprocessors: 13
Kernel execution timeout: No



I observed the same problem when moving from CUDA 11.0 to 11.1 while running an older driver:
CUDA 11.1 GA >= 455.23 >= 456.38
CUDA 11.0.3 Update 1 >= 450.51.06 >= 451.82
https://github.com/hashcat/hashcat/issues/2626

Soon, we see 5.10 LTS kernel and nvidia 460.27. So the best solution (for me) is to add cuda to "IgnorePkg" until the new versions arrived.

Find the same problems. In my case most of my Tensorflow code ran fine, but some tests failed which I traced back to this problem. The suggested solution (--ignore cuda when updating) works, I don't like it that much but I guess the Linux driver will be out of beta soonish.

I wonder why NVIDIA released CUDA 11.2 before release of driver 460.27.4 (with regard to the API change 7.1 -> 7.2).

BTW: The issue is confirmed on other distros, too:
adds uvm kernel module support for Kernel >= 5.9; which is reenabled now, i.e. things like CUDA are working again with kernels >= 5.9
https://opensuse.pkgs.org/15.2/nvidia-x86_64/nvidia-computeG05-460.27.04-lp152.33.1.x86_64.rpm.html

That let me assume, we (Arch) are not the only one running into that issue.

Stay healthy,
cheers
Mike

Can you try the beta driver and see whether actually improves things?

For me it doesn't work. 460.27.04 boot into a black screen with a flashing cursor on top left. Maybe I missed something while building the 460.27 packages which ended in a "dependency hell". It looks like the driver need some more attention as expected.

Maybe my approach (modify your PKGBUILD for nvidia, nvidia-utils and opencl-nvidia) was too simple and must inevitably fail here. Sometimes that had worked, but unfortunately not on 460.27.04.
Tried that on 5.9.14-arch1-1. Maybe 460.27.04 is improved for 5.10.1 - but I'm not sure.

What you did should work as that's pretty much what I do too. Anyway now I'm not too keen on putting the beta drivers into repos.

Yes, for sure, putting a beta driver into the repos is definitely not a good idea.
But I will continue to test the driver in combination with 5.10.1.
Maybe I'm able to figure out, what went wrong.
BTW:
Your PKGBULDs are excellent. Worked before like a charm.

@Svenstro
It looks like that the combination kernel 5.10.1 -> nvidia 460.27.04 is working much better kernel 5.9.14 -> 460.27.04.
We can assume that, if the final driver is released, it will work fine with kernel 5.10 and you shouldn't waste your time, trying to get it work on 5.9.xx

Unfortunately there are still some issues (notebooks) in combination with I have to deal with:
ASUS (TUF gaming) notebook: AMD integrated GPU + NVIDA PCIe card (GTX 1650)
ASUS notebook: Intel integrated GPU + NVIDIA PCIe card (M940)

After turning on the notebook sometimes it takes more than 5 times to reboot the notebook until the NVIDIA card is detected and I'll not run into a black screen.
But I'm not sure if this issue is really related to the beta driver or my xorg config's (attached it - maybe I'm too stupid to generate a correct one and you have a better idea).

20-nvidia.conf.amd_nvidia:
Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:1:0:0"
VendorName "NVIDIA Corporation"
Option "NoLogo" "1"
Option "Interactive" "0"
Option "Coolbits" "12"
Option "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
Identifier "amd"
Driver "amdgpu"
BusID "PCI:5:0:0"
EndSection

Section "Screen"
Identifier "amd"
Device "amd"
EndSection


20-nvidia.conf.intel_nvidia:
Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:1:0:0"
VendorName "NVIDIA Corporation"
Option "NoLogo" "1"
Option "Interactive" "0"
Option "Coolbits" "12"
Option "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
Identifier "intel"
Driver "modesetting"
EndSection

Section "Screen"
Identifier "intel"
Device "intel"
EndSection


Stay healthy
cheers
Mike

@Svenstaro
At least I found the issue. Due to fast SSDs my notebooks booting too fast and I have "slow down" them during boot to prevent systemd attempt to start the display manager before the NVIDIA driver has fully initialized.
After adding an udev rule, the combination of kernel 5.10.1 and nvidia 460.27.04 is working fine.
Now we can wait until final nvidia 460.27 will be released and 5.10 leaves testing.

I was able to reproduce this issue only when I compiled my CUDA program for a GPU architecture that does not match the actual hardware, in which case the CUDA runtime takes the embedded PTX and invokes JIT compiles it. For example, if I compile with "-arch sm_61" instead of "-arch sm_75", I get an error, but compiling directly with "-arch sm_75" works. This explains why svenstaro's tensorflow and pytorch work fine - they are compiled explicitly for many (all?) GPU architectures so JIT was not invoked for svenstaro's hardware.

Fixed by nvidia 460.32.03-1
We can close this report.
Thanks.
Happy new year,
cheers
Mike