FS#49272 - [cuda] Does not work with gcc 6.1.1

Attached to Project: Community Packages
Opened by pavan yalamanchili (pavanky) - Monday, 09 May 2016, 00:53 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Tuesday, 12 July 2016, 08:11 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

g++ defaults to c++14 for gcc 6.1, where as nvcc (which officially only supports) assumes the default is still c++98

Problem 1:

This causes problems when trying to compile simple cuda kernels. Attached is a file called (math.cu) which can reproduce this issue by running the following command:

nvcc -c math.cu

The errors generated are attached as default-options-errors.txt

-------------------------------------

Problem 2:

The above problem has a seemingly simple fix: pass -std=c++11 to nvcc which supports it. This however is causing a new kind of conflict with math operations.

Running the following command generates errors present in attachment named (std_c++11_errors.txt)

nvcc -std=c++11 -c math.cu

Please note the above works fine with gcc-5.3.0. The conflict seems to happen only on gcc 6.1.1 where something in the compilation process is importing the entire std namespace.

----------------------------------------


P.S. This is a problem from upstream, but I am creating the issue on request from the cuda package maintainer as mentioned here:
https://www.reddit.com/r/archlinux/comments/4iczja/a_work_around_if_you_are_using_cuda_and_have/d2xubjm
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Tuesday, 12 July 2016, 08:11 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed by using gcc5 for the foreseeable time.
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 09 May 2016, 03:47 GMT
Alright well, how can we hack this? There has to be a way.
Comment by pavan yalamanchili (pavanky) - Monday, 09 May 2016, 03:49 GMT
@svenstaro

I think the problem is in the nightmarish `math_functions.h` file. I am going to experiment a little bit when I have the time to try it out.
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 09 May 2016, 11:49 GMT
Right, I'd really appreciate it if you could help out here.
Comment by Jakub Klinkovský (lahwaacz) - Tuesday, 10 May 2016, 13:23 GMT
You can work around the problem with default standard by passing "--compiler-options -std=c++98" to nvcc. You'll get some warnings like

cc1: warning: command line option ‘-std=c++98’ is valid for C++/ObjC++ but not for C

even when compiling pure C++ code, but what the hell...
Comment by pavan yalamanchili (pavanky) - Wednesday, 11 May 2016, 22:17 GMT
@Svenstaro

New update:
A commentor on reddit[1] pointed out adding "-std=c++11 -Xcompiler -D__CORRECT_ISO_CPP11_MATH_H_PROTO" fixes the issues with math functions.

I was able to compile the math.cu file without any problems.

ArrayFire is still having issues which I am going to figure out.

[1] https://www.reddit.com/r/archlinux/comments/4iczja/a_work_around_if_you_are_using_cuda_and_have/d30duqi
Comment by Paul Hollensen (pallegro) - Wednesday, 11 May 2016, 22:34 GMT
Ya, I thought that was going to do it, but it appears there are more issues with any STL usage.
The define hopefully wouldn't break any existing code as g++ only started wrapping math.h in v6.

Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 23 May 2016, 15:41 GMT
Any further information on this?
Comment by Paul (dejavu) - Friday, 27 May 2016, 20:38 GMT
CUDA 8 RC is out [1], could the new version solve the problem?!

[1] http://phoronix.com/scan.php?page=news_item&px=CUDA-8-Release-Candidate
Comment by Jakub Klinkovský (lahwaacz) - Friday, 27 May 2016, 21:12 GMT
It's not quite out yet...

Phoronix says (http://phoronix.com/scan.php?page=news_item&px=CUDA-8-Release-Candidate):

> NVIDIA has made public the release candidate for CUDA 8.

NVIDIA says (https://developer.nvidia.com/cuda-toolkit):

> Download the CUDA Toolkit 8 RC today; requires membership of the the Accelerated Computing Developer Program
Comment by Sven-Hendrik Haase (Svenstaro) - Friday, 27 May 2016, 21:27 GMT
The license on this one prohibits me from packaging it before it's publicly accessible.
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 21:33 GMT
@lahwaacz

You just need to sign up for it. I am not sure if they even have a waiting period.

Here are my results so far:

- Anyway the installer complains about not supporting gcc 6.1.1

- host_config.h complains if gcc compiler version > 5.3. I commented this out for testing.

- still need to pass '-std=c++11 -Xcompiler -D__CORRECT_ISO_CPP11_MATH_H_PROTO' to get kernels to compile

- including stl headers in files with just CUDA kernels is not an issue

- including stl headers in files with any host code is still giving issues
Comment by Jakub Klinkovský (lahwaacz) - Friday, 27 May 2016, 21:43 GMT
@pavanky: I have, there is an entrance proceeding and an implied waiting period, so it's definitely not like they'd pass anybody. Even if that was the case, it couldn't be classified as "public access".
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 21:44 GMT
CUDA 8 seems to support clang as the host compiler, running the following command seems to work so far:

> /opt/cuda-8.0-rc1/bin/nvcc -ccbin clang++ -Xcompiler -stdlib=libc++ -Xlinker -lc++abi /tmp/math.cu -std=c++11

Note: clang with libstdc++ does not work either.

I am going to test building with clang 3.8 on a larger code base and let you know how it goes.
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 21:45 GMT
@lahwaacz, ah alright. I have had it for a very long time so it looks like my memory is not the best :)
Comment by Sebastiaan Lokhorst (lonaowna) - Friday, 27 May 2016, 22:11 GMT
We won't be able to package the RC since the download is password-protected (you need to log in with a developer program account):
https://developer.nvidia.com/compute/cuda/8.0/rc/local_installers/cuda_8.0.27_linux-run

But like pavanky said, it is still broken with GCC 6, so it doesn't really matter. The only option seems to use an old GCC.
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 23:04 GMT
OK I compiled arrayfire cuda backend with CUDA 8 + clang 3.8 + libc++. No issues other than annoying warnings.

As it stands, here are the work arounds:

- install gcc 5.3 to use CUDA without any problems (works with CUDA 7.5, CUDA 8.0)
- pass -std=c++98, if you want to use gcc 6.1 (works with CUDA 7.5, CUDA 8.0)
- pass -std=c++11 -ccbin clang++ -Xcompiler -stdlib=libc++ -Xlinker -lc++abi if you want C++11 (works only with CUDA 8.0, clang, libc++)

None of this provides an actual solution to packaging, but I am noting it down if someone want's to patch their project manually.
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 23:14 GMT
OK, the problem is just with stl headers from gcc. Using the clang headers seems to work for both CUDA 7.5 and CUDA 8.0 by passing the following options:

-Xcompiler -nostdinc++ -Xcompiler -I/usr/include/c++/v1 -Xlinker -lc++
Comment by Paul Hollensen (pallegro) - Friday, 27 May 2016, 23:38 GMT
@pavanky Good to know, thanks for testing out the possibilities.

As for the last option, using gcc with clang's stl, do you know of any project that does this?

Next time I'm at the lab I'll figure out how to tell bazel to compile tensorflow (... sigh) with both gcc6 and clang.
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 23:45 GMT
@pallegro, I don't know of any projects that use gcc with clang's STL. I just thought it would be interesting to try it out. I was able to compile and run a simple CUDA kernel using this.
Comment by Sven-Hendrik Haase (Svenstaro) - Friday, 27 May 2016, 23:46 GMT
Sooo... how do I package this? ;)
Comment by pavan yalamanchili (pavanky) - Friday, 27 May 2016, 23:53 GMT
@svenstaro I don't think you should yet. I'm just documenting the work arounds.

Im also digging into the temporary files to see what's causing the breakage. Aa you can understand, parsing through 30k-60k lines is a bit daunting.
Comment by sleepywitti (sleepywitti) - Tuesday, 31 May 2016, 15:41 GMT
A small addition for those who also had troubles using clang with the clang headers (I got some errors about the string class). Here is a small work around to use clang with gcc5 (and its headers):

- install gcc5 from AUR
- copy the stdc++ headers from an old gcc 5.3.0 package to /usr/include/c++/5.3.0/ (just google for `gcc-5.3.0-5-x86_64.pkg.tar.xz`, they are not part of the AUR gcc5 package)
- pass -std=c++11 -ccbin clang++ -Xcompiler --gcc-toolchain=/usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/ -Xcompiler -I/usr/include/c++/5.3.0 -Xcompiler -I/usr/include/c++/5.3.0/x86_64-unknown-linux-gnu to nvcc

I only tested it using CUDA 8.0 (with clang 3.8), but it might also work using CUDA 7.5 (with clang 3.6)

EDIT:
for the gcc5 package from the community repo you can skip the second step and pass the following to nvcc:
-std=c++11 -ccbin clang++ -Xcompiler --gcc-toolchain=/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/ -Xcompiler -I/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/c++ -Xcompiler -I/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/c++/x86_64-pc-linux-gnu
Comment by Sven-Hendrik Haase (Svenstaro) - Friday, 08 July 2016, 15:08 GMT
Any news on this perhaps?
Comment by Sebastiaan Lokhorst (lonaowna) - Monday, 11 July 2016, 09:55 GMT
I think the current approach (packaging gcc5) is the only proper way until Nvidia fixes their stuff. Messing around with the STL headers is too much of a hassle and prone to breakage. Thanks for adopting my AUR package.

By the way, CUDA 8.0 will check if GCC<=5.x.x, so we can just keep that check (not remove it[1]) and users will get the nice original notification that CUDA is not compatible with GCC 6 if they try to use it.

[1] https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/cuda#n51
Comment by pavan yalamanchili (pavanky) - Monday, 11 July 2016, 10:07 GMT
I mean if people are going to install a new compiler and header, then standard version of clang and libc could be a potential solution as well.

On the news side, it seems highly unlikely NVIDIA is going to support gcc6 for a very long time.

@svenstaro, Would it be possible to add a message to the package mentioning that GCC6 is not compatible out of the box and point to potential workarounds (e.g this issue).
Comment by Sebastiaan Lokhorst (lonaowna) - Monday, 11 July 2016, 10:19 GMT

Loading...