Community Packages

Please read this before reporting a bug:
http://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in Unsupported. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#40231 - [julia] Use of system LLVM 3.4 is not supported and can lead to some problems

Attached to Project: Community Packages
Opened by CDSousa (cdsousa) - Monday, 05 May 2014, 23:10 GMT
Last edited by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 13 May 2014, 19:25 GMT
Task Type Bug Report
Category Upstream Bugs
Status Closed
Assigned To Alexander F Rødseth (xyproto) (trontonic)
Architecture x86_64
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Julia does not officially support LLVM 3.4 as stated many times by Julia devs.
See, for example, here
https://github.com/JuliaLang/julia/pull/4597#issuecomment-32568377
and here
https://github.com/JuliaLang/julia/issues/6275#issuecomment-41950714

Current Julia package uses system LLVM, which is in version 3.4.
This had lead to segfaults (only in the past it seems, as I haven't got any in recent versions).

Another problem is the "suffer in silence" due to system LLVM not providing backtraces, see here https://github.com/JuliaLang/julia/issues/6275#issuecomment-42220595
For this I can't tell whether the issue comes from the LLVM version or from the fact of Julia being using the system LLVM.

If no solution is found, I propose doing USE_SYSTEM_LLVM=0 and check again when Arch ships LLVM 3.5.

Additional info:
* package version(s)
julia 0.3.0_prerelease-4
* config and/or log files etc.


Steps to reproduce:
This task depends upon

Closed by  Alexander F Rødseth (xyproto) (trontonic)
Tuesday, 13 May 2014, 19:25 GMT
Reason for closing:  Fixed
Additional comments about closing:  Needs to set USE_SYSTEM_LLVM=0 and a specific commit in order to make julia compile and also get proper error messages.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 06 May 2014, 14:02 GMT
Hi,

Thanks for reporting.

I looked through the included links and I could not reproduce the issue. Everything I tried worked fine here. What are the steps to reproduce the problem? This would help pinpointing the issue.

It's an unusual practice that packages try to include their own version of everything, and I would like to avoid it, if possible.

While the unofficial opinion from one of the Julia contributors is that only LLVM 3.3 (and then 3.5 in the future) are supported, the official Julia documentation says nothing of this sort:
https://github.com/JuliaLang/julia/blob/master/README.md#required-build-tools-and-external-libraries

If the system version of LLVM (version 3.4) causes problems together with Julia, I would say that this is an upstream bug in Julia, as long as the official documentation says that they do support LLVM in general (with an unspecified version number).

Steps to reproduce the issue would be most helpful. Corrections to the official documentation would also be clarifying, but this might require filing a bug report with Julia.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 06 May 2014, 14:29 GMT
On a second read, I found this section:

"Please be aware that this procedure is not officially supported, as it introduces additional variability into the installation and versioning of the dependencies, and is recommended only for system package maintainers. Unexpected compile errors may result, as the build system will do no further checking to ensure the proper packages are installed."

While this could be thought to apply to the situation at hand (to not use the system version of LLVM), they still seem to do recommend it for system package maintainers. And I have not seen any compilation errors as a result of using LLVM 3.4, nor problems with the build system.

It still seems to me like their official stance is to support LLVM. If LLVM is installed on the system and Julia has problems with LLVM, it seems to me like it is a problem with Julia.

In any case, a list of steps to reproduce the issue is the first matter to look into.
Comment by CDSousa (cdsousa) - Tuesday, 06 May 2014, 15:04 GMT
Hi, sorry for the missing steps to reproduce the issue:

In my x86_64 Arch, with the Julia repository package, for this code (taken from [1]):

julia> function myfunc()
b = true
if b
error("1")
else
error("0")
end
end
julia> myfunc()

only "ERROR: 1" is output, when it should output the backtrace:
"ERROR: 1
in myfunc at none:4"

While in this example the backtrace is not that necessary, it is very much in the general case.
"not having good backtraces might lead to a "suffer in silence" kind of problem,
and it might take a while for people to realize what they are missing" [2]


While the docs don't state 3.4 is not supported, the opinion from Jeff Bezanson is very strong as he is
one of the Julia founders. It seems this issue is well known: "Our difficulty is that LLVM 3.4 removed
exception support from the legacy JIT in 3.4".


At least for the AUR julia-git package the issue appears/disappears by just changing USE_SYSTEM_LLVM=1/0.
I'll try to compile this community package by myself with USE_SYSTEM_LLVM=0 to double check the issue.
I'll do some test using a downgraded LLVM 3.3 too.

(A strange thing it that aur package with USE_SYSTEM_LLVM=0 is only 1MB bigger than the community package...)


[1] https://github.com/JuliaLang/julia/issues/6275#issuecomment-38965669
[2] https://github.com/JuliaLang/julia/issues/6275#issuecomment-42220595
Comment by CDSousa (cdsousa) - Tuesday, 06 May 2014, 15:09 GMT
AH, it seems that with last versions a "make testall" will spot the missing backtraces
https://github.com/JuliaLang/julia/issues/6275#issuecomment-42220595
Comment by CDSousa (cdsousa) - Tuesday, 06 May 2014, 16:05 GMT
I've just did some tests and I got some strange results.

What I did:
- started with llvm and llvm-libs 3.4
- compiled julia (this community package) letting USE_SYSTEM_LLVM=1
- tested the issue: got NO backtraces as expected
- downgraded llvm and llvm-libs to 3.3 (from A.R.M.)
- tested the issue: still got NO backtraces... hum
- compiled julia with USE_SYSTEM_LLVM=1 once again
- tested the issue: GOT backtraces now
- upgrade llvm and llvm-libs back to 3.4
- tested the issue: still GOT backtraces...
- force-removed llvm and llvm-libs
- tested the issue: julia still runs and GET backtraces

So it seems LLVM is a "makedependency" rather than a normal dependency...
Comment by Alexander F Rødseth (xyproto) (trontonic) - Wednesday, 07 May 2014, 10:52 GMT
Thanks for testing variations of USE_SYSTEM_LLVM and different versions of llvm! Will attempt to build with "make testall" with the very latest version of Julia and see what happens.
And thanks for the steps to reproduce as well. Will see what I can make of this and if it is at all avoidable to include LLVM in the Julia package. That the package works as it should is first priority, of course.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Thursday, 08 May 2014, 10:56 GMT
Tried compiling the latest commit of Julia with and without llvm as deps/makedeps and with USE_SYSTEM_LLVM=0, but the compilation failed here, possibly in connection with SuiteSparse. Will test more.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Thursday, 08 May 2014, 14:32 GMT
Were you able to compile the latest version of Julia with USE_SYSTEM_LLVM=0? The julia-git package in AUR also seems to use USE_SYSTEM_LLVM=1.

I consider this to be an upstream issue. Please consider filing a bug report/support request with Julia, requesting USE_SYSTEM_LLVM=0 to work and/or LLVM 3.4 to be supported.
Comment by CDSousa (cdsousa) - Thursday, 08 May 2014, 16:42 GMT
I was able to compile Julia "Version 0.3.0-prerelease+2954 Commit 3985890" today,
using AUR PKGBUILD julia-git modified to USE_SYSTEM_LLVM=0 (in both build() and package()).

I also compiled "Version 0.3.0-prerelease+2956 Commit e7694b6" with USE_SYSTEM_LLVM=1 with no
build problem (missing backtraces issue is still there).

The package size difference between USE_SYSTEM_LLVM=0 and USE_SYSTEM_LLVM=1 is around 2MB.
The difference in file listing between USE_SYSTEM_LLVM=0 and USE_SYSTEM_LLVM=1, since
$ pacman -Qlp julia-git-0.2.0.2954.g3985890-1-x86_64.pkg.tar.xz > a
$ pacman -Qlp julia-git-0.2.0.2956.ge7694b6-1-x86_64.pkg.tar.xz > b
$ diff a b
returns nothing.

$ yaourt -Ql julia-git | grep -i llvm
returns nothing...

It seems that Julia depends on LLVM on build only, maybe it links LLVM libraries statically.

Maybe it would be not a big deal to compile Julia community package with USE_SYSTEM_LLVM=0.
Anyway, I not an Arch package maintainer and I'm not completely aware of how much that is not recomendable.

Comment by CDSousa (cdsousa) - Thursday, 08 May 2014, 16:45 GMT
Sorry,

"The package size difference between USE_SYSTEM_LLVM=0 and USE_SYSTEM_LLVM=1 is around 2MB."
must be
"The package installation size difference between USE_SYSTEM_LLVM=0 and USE_SYSTEM_LLVM=1 is around 2MB."

"The difference in file listing between USE_SYSTEM_LLVM=0 and USE_SYSTEM_LLVM=1, since"
must be
"The difference in file listing between USE_SYSTEM_LLVM=0 and USE_SYSTEM_LLVM=1 is none, since"
Comment by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 13 May 2014, 12:26 GMT
Building the latest commit with USE_SYSTEM_LLVM=0 now. If it builds and runs fine, and the problem is solved, I'll push that.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 13 May 2014, 12:32 GMT
The latest commit does not build with USE_SYSTEM_LLVM=0 here. Trying with USE_SYSTEM_LLVM=1 instead.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 13 May 2014, 12:33 GMT
Trying commit 3985890 + LLVM=0 next.
Comment by Alexander F Rødseth (xyproto) (trontonic) - Tuesday, 13 May 2014, 18:32 GMT
Commit 3985890 and USE_SYSTEM_LLVM=0 both builds and runs fine and also fixes this problem. I see no other option than setting it to 0, for now.

Uploaded the new package to [community]. It should appear shortly.

Many thanks for all the testing and research!

Loading...