FS#65206 - [syslog-ng] 3.25.1-2 doesn't log anything

Attached to Project: Arch Linux
Opened by Lone_Wolf (Lone_Wolf) - Sunday, 19 January 2020, 13:52 GMT
Last edited by Florian Pritz (bluewind) - Sunday, 07 May 2023, 09:21 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Florian Pritz (bluewind)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 17
Private No

Details

Description:

[2020-01-17T23:36:31+0100] [PACMAN] Running 'pacman -Syu'
[2020-01-17T23:36:31+0100] [PACMAN] synchronizing package lists
[2020-01-17T23:36:32+0100] [PACMAN] starting full system upgrade
[2020-01-17T23:36:43+0100] [ALPM] transaction started
[2020-01-17T23:36:44+0100] [ALPM] upgraded qt5-base (5.14.0-1 -> 5.14.0-2)
[2020-01-17T23:36:44+0100] [ALPM] upgraded snappy (1.1.7-1 -> 1.1.8-1)
[2020-01-17T23:36:44+0100] [ALPM] upgraded syslog-ng (3.25.1-1 -> 3.25.1-2)
[2020-01-17T23:36:44+0100] [ALPM] transaction completed
[2020-01-17T23:36:44+0100] [ALPM] running '30-systemd-daemon-reload.hook'...
[2020-01-17T23:36:44+0100] [ALPM] running '30-systemd-update.hook'...After updating

Since this update syslog-ng has not logged anything.
journal doesn't show errrors upon starting syslog-ng@default.service and there is a process running for it.
On shutdown a message about a stop job is shown.
full journal from 1 boot at http://ix.io/27P5

Investigating files created by syslog-ng to log things like /var/log/auth.log & /var/log/syslog.log show they have are created but stay at 0 bytes
Downgrading syslog-ng to 3.25.1-1 restores all functionality.

Additional info:
syslog-ng 3.25.1-2
/etc/syslog-ng/syslog-ng.conf : http://ix.io/27P6

Steps to reproduce:
install syslog-ng 3.25.1-2 and start syslog-ng@default.service
check content of var/log/syslog.log , it should look similar to this :

Jan 19 14:11:13 silverbolt syslog-ng[804]: syslog-ng starting up; version='3.25.1'
Jan 19 14:11:13 silverbolt syslog-ng[804]: [2020-01-19T14:11:13.363900] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!;
This task depends upon

Closed by  Florian Pritz (bluewind)
Sunday, 07 May 2023, 09:21 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed long ago. Closing.
Comment by Lone_Wolf (Lone_Wolf) - Sunday, 19 January 2020, 16:29 GMT
For clarity :

Jan 19 14:11:13 silverbolt syslog-ng[804]: syslog-ng starting up; version='3.25.1'
Jan 19 14:11:13 silverbolt syslog-ng[804]: [2020-01-19T14:11:13.363900] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!;

Lines like those are shown in /var/log/syslog.log when syslog-ng functions as intended and DOES log messages.
With 3.25.1-2 that file is empty.
Comment by stargazer (bernie) - Sunday, 19 January 2020, 17:56 GMT
I have the same problem with 3.25.1-2.

syslog-ng 3.25.1-1 is ok.

Comment by Ash (eta-carinae) - Monday, 20 January 2020, 10:25 GMT
Same issue here. I have a server where the syslog-ng sources are the local system plus several other remote systems using the syslog protocol over UDP. Some of the systems are debian-based and use syslog-ng 3.19. None of their logs appear with 3.25.1-2. The physical destination files are pretty typical - auth.log, daemon.log, debug, error, messages, and so on. All have file size zero with this version.
Comment by Klaus Alexander Seistrup (kseistrup) - Monday, 20 January 2020, 15:10 GMT
Same issue here. Seen on a handful of machines.
Comment by Kelly Prescott (krprescott) - Wednesday, 22 January 2020, 01:12 GMT
I have the exact same problem on several machines, some at home and some at work... downgrading to 3.25.1-1 fixes problem.
Comment by László Várady (MrAnno) - Wednesday, 22 January 2020, 13:33 GMT
Rebuilding 3.25.1-1 (the one that worked before) with makechrootpkg also produces a buggy package, so the issue must have been triggered with something environmental.
Comment by hamelg (hamelg) - Wednesday, 22 January 2020, 21:55 GMT
Me too :(
Something is broken with 3.25.1-2.
When you stop the service, syslog-ng never terminates.
you must kill it with SIGKILL.
Comment by Andreas (bjoe2k4) - Thursday, 23 January 2020, 07:05 GMT
Same here 3.25.1-2 is broken, 3.25.1-1 works. 5 days of logs lost due to this :-(
Comment by edac val (edacval) - Thursday, 23 January 2020, 10:51 GMT
Think i found the cause - thanks for the hint @MrAnno ! :

syslog-ng deadlocks at start when built against libcap-2.29-1, can be killed only with `kill -9 $(cat /run/syslog-ng/pid)`
syslog-ng works properly, when built against libcap-2.28-1.

Steps to reproduce:

cd ~
sudo pacman -Syu
sudo pacman -S --needed base-devel asp wget
wget https://archive.archlinux.org/repos/2020/01/07/core/os/x86_64/libcap-2.28-1-x86_64.pkg.tar.xz
# Temporary downgrade libcap
sudo pacman -U libcap-2.28-1-x86_64.pkg.tar.xz
asp checkout syslog-ng
cd syslog-ng/repos/extra-x86-64
makepkg --clean --cleanbuild --force --install --syncdeps --rmdeps
# Back to current libcap
sudo pacman -S libcap
# enjoy working syslog-ng :)
sudo systemctl start syslog-ng@default.service



Comment by László Várady (MrAnno) - Thursday, 23 January 2020, 13:20 GMT
@edacval Wow, nice.
Comment by László Várady (MrAnno) - Thursday, 23 January 2020, 14:04 GMT
It seems one of syslog-ng's libraries (ivykis) does not spawn worker threads, so the end result looks like a deadlock.
The reason why syslog-ng is not able to stop is because it waits for the non-existing thread pool to stop.

The root cause might be somewhere in the ivykis submodule and it is triggered by the libcap upgrade.
Comment by edac val (edacval) - Thursday, 23 January 2020, 14:35 GMT
Seems not only syslog-ng has problems with libcap 2.29 - systemd is affected too:
https://bugzilla.kernel.org/show_bug.cgi?id=206183
Comment by lamberto (lamberto65) - Friday, 24 January 2020, 13:28 GMT
I have the same problem with 3.25.1-2 on 4 pc .
syslog-ng 3.25.1-1 is ok.
Comment by astd (r0b0t) - Saturday, 25 January 2020, 00:58 GMT
@edacval thanks, it worked for me.
Comment by Florian Pritz (bluewind) - Saturday, 25 January 2020, 14:28 GMT
FWIW, building syslog-ng with libcap 2.30 does NOT appear to fix this problem so this is probably a different issue than the one that was linked here.
Comment by Florian Pritz (bluewind) - Saturday, 25 January 2020, 14:59 GMT
I've released syslog-ng 3.25.1-3 to [extra] built against libcap 2.28-1. At least for me, this package works. Please test.

Edit: Released to [extra], not [core].
Comment by Phil Collins (murr) - Saturday, 25 January 2020, 16:30 GMT
3.25.1-3 is working for me. Thanks.
Comment by László Várady (MrAnno) - Saturday, 25 January 2020, 17:29 GMT
3.25.1-3 works fine, thank you.

libcap >= 2.29 has a new compile flag:

```
$ pkg-config --libs libcap
-L/lib64 -lcap -lpsx -lpthread -Wl,-wrap,pthread_create
```

They actually wrap pthread_create, which somehow makes ivykis not to start threads if it was compiled with those flags (through syslog-ng).
It might have something to with the fact that ivykis uses weak symbols:
```
#pragma weak pthread_create
```
And this: https://github.com/buytenh/ivykis/blob/f1b14555fb0b5d9acfbcfaf35b1313bb28d858c2/src/pthr.h#L31-L34
Comment by Ash (eta-carinae) - Saturday, 25 January 2020, 19:13 GMT
3.25.1-3 works for me. Getting both local and remote logs now. Thanks.
Comment by László Várady (MrAnno) - Friday, 21 February 2020, 12:27 GMT
FYI: In libcap >= 2.31, they removed the linker flag that caused the problem:
https://bugzilla.kernel.org/show_bug.cgi?id=206113

They've made libpsx optional, so `-lpsx -lpthread -Wl,-wrap,pthread_create` is no longer part of the required linker flags of libcap.

While this makes syslog-ng work again when it is compiled with the latest libcap Arch package, the issue with the statically linked ivykis library and `-Wl,-wrap,pthread_create` still exists, so it probably has to be fixed upstream, because libpsx might become a hard dependency again in the future.

Loading...