FS#45451 - [systemd] journalctl broken on upgrade to 221

Attached to Project: Arch Linux
Opened by Daurnimator (daurnimator) - Wednesday, 24 June 2015, 13:25 GMT
Last edited by Evangelos Foutras (foutrelis) - Thursday, 30 July 2015, 05:54 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Thomas Bächler (brain0)
Evangelos Foutras (foutrelis)
Dave Reisner (falconindy)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 12
Private No

Details

since upgrading to 221 I can't use journalctl:

$ sudo journalctl
Error was encountered while opening journal files: Invalid argument

Looking via strace doesn't seem to give me any hints.

$ sudo journalctl --version
systemd 221
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN

$ pacman -Qi libsystemd systemd
Name : libsystemd
Version : 221-1
Description : systemd client libraries
Architecture : x86_64
URL : http://www.freedesktop.org/wiki/Software/systemd
Licenses : GPL2
Groups : None
Provides : libsystemd.so=0-64 libsystemd-daemon.so=0-64 libsystemd-id128.so=0-64 libsystemd-journal.so=0-64
libsystemd-login.so=0-64 libudev.so=1-64
Depends On : glib2 glibc libgcrypt lz4 xz
Optional Deps : None
Required By : gnome-settings-daemon libatasmart libgsystem libgudev openvpn procps-ng systemd util-linux
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 878.00 KiB
Packager : Evangelos Foutras <evangelos@foutrelis.com>
Build Date : Sat 20 Jun 2015 02:10:56 AEST
Install Date : Tue 23 Jun 2015 13:29:12 AEST
Install Reason : Installed as a dependency for another package
Install Script : No
Validated By : Signature

Name : systemd
Version : 221-1
Description : system and service manager
Architecture : x86_64
URL : http://www.freedesktop.org/wiki/Software/systemd
Licenses : GPL2 LGPL2.1
Groups : None
Provides : nss-myhostname systemd-tools=221 udev=221
Depends On : acl bash dbus glib2 iptables kbd kmod hwids libcap libgcrypt libsystemd libidn lz4 pam libseccomp
util-linux xz
Optional Deps : python: systemd library bindings [installed]
cryptsetup: required for encrypted block devices [installed]
libmicrohttpd: remote journald capabilities
quota-tools: kernel-level quota management
systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
polkit: allow administration as unprivileged user [installed]
Required By : accountsservice bluez-utils chromium colord cups device-mapper gnome-session lib32-systemd libgdm libgusb
libinput libpulse libusb libwacom lvm2 mesa mkinitcpio netctl pcmciautils polkit rtkit subversion
systemd-sysvcompat udisks2 upower xf86-input-evdev
Optional For : None
Conflicts With : nss-myhostname systemd-tools udev
Replaces : nss-myhostname systemd-tools udev
Installed Size : 29.75 MiB
Packager : Evangelos Foutras <evangelos@foutrelis.com>
Build Date : Sat 20 Jun 2015 02:10:07 AEST
Install Date : Tue 23 Jun 2015 13:29:13 AEST
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By : Signature
This task depends upon

Closed by  Evangelos Foutras (foutrelis)
Thursday, 30 July 2015, 05:54 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 223-1
Comment by Tomasz Przybył (FadeMind) - Wednesday, 24 June 2015, 15:37 GMT
I can't reproduce this.
Screenshot: http://i.imgur.com/rVLPrml.png
Comment by Doug Newgard (Scimmia) - Wednesday, 24 June 2015, 15:42 GMT
works fine here. What are the ownership and permissions of /var/log/journal and it's subdirs?
Comment by Doug Newgard (Scimmia) - Wednesday, 24 June 2015, 21:27 GMT
Another user with the same issue: https://bbs.archlinux.org/viewtopic.php?pid=1539403
Comment by Dave Reisner (falconindy) - Wednesday, 24 June 2015, 21:33 GMT
Please provide debug output...

SYSTEMD_LOG_LEVEL=debug journalctl ....
Comment by Daurnimator (daurnimator) - Wednesday, 24 June 2015, 21:36 GMT
$ sudo SYSTEMD_LOG_LEVEL=debug journalctl
Root directory /var/log/journal added.
Considering /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1.
Directory /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1 added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000513bb0d970dc1-3589e0c28cd1d793.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000513cb12082c27-a6aff40abe4082ad.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@000513cb12a728a9-a79b3dcc564e46ea.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000513d3de9a0579-ce9024cadb06e9db.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@000513d3df4b8e80-18b249d5a6a7ac9d.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005146b86a7b520-033acd0d55f93ea2.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@0005146b96f94e51-d4055883785bc59a.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@ce759f3259e44bd1adff1de5fa866867-0000000000000001-0005146b86a68ca5.journal added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@3565656ec3bd49379db41bdf08620dc5-0000000000000544-0005146b96f99525.journal added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000515ff52e63852-2abd2657e3ddf300.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@000515ff5560873a-c36056afc2704195.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005161cf990db7c-1e723c74bb01899c.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@0005161cfa3d561f-a7a29c4d0793bd88.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005166333061a6a-d769523b21e738f9.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@0005166333a58e0a-f880ed3d63247fe1.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000516774962804a-ada6e3902b802507.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@000516774aea0d57-82e090e63b4a74db.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005168bd6ccf72c-7b3ab5244ce67916.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@0005168bd9ccc2e1-bafb386c2015bde1.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@eda574044b924c7b97e7d7416969af00-0000000000000001-0005168bd6cc8f9a.journal added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000@00000000000000000000000000000000-0000000000000000-0000000000000000.journal added.
Failed to add file /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/user-1000.journal: Invalid argument
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000516f7f96449e9-9f23b92c24a87717.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005170b07da5f27-b5fc5d8839bdda79.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@00051768e5d55487-6c9cfae7c0d07f00.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005177cdb793928-9cde34e4e3b83870.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000517a543e39bad-d6e68ceea0fcea25.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000517ad23c024ea-a82704945f57af87.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000517d62dee4116-073ba904a058358e.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@00051846214fdbf9-e028260870648d33.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005184fac2b082f-05db73567fe75db0.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000518aaad76f940-04165f4293f17841.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000518c66434ecc3-d26e6ea38e33cf68.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@000518da0c47f5fe-deff6721f553df79.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@00051922fedb6189-117bd87cf4d3b79c.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system@0005193dd49a154b-f8f84e2f6378caa9.journal~ added.
File /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1/system.journal added.
Considering /var/log/journal/b191aef59d23475a98a74f0f78a3801c.
Error was encountered while opening journal files: Invalid argument
Directory /var/log/journal/b014ee79e9bd4934b94cb1a29441dfe1 removed.
Root directory /var/log/journal removed.
mmap cache statistics: 72 hit, 37 miss
Comment by Daurnimator (daurnimator) - Wednesday, 24 June 2015, 21:37 GMT
So, noticing the error is with a user journal, I checked and found that using `journalctl --system` works fine
Comment by Emanuel Duss (mindfuckup) - Wednesday, 24 June 2015, 21:39 GMT
emanuel@eris:~
$ SYSTEMD_LOG_LEVEL=debug journalctl
Root directory /var/log/journal added.
Considering /var/log/journal/45a16f5406e346168a94791581a8b952.
Directory /var/log/journal/45a16f5406e346168a94791581a8b952 added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@00050e0533021e71-65bf8240664ca8e0.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@00050f20269785cb-91e017783616b4bc.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@000512a998c8df40-2790375a365800d8.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@217b4ba45fe5498ea5ab4fd1d1471185-000000000001b732-00050d2450674f1e.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@00050957aae0745e-976066bc167bea9f.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@aa8363e58c57456a913370d3a9d75d7d-0000000000000001-0005170a15cece44.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@000509745c8cd2be-a8506188e84fbb38.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@5b8dc53e0c204e68a03235e7ac168458-0000000000004f5b-0005085bf23fa1f4.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@5b8dc53e0c204e68a03235e7ac168458-0000000000002bb3-00050846d6913912.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@0005067ec04cf6d2-cbb69598395f082a.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@00050f22f1ef4d60-04370576e114df9c.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@fcbf4630d2b848c8ae3001f4945d0c9f-000000000000fad2-0005060ac1a5a083.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@000514e40a861f50-b3f0ff21bde333d4.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@0005170a15e45fbf-84c66feafd03b83b.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@0005082a24c24f09-7bc14a593f38ea72.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@5b8dc53e0c204e68a03235e7ac168458-0000000000007087-0005085de588e95f.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@f17e90e9ae284d51a996546df0901a5d-0000000000000001-00050f22f1d7789c.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@aa8363e58c57456a913370d3a9d75d7d-000000000000563e-000517f5b2f6c587.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@aa8363e58c57456a913370d3a9d75d7d-0000000000007725-0005188de1217bbe.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@0005097ab5f68498-46018cf823805f12.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@52168dbc49324797a2e03f5d9ccc7717-0000000000000001-000506bde276439f.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@5b8dc53e0c204e68a03235e7ac168458-000000000000972b-00050899b907d7b3.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@000506bde2b46630-6291be9316c8ac61.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@000516879bf994fd-df8bb722f3e1efc8.journal~ added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@39e10dd058fa4ba8b8cd60eaf72afd76-0000000000000001-000512a998a86073.journal added.
Failed to add file /var/log/journal/45a16f5406e346168a94791581a8b952/user-1000.journal: Invalid argument
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@217b4ba45fe5498ea5ab4fd1d1471185-0000000000000001-0005097ab5e2a93d.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@217b4ba45fe5498ea5ab4fd1d1471185-0000000000015f3e-00050be142e6c13f.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@5b8dc53e0c204e68a03235e7ac168458-0000000000000001-0005082a24ac5e7a.journal added.
File /var/log/journal/45a16f5406e346168a94791581a8b952/system@39e10dd058fa4ba8b8cd60eaf72afd76-0000000000002858-000512e7e98e0bcc.journal added.
Error was encountered while opening journal files: Invalid argument
Directory /var/log/journal/45a16f5406e346168a94791581a8b952 removed.
Root directory /var/log/journal removed.
mmap cache statistics: 60 hit, 31 miss

It looks the error is produced in the following line:


Failed to add file /var/log/journal/45a16f5406e346168a94791581a8b952/user-1000.journal: Invalid argument
Comment by S Joseph (josephatwork) - Thursday, 25 June 2015, 06:07 GMT
Same issue here.
$journalctl
Error was encountered while opening journal files: Invalid argument



$SYSTEMD_LOG_LEVEL=debug journalctl
Root directory /run/log/journal added.
Considering /run/log/journal/5417f81b160343999f5fe0018effff19.
Directory /run/log/journal/5417f81b160343999f5fe0018effff19 added.
File /run/log/journal/5417f81b160343999f5fe0018effff19/system.journal added.
Root directory /var/log/journal added.
Considering /var/log/journal/5417f81b160343999f5fe0018effff19.
Directory /var/log/journal/5417f81b160343999f5fe0018effff19 added.
Failed to add file /var/log/journal/5417f81b160343999f5fe0018effff19/system.journal: Invalid argument
Failed to add file /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000.journal: Invalid argument
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@00050dbe7d99fa27-b0f2e196806d2296.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@00050dbe7e4ea0aa-52826012e901f88d.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@00050dbebab16fed-e737542e1014d6a5.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@00050e856af919bc-07e73b9cf363fae2.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@00050e85792ad96f-8cabdec867b9d1ce.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@c5af8d03c0ef4c6d934e657575d3c782-0000000000000001-00050e85792ac33f.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@e30fb73696ef4fc4b29f6f6e0b8bd4d0-000000000000047f-00050dbe7e4eb9cf.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@c5af8d03c0ef4c6d934e657575d3c782-000000000002bbba-00051022d27928dc.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@e30fb73696ef4fc4b29f6f6e0b8bd4d0-000000000002bbb8-00051022cd4e91e4.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@000513a029b9f974-65cd08a44b9c1797.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@000513a02b420b0c-8e5b6a6e1a6ef0bc.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@000513fb10839fbf-c3678c92968b26a3.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@00051422a7bb1046-55fefad64c878ee5.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@0005142bd0046968-4eef77e7c510b3a1.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@0005142bd094ebd2-10bd6205a9617a5d.journal~ added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000000001-0005142bd0043aaf.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-00000000000004c6-0005142bd095014e.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000013ef7-0005147363c2d85b.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-00000000000131bc-000514736209df97.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000023d82-00051473840065a8.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-000000000002378d-0005147382dced25.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000033a6d-00051473a60a49cf.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-00000000000337e0-00051473a4a2324f.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000035637-00051473ad3d1a37.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-0000000000033c43-00051473a637c4c1.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000034187-00051473a7d42cb4.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-000000000003415e-00051473a7b6fc3a.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-00000000000348a0-00051473a99f10f0.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-00000000000347c3-00051473a9132117.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000034f84-00051473ab6b59a9.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-0000000000034e3e-00051473aa8cbcb5.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-00000000000354ae-00051473ac3286ad.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000035c9c-00051473af07148b.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-0000000000035b13-00051473ade149d3.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-00000000000362c8-00051473b0d4068a.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-0000000000036190-00051473afd0d85a.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/system@62f70035b4ae4c43bc7bf4f47d8666c8-0000000000036888-00051473b2e96c49.journal added.
File /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000@bc886722fe9d439db36fd05df0d4e9e2-0000000000036803-00051473b1e1a4e0.journal added.
Error was encountered while opening journal files: Invalid argument
Directory /var/log/journal/5417f81b160343999f5fe0018effff19 removed.
Root directory /run/log/journal removed.
Directory /run/log/journal/5417f81b160343999f5fe0018effff19 removed.
Root directory /var/log/journal removed.
mmap cache statistics: 78 hit, 41 miss
$
Comment by Michael Taboada (lilmike) - Thursday, 25 June 2015, 10:20 GMT
Same here.

Root directory /run/log/journal added.
Considering /run/log/journal/d05369c84ab645118b4389437fc0569a.
Directory /run/log/journal/d05369c84ab645118b4389437fc0569a added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system.journal added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system@990eb234c0754bb7988bcfa878fe2ac5-00000000000177e7-000519549acf2df5.journal added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system@990eb234c0754bb7988bcfa878fe2ac5-0000000000013a3b-000519548002c712.journal added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system@990eb234c0754bb7988bcfa878fe2ac5-000000000000fb45-000519546a8e9090.journal added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system@990eb234c0754bb7988bcfa878fe2ac5-000000000000bc24-0005195455120e09.journal added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system@990eb234c0754bb7988bcfa878fe2ac5-0000000000007d22-0005195441e94cf2.journal added.
File /run/log/journal/d05369c84ab645118b4389437fc0569a/system@990eb234c0754bb7988bcfa878fe2ac5-0000000000003e06-000519542f91d1b1.journal added.
00000000001-000519541ede8f89.journal added.
Root directory /var/log/journal added.
Considering /var/log/journal/d05369c84ab645118b4389437fc0569a.
Directory /var/log/journal/d05369c84ab645118b4389437fc0569a added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fd4864-000518c285df6765.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fe22ad-000518c3d972974b.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fe003f-000518c393685179.journal added.
Failed to add file /var/log/journal/d05369c84ab645118b4389437fc0569a/system.journal: Invalid argument
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fd6d6a-000518c2b9103b2e.journal added.
Failed to add file /var/log/journal/d05369c84ab645118b4389437fc0569a/user-1000.journal: Invalid argument
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fe44e1-000518c422ae8604.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fdb6ad-000518c3256dd969.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fd19ae-000518c241b1c9fb.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fddbb3-000518c358cd44f2.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fceadc-000518c1fa638483.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fe671e-000518c46db83991.journal added.
File /var/log/journal/d05369c84ab645118b4389437fc0569a/system@1ec037b60cb2411fb050394c74e02df0-0000000000fd9270-000518c2ed9ce1ca.journal added.
Error was encountered while opening journal files: Invalid argument
Directory /run/log/journal/d05369c84ab645118b4389437fc0569a removed.
Root directory /run/log/journal removed.
Root directory /var/log/journal removed.
Directory /var/log/journal/d05369c84ab645118b4389437fc0569a removed.
mmap cache statistics: 38 hit, 21 miss
Comment by Edvard Holst (Zygote) - Monday, 29 June 2015, 06:58 GMT
I have the same issue. journalctl just returns "Error was encountered while opening journal files: Invalid argument"
Comment by stoffl (stoffl) - Monday, 29 June 2015, 08:03 GMT
Same errors here, but only the system.journal is affected not the user-1000.journal in my case.
But with the old 219 version both journal files are accepted without errors.

$ sudo journalctl --verify --file /var/log/journal/*/system.journal
Failed to open /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system.journal: Invalid argument
Failed to open files: Invalid argument
$ sudo journalctl --verify --file /var/log/journal/*/user-1000.journal
PASS: /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000.journal

The debug output confirms this system.journal with invalid argument:

$ SYSTEMD_LOG_LEVEL=debug journalctl
Root directory /run/log/journal added.
Considering /run/log/journal/22207cfa7a3d42cbb48e15413f10137b.
Directory /run/log/journal/22207cfa7a3d42cbb48e15413f10137b added.
File /run/log/journal/22207cfa7a3d42cbb48e15413f10137b/system.journal added.
Root directory /var/log/journal added.
Considering /var/log/journal/22207cfa7a3d42cbb48e15413f10137b.
Directory /var/log/journal/22207cfa7a3d42cbb48e15413f10137b added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000133fd7-000513823c5f5c8a.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@907a0b46df624ae6a0a26e6c05dfae0f-0000000000000001-00050e8fe588ee02.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-00000000001357a8-000513826dfca51e.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@162234b90c79446b81847eef6a378c27-0000000000000001-00050a50f2d222db.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-00000000001343cb-00051382448173e4.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000135ba7-00051382766685fe.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000136397-0005138286ec73b9.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000134fb7-000513825d7033d3.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-000000000013776a-00051382b06d1ef1.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000135f95-000513827e8884fe.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000138cbd-00051382e93af095.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000137f6e-00051382c3c2d203.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@907a0b46df624ae6a0a26e6c05dfae0f-00000000000c1aa0-00050fa957605fc0.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000@ad58fd4ada754ed7a8e4f34f48f7d404-000000000000069d-00050dd20e6b8ee3.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000137b7c-00051382b8dbf680.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000137371-00051382a7fcde5e.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000@ad58fd4ada754ed7a8e4f34f48f7d404-000000000008ca67-00050f1ac844d6f3.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@f73d577bcd5e42c4ae9d2aee7c940bd8-0000000000000001-00050c18685b675f.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000136f8d-000513829fe4a5ff.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-00000000001347cf-000513824ceaa2e8.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000134bd5-00051382555da4cc.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-000000000013836d-00051382d2eb369c.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-000000000012eff3-00051363533be320.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@162234b90c79446b81847eef6a378c27-0000000000056cbf-00050a83e98645d9.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000000001-000512ee0380b441.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@2b559fa0ac5a4d91b48b618fcd8b0bfe-0000000000000001-00050dd20a82ee11.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-00000000001353c7-0005138265e2b347.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@907a0b46df624ae6a0a26e6c05dfae0f-000000000008c54a-00050f19dcc3a37f.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-0000000000136b87-0005138297768761.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000@ad58fd4ada754ed7a8e4f34f48f7d404-00000000000e5de4-00051292723891ca.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-120.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000@6856d051d3a74eb388cac7f4e4a28857-0000000000056a8e-00050a83e2dc4a31.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000@ad58fd4ada754ed7a8e4f34f48f7d404-0000000000034693-00050e71eeb13577.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000@3748a0d9714947e895474cde2dfa1509-0000000000000759-00050c186cc16a78.journal added.
Failed to add file /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system.journal: Invalid argument
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@907a0b46df624ae6a0a26e6c05dfae0f-00000000000e5a10-0005129270971d89.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/user-1000.journal added.
File /var/log/journal/22207cfa7a3d42cbb48e15413f10137b/system@75438d26bd664af5bf188f5de40f0668-00000000001367a3-000513828f5329d2.journal added.
Error was encountered while opening journal files: Invalid argument
Root directory /run/log/journal removed.
Directory /run/log/journal/22207cfa7a3d42cbb48e15413f10137b removed.
Root directory /var/log/journal removed.
Directory /var/log/journal/22207cfa7a3d42cbb48e15413f10137b removed.
mmap cache statistics: 76 hit, 39 miss

I have also attached a strace in case some one is interested:
$ strace -o <ouputfile> journalctl
   strace-journalctl-systemd-221 (34.7 KiB)
Comment by Eulk11 (wrm) - Tuesday, 30 June 2015, 10:02 GMT
I am also getting same errors, executing as root:

journalctl -f
Error was encountered while opening journal files: Invalid argument

SYSTEMD_LOG_LEVEL=debug journalctl
Error was encountered while opening journal files: Invalid argument

When I try to verify journal files:

journalctl --verify --file /var/log/journal/*/system.journal
Failed to open /var/log/journal/de4d179506854cccaa77c7eab6de15b9/system.journal: Invalid argument
Failed to open files: Invalid argument


Comment by Jörg Thalheim (Mic92) - Saturday, 04 July 2015, 10:22 GMT
The error is actually generated by journald not journalctl.
It seems, that it cannot serialize the state of the new journalctl stream to /run/systemd/notify
The attached file is output of strace -p <journald-pid> -o /tmp/strace-journald

Here is the relevant part, where journalctl fails:

epoll_wait(7, {{EPOLLIN, {u32=3856675776, u64=139684783076288}}}, 92, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {2601, 986570503}) = 0
accept4(3, 0, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = 88
getsockopt(88, SOL_SOCKET, SO_PEERCRED, {pid=20144, uid=0, gid=0}, [12]) = 0
shutdown(88, SHUT_WR) = 0
epoll_ctl(7, EPOLL_CTL_ADD, 88, {EPOLLIN, {u32=3856766064, u64=139684783166576}}) = 0
epoll_wait(7, {{EPOLLIN, {u32=3856675776, u64=139684783076288}}, {EPOLLIN, {u32=3856766064, u64=139684783166576}}}, 93, -1) = 2
clock_gettime(CLOCK_BOOTTIME, {2601, 986663669}) = 0
read(88, "stop-php-fpm\nphp-fpm-stop.servic"..., 2048) = 45
fstat(88, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
stat("/run/systemd/journal", {st_mode=S_IFDIR|0755, st_size=160, ...}) = 0
mkdir("/run/systemd/journal/streams", 0755) = -1 EEXIST (File exists)
stat("/run/systemd/journal/streams", {st_mode=S_IFDIR|0755, st_size=1680, ...}) = 0
umask(077) = 022
open("/run/systemd/journal/streams/.#7:138529nFhsd0", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 89
umask(022) = 077
fcntl(89, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(89, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0ae41aa000
write(89, "# This is private data. Do not p"..., 173) = 173
rename("/run/systemd/journal/streams/.#7:138529nFhsd0", "/run/systemd/journal/streams/7:138529") = 0
socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 93
sendmsg(93, {msg_name(21)={sa_family=AF_LOCAL, sun_path="/run/systemd/notify"}, msg_iov(1)=[{"FDSTORE=1", 9}], msg_controllen=40, [{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, [88]}, {cmsg_len=139684753596968, cmsg_level=0xac9780 /* SOL_??? */, cmsg_type=32766, ...}], msg_flags=0}, MSG_NOSIGNAL) = -1 EINVAL (Invalid argument)
close(93) = 0
close(89) = 0
munmap(0x7f0ae41aa000, 4096) = 0

The failing line is:

sendmsg(93, {msg_name(21)={sa_family=AF_LOCAL, sun_path="/run/systemd/notify"}, msg_iov(1)=[{"FDSTORE=1", 9}], msg_controllen=40, [{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, [88]}, {cmsg_len=139684753596968, cmsg_level=0xac9780 /* SOL_??? */, cmsg_type=32766, ...}], msg_flags=0}, MSG_NOSIGNAL) = -1 EINVAL (Invalid argument)
   strace-journald (24.1 KiB)
Comment by Dave Reisner (falconindy) - Saturday, 04 July 2015, 18:26 GMT
Your logic seems highly suspect since journalctl never connects to journald.

There's nothing for Arch to do here. Please bisect the failure in journalctl and report the breakage upstream.
Comment by stoffl (stoffl) - Saturday, 04 July 2015, 20:22 GMT
Opened issue on github for this bug https://github.com/systemd/systemd/issues/487
Comment by Dave Reisner (falconindy) - Saturday, 04 July 2015, 20:27 GMT
Without the meaningful bit -- bisecting the failure. If upstream cannot reproduce the failure (and they likely can't), they're just going to ask you to do the same.
Comment by stoffl (stoffl) - Saturday, 04 July 2015, 21:49 GMT
Any help would be appreciated on how to bisect this particular failure...
Comment by S Joseph (josephatwork) - Sunday, 05 July 2015, 02:13 GMT
While not a solution, this worked for me:


$ mv /var/log/journal /var/log/journal_old


$ vim /etc/systemd/journald.conf
..
Storage=persistent
..

$systemctl restart systemd-journald

$journalctl --verify

PASS: /var/log/journal/5417f81b160343999f5fe0018effff19/system.journal
PASS: /var/log/journal/5417f81b160343999f5fe0018effff19/user-1000.journal


EDIT: On a reboot the permissions on the new directory was modified as below, but journalctl now works.

drwxr-sr-x 3 root systemd-journal 45 Jul 5 14:01 journal



I noticed that the directory ownership and permissions of the newly created journal dir is different to the original one, I am certain, I didn't explicitly set the permissions or acls in the original directory.

==new==
$getfacl /var/log/journal
getfacl: Removing leading '/' from absolute path names
# file: var/log/journal
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

=====original===
[root@blackrock ~]# getfacl /var/log/journal_old/
getfacl: Removing leading '/' from absolute path names
# file: var/log/journal_old/
# owner: root
# group: systemd-journal
# flags: -s-
user::rwx
group::r-x
group:adm:r-x
group:wheel:r-x
mask::r-x
other::r-x
default:user::rwx
default:group::r-x
default:group:adm:r-x
default:group:wheel:r-x
default:mask::r-x
default:other::r-x

Comment by stoffl (stoffl) - Sunday, 05 July 2015, 07:13 GMT
I can confirm removing and letting systemd recreate the journal files removes the problem.

Either way be removing them individually or changing the journald config and restarting the daemon results in journalctl being happy again.

So it really seams like these journal file got/were corrupted either during upgrade, or have been ever since but the old version 219 was not verifying correctly.

The permissions on the original directory as reported by joseph are the same on my affected machine and on my unaffected machines.

I have several machines where I can not reproduce this error, where the upgrade from 219 to 221 went smoothly. There is only one machine where this issue came up.

While removing the journals fixes the issue, I am hesitant to just do the nuke and pave thingy and forget about it.

But hopefully someone more knowledgeable upstream could shed some light onto this issue
Comment by Florian Pritz (bluewind) - Sunday, 05 July 2015, 17:13 GMT
Upstream report (with bisect): https://bugs.freedesktop.org/show_bug.cgi?id=90815
Comment by Daurnimator (daurnimator) - Sunday, 05 July 2015, 23:34 GMT
> Upstream report (with bisect): https://bugs.freedesktop.org/show_bug.cgi?id=90815

Systemd bugs are now tracked in github: https://github.com/systemd/systemd/issues
Comment by stoffl (stoffl) - Monday, 06 July 2015, 13:34 GMT
Yes take a look here: https://github.com/systemd/systemd/issues/487#

Issue is open and I am adding more information to it as I make progress.

But next time can you please give more specific directions to newbies, it took me some time to understand, that you guys were talking about "git bisect".

Loading...