Arch Linux

[1] suggest that the getenv [2] call failing might be a symptom of earlier corruption.
Edit:
Does rebuilding with the address sanitizer '-fsanitize=address' detect any issues?

[1]: https://bugs.launchpad.net/ubuntu/+bug/1979118/comments/3
[2]: https://github.com/libexpat/libexpat/blob/R_2_5_0/expat/lib/xmlparse.c#L8389

I rebuilt the gnome-shell package with -Db_sanitize=address,undefined and after several reboots I got "just" this:

[ 4.162456] archlinux dbus-daemon[439]: [session uid=120 pid=439] Successfully activated service 'org.freedesktop.systemd1'
[ 4.295723] archlinux /usr/lib/gdm-wayland-session[442]: dbus-daemon[442]: [session uid=120 pid=442] Activating service name='org.freedesktop.systemd1' requested by ':1.2' (uid=120 pid=443 comm="/usr/lib/gnome-session-binary --autostart /usr/sha")
[ 4.299184] archlinux /usr/lib/gdm-wayland-session[442]: dbus-daemon[442]: [session uid=120 pid=442] Activated service 'org.freedesktop.systemd1' failed: Process org.freedesktop.systemd1 exited with status 1
[ 4.299562] archlinux gnome-session[443]: gnome-session-binary[443]: WARNING: Could not check if unit gnome-session-wayland@gnome-login.target is active: Error calling StartServiceByName for org.freedesktop.systemd1: Process org.freedesktop.systemd1 exited with status 1
[ 4.299989] archlinux gnome-session-binary[443]: WARNING: Could not check if unit gnome-session-wayland@gnome-login.target is active: Error calling StartServiceByName for org.freedesktop.systemd1: Process org.freedesktop.systemd1 exited with status 1
[ 4.312398] archlinux gnome-session[443]: gnome-session-binary[443]: WARNING: Desktop file /usr/share/gdm/greeter/autostart/orca-autostart.desktop for application orca-autostart.desktop could not be parsed or references a missing TryExec binary
[ 4.312555] archlinux gnome-session-binary[443]: WARNING: Desktop file /usr/share/gdm/greeter/autostart/orca-autostart.desktop for application orca-autostart.desktop could not be parsed or references a missing TryExec binary
[ 4.562029] archlinux gnome-shell[455]: Running GNOME Shell (using mutter 45.0) as a Wayland display server
[ 4.621103] archlinux gnome-shell[455]: Failed to make thread 'KMS thread' realtime scheduled: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.freedesktop.RealtimeKit1" does not exist
[ 4.625626] archlinux org.gnome.Shell.desktop[455]: pci id for fd 12: 1013:00b8, driver (null)
[ 4.625962] archlinux org.gnome.Shell.desktop[455]: MESA-LOADER: failed to open cirrus: /usr/lib/dri/cirrus_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
[ 4.906052] archlinux org.gnome.Shell.desktop[455]: pci id for fd 13: 1013:00b8, driver (null)
[ 4.906052] archlinux org.gnome.Shell.desktop[455]: kmsro: driver missing
[ 4.961439] archlinux gnome-shell[455]: Added device '/dev/dri/card0' (cirrus) using atomic mode setting.
[ 4.962980] archlinux gnome-shell[455]: Failed to initialize accelerated iGPU/dGPU framebuffer sharing: Not hardware accelerated
[ 4.963068] archlinux gnome-shell[455]: Created gbm renderer for '/dev/dri/card0'
[ 4.963184] archlinux gnome-shell[455]: Boot VGA GPU /dev/dri/card0 selected as primary
[ 5.183502] archlinux gnome-shell[455]: Disabling DMA buffer screen sharing (not hardware accelerated)
[ 5.193503] archlinux /usr/lib/gdm-wayland-session[442]: dbus-daemon[442]: [session uid=120 pid=442] Activating service name='org.a11y.Bus' requested by ':1.4' (uid=120 pid=455 comm="/usr/bin/gnome-shell")
[ 5.202405] archlinux /usr/lib/gdm-wayland-session[442]: dbus-daemon[442]: [session uid=120 pid=442] Successfully activated service 'org.a11y.Bus'
[ 5.214010] archlinux kernel: gnome-shell[455]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set
[ 5.216770] archlinux gnome-shell[455]: Using public X11 display :1024, (using :1025 for managed services)
[ 5.216932] archlinux gnome-shell[455]: Using Wayland display name 'wayland-0'
[ 5.219519] archlinux org.gnome.Shell.desktop[455]: AddressSanitizer:DEADLYSIGNAL
[ 5.219519] archlinux org.gnome.Shell.desktop[455]: =================================================================
[ 5.219710] archlinux org.gnome.Shell.desktop[455]: ==455==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000007c (pc 0x7fd6bc05f93d bp 0x612000027940 sp 0x7fd6864e8820 T28)
[ 5.219774] archlinux org.gnome.Shell.desktop[455]: ==455==The signal is caused by a READ memory access.
[ 5.219833] archlinux org.gnome.Shell.desktop[455]: ==455==Hint: address points to the zero page.
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #0 0x7fd6bc05f93d in getenv (/usr/lib/libc.so.6+0x4193d) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #1 0x7fd6b73945df (/usr/lib/libexpat.so.1+0x1f5df) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #2 0x7fd6b737add9 (/usr/lib/libexpat.so.1+0x5dd9) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #3 0x7fd6b73845af (/usr/lib/libexpat.so.1+0xf5af) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #4 0x7fd6b83eb842 (/usr/lib/libfontconfig.so.1+0x2d842) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #5 0x7fd6b83ec296 (/usr/lib/libfontconfig.so.1+0x2e296) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #6 0x7fd6b83ec479 (/usr/lib/libfontconfig.so.1+0x2e479) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #7 0x7fd6b83ef475 (/usr/lib/libfontconfig.so.1+0x31475) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #8 0x7fd6b738263e (/usr/lib/libexpat.so.1+0xd63e) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #9 0x7fd6b737fc13 (/usr/lib/libexpat.so.1+0xac13) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #10 0x7fd6b73817ec (/usr/lib/libexpat.so.1+0xc7ec) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #11 0x7fd6b73858e9 in XML_ParseBuffer (/usr/lib/libexpat.so.1+0x108e9) (BuildId: a98bfab551dfa3df6889c33d5fd2ccfa6d505608)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #12 0x7fd6b83eb944 (/usr/lib/libfontconfig.so.1+0x2d944) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #13 0x7fd6b83ec296 (/usr/lib/libfontconfig.so.1+0x2e296) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #14 0x7fd6b83d4190 (/usr/lib/libfontconfig.so.1+0x16190) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #15 0x7fd6b83cf15c (/usr/lib/libfontconfig.so.1+0x1115c) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #16 0x7fd6b83d448c in FcInit (/usr/lib/libfontconfig.so.1+0x1648c) (BuildId: 2f7305d108d26daad426b3855fe9225ddfef356b)
[ 5.352442] archlinux org.gnome.Shell.desktop[455]: #17 0x7fd6b8416411 (/usr/lib/libpangoft2-1.0.so.0+0x9411) (BuildId: c4942d7c23fe50db42934220b31981cfbf464e48)
[ 5.353672] archlinux org.gnome.Shell.desktop[455]: #18 0x7fd6bcf3f9a4 (/usr/lib/libglib-2.0.so.0+0x8b9a4) (BuildId: 1916d89bc0f8f0932e584f87427c2fedfc8a293b)
[ 5.353672] archlinux org.gnome.Shell.desktop[455]: #19 0x7fd6bc0aa9ea (/usr/lib/libc.so.6+0x8c9ea) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
[ 5.353672] archlinux org.gnome.Shell.desktop[455]: #20 0x7fd6bc12e7cb (/usr/lib/libc.so.6+0x1107cb) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)
[ 5.353672] archlinux org.gnome.Shell.desktop[455]: AddressSanitizer can not provide additional info.
[ 5.353672] archlinux org.gnome.Shell.desktop[455]: SUMMARY: AddressSanitizer: SEGV (/usr/lib/libc.so.6+0x4193d) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658) in getenv
[ 5.353672] archlinux org.gnome.Shell.desktop[455]: Thread T28 created by T0 here:
[ 5.394804] archlinux /usr/lib/gdm-wayland-session[494]: dbus-daemon[494]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=120 pid=455 comm="/usr/bin/gnome-shell")
[ 5.399525] archlinux dbus-daemon[370]: [system] Activating via systemd: service name='org.freedesktop.ColorManager' unit='colord.service' requested by ':1.16' (uid=120 pid=455 comm="/usr/bin/gnome-shell")
[ 5.404267] archlinux org.gnome.Shell.desktop[496]: Failed to initialize glamor, falling back to sw
[ 5.430379] archlinux org.gnome.Shell.desktop[455]: #0 0x7fd6bd04a497 in __interceptor_pthread_create /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_interceptors.cpp:208
[ 5.430379] archlinux org.gnome.Shell.desktop[455]: #1 0x7fd6bcf40f53 (/usr/lib/libglib-2.0.so.0+0x8cf53) (BuildId: 1916d89bc0f8f0932e584f87427c2fedfc8a293b)
[ 5.430379] archlinux org.gnome.Shell.desktop[455]: ==455==ABORTING
[ 5.430518] archlinux /usr/lib/gdm-wayland-session[498]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
[ 5.430758] archlinux /usr/lib/gdm-wayland-session[494]: dbus-daemon[494]: Successfully activated service 'org.a11y.atspi.Registry'
[ 5.430825] archlinux systemd[1]: Starting colord.service...
[ 5.431582] archlinux gnome-session[443]: gnome-session-binary[443]: WARNING: App 'org.gnome.Shell.desktop' exited with code 1

But I'm currently debugging a different issue that also involves rebooting a lot, so I'll keep an eye on the logs if something else (and possibly more helpful) pops up.

FWIW, it can be relatively reliably reproduced using the official Vagrant box[0] with the following crude reproducer:

$ mkdir arch-debug
$ cd arch-debug
$ cat >Vagrantfile <<EOF
Vagrant.configure("2") do |config|
config.vm.box = "archlinux/archlinux"
config.vm.synced_folder ".", "/vagrant", disabled: true
end
EOF
$ vagrant up --provider=libvirt
$ vagrant ssh -c 'sudo bash -c "systemctl disable systemd-time-wait-sync; pacman --noconfirm -Sy gdm; systemctl set-default graphical.target; systemctl enable gdm"'
$ vagrant reload
$ while ! vagrant ssh -c 'systemctl --wait is-system-running; sleep 10; sudo journalctl -b --grep "[k]illed by signal"'; do vagrant reload; done
...
default: Warning: Connection refused. Retrying...
==> default: Machine booted and ready!
==> default: Creating shared folders metadata...
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
running
Oct 23 17:16:49 archlinux gnome-session-binary[356]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 11
Oct 23 17:16:49 archlinux gnome-session[356]: gnome-session-binary[356]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 11

"Relatively" meaning that it can take a couple dozen tries before gnome-shell crashes.

[0] https://gitlab.archlinux.org/archlinux/arch-boxes

It sounds like an upstream issue...assigning to GNOME PM's for a look.

This sounds like another thread is using setenv/unsetenv/putenv at the same time as your crashing thread is using getenv. Manipulating the environment is not thread-safe.

> FWIW, it can be relatively reliably reproduced using the official Vagrant box[0] with the following crude reproducer:

Can repro using your instructions. Backtrace with debug symbols attached. Definitely seems fontconfig related.. but it still doesn't seem like an Arch packaging issue...

If you can reproduce this, try to backtrace all threads (`t apply all bt`).

Working in a VM memory constrained environment...so it was an effort :)

Thanks for the effort!

Hm, Mutter is calling setenv at various points during startup so I wouldn't be surprised if this is racing with Pango's threaded initialization of FontConfig.

This is an bug in Mutter and/or GNOME Shell. They shouldn't be using setenv after threads are spawned. Could you please file this upstream?

Could it be the same issue as [1]?

[1]: https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/6974

> Could it be the same issue

Thanks @loqs. It does indeed look very similar. I've linked this report there. After the analysis from @heftig, there is not much doubt about this being an upstream issue.