FS#79760 - [libnvme][libblockdev] udisks2 fails on startup since update 2.9.4-4 -> 2.10.1-1

Attached to Project: Arch Linux
Opened by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 00:44 GMT
Last edited by David Runge (dvzrv) - Friday, 13 October 2023, 15:59 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To David Runge (dvzrv)
Morten Linderud (Foxboron)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 11
Private No

Details

udisksd core dumps and udisks2.service fails at startup every time since the following updates:

2023-09-21 07:23:34 libblockdev 2.28-4 -> 3.0.3-4
2023-09-21 07:23:34 udisks2 2.9.4-4 -> 2.10.1-1

Downgrading the above 2 packages fixes the issues. Both need to be downgraded.
This task depends upon

Closed by  David Runge (dvzrv)
Friday, 13 October 2023, 15:59 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed with libblockdev 3.0.4-1 and libnvme 1.6-2
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 00:45 GMT Comment by Toolybird (Toolybird) - Saturday, 23 September 2023, 00:52 GMT
Related  FS#79726 

> core dumps

Please supply a backtrace containing debugging info [1]. It's usually as simple as this (with gdb installed):

$ coredumpctl gdb (then answer y when it asks "Enable debuginfod for this session?")
(gdb) set logging enabled
(gdb) bt (or bt full)

Then post gdb.txt

[1] https://wiki.archlinux.org/title/Debugging/Getting_traces
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 00:55 GMT
I did that but it looks like I first need to reinstall the latest package versions. Is that required?
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 01:06 GMT
I reinstalled the latest 2 libblockdev and udisks2 then rebooted and got the following bt. Doesn't seem that useful though.

lt:~ sudo coredumpctl gdb
PID: 1367 (udisksd)
UID: 0 (root)
GID: 0 (root)
Signal: 4 (ILL)
Timestamp: Sat 2023-09-23 11:02:39 AEST (36s ago)
Command Line: /usr/lib/udisks2/udisksd
Executable: /usr/lib/udisks2/udisksd
Control Group: /system.slice/udisks2.service
Unit: udisks2.service
Slice: system.slice
Boot ID: 5dfc2672582c482a999b1158af6009b1
Machine ID: 9eabba1a60b9408c970e3e88cfc74c46
Hostname: lt
Storage: /var/lib/systemd/coredump/core.udisksd.0.5dfc2672582c482a999b1158af6009b1.1367.1695430959000000.zst (present)
Size on Disk: 381.6K
Message: Process 1367 (udisksd) of user 0 dumped core.

Stack trace of thread 1367:
#0 0x00005632f596b480 n/a (udisksd + 0x1c480)
#1 0xfffffffffffffe48 n/a (n/a + 0x0)
ELF object binary architecture: AMD x86-64

GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/udisks2/udisksd...
(No debugging symbols found in /usr/lib/udisks2/udisksd)
[New LWP 1367]
[New LWP 1373]
[New LWP 1374]
[New LWP 1401]
[New LWP 1376]
[New LWP 1377]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/lib/udisks2/udisksd'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x00005632f596b480 in ?? ()
[Current thread is 1 (Thread 0x7ff3d384e880 (LWP 1367))]
warning: File "/home/mark/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /home/mark/.gdbinit
line to your configuration file "/root/.config/gdb/gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
(gdb) set debuginfod enabled on
(gdb) set logging enabled
Copying output to gdb.txt.
Copying debug output to gdb.txt.
(gdb) bt
#0 0x00005632f596b480 in ()
#1 0xfffffffffffffe48 in ()
#2 0x0000000000000000 in ()
(gdb) bt full
#0 0x00005632f596b480 in ()
#1 0xfffffffffffffe48 in ()
#2 0x0000000000000000 in ()
(gdb)
Comment by Toolybird (Toolybird) - Saturday, 23 September 2023, 01:13 GMT
> Doesn't seem that useful though

It appears you are missing DEBUGINFOD_URLS in your environment [1]. This can happen if you have to use sudo. Maybe try sudo -E <...>?

[1] https://wiki.archlinux.org/title/Debugging/Getting_traces#Debuginfod
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 01:17 GMT
@Toolybird, do I need to put the latest broken packages back and reboot before I do this?
Comment by Toolybird (Toolybird) - Saturday, 23 September 2023, 01:22 GMT
Not 100% certain, but I'd guess "yes" for the pkgs, "no" for the reboot.
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 01:22 GMT
OK, I put the broken packages back, rebooted, and got:

lt:~ sudo -E coredumpctl gdb
PID: 1020 (udisksd)
UID: 0 (root)
GID: 0 (root)
Signal: 6 (ABRT)
Timestamp: Sat 2023-09-23 11:19:48 AEST (1min 14s ago)
Command Line: /usr/lib/udisks2/udisksd
Executable: /usr/lib/udisks2/udisksd
Control Group: /system.slice/udisks2.service
Unit: udisks2.service
Slice: system.slice
Boot ID: e96bbed587a441129612804ab0e5e5ca
Machine ID: 9eabba1a60b9408c970e3e88cfc74c46
Hostname: lt
Storage: /var/lib/systemd/coredump/core.udisksd.0.e96bbed587a441129612804ab0e5e5ca.1020.1695431988000000.zst (present)
Size on Disk: 382.6K
Message: Process 1020 (udisksd) of user 0 dumped core.

Stack trace of thread 1020:
#0 0x00007f11d2e8e83c n/a (libc.so.6 + 0x8e83c)
#1 0x00007f11d2e3e668 raise (libc.so.6 + 0x3e668)
#2 0x00007f11d2e264b8 abort (libc.so.6 + 0x264b8)
#3 0x00007f11d2e27390 n/a (libc.so.6 + 0x27390)
#4 0x00007f11d2e987b7 n/a (libc.so.6 + 0x987b7)
#5 0x00007f11d2e9babc n/a (libc.so.6 + 0x9babc)
#6 0x00007f11d2e9dd08 __libc_calloc (libc.so.6 + 0x9dd08)
#7 0x00007f11d317e26b g_malloc0 (libglib-2.0.so.0 + 0x6426b)
#8 0x00007f11c3f08f99 bd_nvme_get_namespace_info (libbd_nvme.so.3 + 0x3f99)
#9 0x000055c3c61a3c78 udisks_linux_device_reprobe_sync (udisksd + 0x56c78)
#10 0x000055c3c61a3fe6 udisks_linux_device_new_sync (udisksd + 0x56fe6)
#11 0x000055c3c616e229 n/a (udisksd + 0x21229)
#12 0x000055c3c6170750 n/a (udisksd + 0x23750)
#13 0x000055c3c616c678 n/a (udisksd + 0x1f678)
#14 0x00007f11d328ae63 n/a (libgobject-2.0.so.0 + 0x24e63)
#15 0x00007f11d328cf0b g_object_new_valist (libgobject-2.0.so.0 + 0x26f0b)
#16 0x00007f11d328d29e g_object_new (libgobject-2.0.so.0 + 0x2729e)
#17 0x000055c3c6169de6 udisks_daemon_new (udisksd + 0x1cde6)
#18 0x000055c3c6169e47 n/a (udisksd + 0x1ce47)
#19 0x00007f11d33d87f8 n/a (libgio-2.0.so.0 + 0x1107f8)
#20 0x00007f11d3371ce4 n/a (libgio-2.0.so.0 + 0xa9ce4)
#21 0x00007f11d3375bfd n/a (libgio-2.0.so.0 + 0xadbfd)
#22 0x00007f11d33d3e63 n/a (libgio-2.0.so.0 + 0x10be63)
#23 0x00007f11d3371ce4 n/a (libgio-2.0.so.0 + 0xa9ce4)
#24 0x00007f11d3371d1d n/a (libgio-2.0.so.0 + 0xa9d1d)
#25 0x00007f11d3173f19 n/a (libglib-2.0.so.0 + 0x59f19)
#26 0x00007f11d31d22b7 n/a (libglib-2.0.so.0 + 0xb82b7)
#27 0x00007f11d3174b47 g_main_loop_run (libglib-2.0.so.0 + 0x5ab47)
#28 0x000055c3c616847d main (udisksd + 0x1b47d)
#29 0x00007f11d2e27cd0 n/a (libc.so.6 + 0x27cd0)
#30 0x00007f11d2e27d8a __libc_start_main (libc.so.6 + 0x27d8a)
#31 0x000055c3c61685c5 _start (udisksd + 0x1b5c5)

Stack trace of thread 1030:
#0 0x00007f11d2f0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007f11d31cdc23 g_cond_wait_until (libglib-2.0.so.0 + 0xb3c23)
#2 0x00007f11d313f185 n/a (libglib-2.0.so.0 + 0x25185)
#3 0x00007f11d31a84db n/a (libglib-2.0.so.0 + 0x8e4db)
#4 0x00007f11d31a59a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007f11d2e8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007f11d2f10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1031:
#0 0x00007f11d2f0359f __poll (libc.so.6 + 0x10359f)
#1 0x00007f11d31d2206 n/a (libglib-2.0.so.0 + 0xb8206)
#2 0x00007f11d3174b47 g_main_loop_run (libglib-2.0.so.0 + 0x5ab47)
#3 0x00007f11d33da0bc n/a (libgio-2.0.so.0 + 0x1120bc)
#4 0x00007f11d31a59a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007f11d2e8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007f11d2f10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1028:
#0 0x00007f11d2f0359f __poll (libc.so.6 + 0x10359f)
#1 0x00007f11d31d2206 n/a (libglib-2.0.so.0 + 0xb8206)
#2 0x00007f11d3172112 g_main_context_iteration (libglib-2.0.so.0 + 0x58112)
#3 0x00007f11d3172162 n/a (libglib-2.0.so.0 + 0x58162)
#4 0x00007f11d31a59a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007f11d2e8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007f11d2f10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1029:
#0 0x00007f11d2f0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007f11d31cd247 g_cond_wait (libglib-2.0.so.0 + 0xb3247)
#2 0x00007f11d313f1b4 n/a (libglib-2.0.so.0 + 0x251b4)
#3 0x00007f11d31a7a2e n/a (libglib-2.0.so.0 + 0x8da2e)
#4 0x00007f11d31a59a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007f11d2e8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007f11d2f10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1060:
#0 0x00007f11d2f0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007f11d31cd247 g_cond_wait (libglib-2.0.so.0 + 0xb3247)
#2 0x00007f11d313f1b4 n/a (libglib-2.0.so.0 + 0x251b4)
#3 0x00007f11d313f21c g_async_queue_pop (libglib-2.0.so.0 + 0x2521c)
#4 0x000055c3c616e08b probe_request_thread_func (udisksd + 0x2108b)
#5 0x00007f11d31a59a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#6 0x00007f11d2e8c9eb n/a (libc.so.6 + 0x8c9eb)
#7 0x00007f11d2f10dfc n/a (libc.so.6 + 0x110dfc)
ELF object binary architecture: AMD x86-64

GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/udisks2/udisksd...

This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.archlinux.org>
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
(No debugging symbols found in /usr/lib/udisks2/udisksd)
[New LWP 1020]
[New LWP 1030]
[New LWP 1031]
[New LWP 1028]
[New LWP 1029]
[New LWP 1060]
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error

warning: File "/usr/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /usr/lib/libthread_db.so.1
line to your configuration file "/home/mark/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/mark/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/usr/lib/udisks2/udisksd'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f11d2e8e83c in ?? () from /usr/lib/libc.so.6
[Current thread is 1 (LWP 1020)]
(gdb) set debuginfod enabled on
(gdb) set logging enabled
Copying output to gdb.txt.
Copying debug output to gdb.txt.
(gdb) bt full
#0 0x00007f11d2e8e83c in () at /usr/lib/libc.so.6
#1 0x00007f11d2e3e668 in raise () at /usr/lib/libc.so.6
#2 0x00007f11d2e264b8 in abort () at /usr/lib/libc.so.6
#3 0x00007f11d2e27390 in () at /usr/lib/libc.so.6
#4 0x00007f11d2e987b7 in () at /usr/lib/libc.so.6
#5 0x00007f11d2e9babc in () at /usr/lib/libc.so.6
#6 0x00007f11d2e9dd08 in calloc () at /usr/lib/libc.so.6
#7 0x00007f11d317e26b in g_malloc0 () at /usr/lib/libglib-2.0.so.0
#8 0x00007f11c3f08f99 in bd_nvme_get_namespace_info () at /usr/lib/libbd_nvme.so.3
#9 0x000055c3c61a3c78 in udisks_linux_device_reprobe_sync ()
#10 0x000055c3c61a3fe6 in udisks_linux_device_new_sync ()
#11 0x000055c3c616e229 in ()
#12 0x000055c3c6170750 in ()
#13 0x000055c3c616c678 in ()
#14 0x00007f11d328ae63 in () at /usr/lib/libgobject-2.0.so.0
#15 0x00007f11d328cf0b in g_object_new_valist () at /usr/lib/libgobject-2.0.so.0
#16 0x00007f11d328d29e in g_object_new () at /usr/lib/libgobject-2.0.so.0
#17 0x000055c3c6169de6 in udisks_daemon_new ()
#18 0x000055c3c6169e47 in ()
#19 0x00007f11d33d87f8 in () at /usr/lib/libgio-2.0.so.0
#20 0x00007f11d3371ce4 in () at /usr/lib/libgio-2.0.so.0
#21 0x00007f11d3375bfd in () at /usr/lib/libgio-2.0.so.0
#22 0x00007f11d33d3e63 in () at /usr/lib/libgio-2.0.so.0
#23 0x00007f11d3371ce4 in () at /usr/lib/libgio-2.0.so.0
#24 0x00007f11d3371d1d in () at /usr/lib/libgio-2.0.so.0
#25 0x00007f11d3173f19 in () at /usr/lib/libglib-2.0.so.0
#26 0x00007f11d31d22b7 in () at /usr/lib/libglib-2.0.so.0
#27 0x00007f11d3174b47 in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#28 0x000055c3c616847d in main ()
(gdb)
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 01:24 GMT
Not sure if wanted but gdb.txt was also created:

lt:~ cat gdb.txt
#0 0x00007f46bf40b258 in g_log_structured_array () at /usr/lib/libglib-2.0.so.0
#1 0x00007f46bf40b553 in g_log_default_handler () at /usr/lib/libglib-2.0.so.0
#2 0x00007f46bf40c405 in g_logv () at /usr/lib/libglib-2.0.so.0
#3 0x00007f46bf40c6d4 in g_log () at /usr/lib/libglib-2.0.so.0
#4 0x0000559c420f3832 in ()
#5 0x00007f46bf529b73 in () at /usr/lib/libgobject-2.0.so.0
#6 0x00007f46bf529f50 in g_signal_emit_by_name () at /usr/lib/libgobject-2.0.so.0
#7 0x00007f46bf636240 in () at /usr/lib/libgio-2.0.so.0
#8 0x00007f46bf65b198 in () at /usr/lib/libgio-2.0.so.0
#9 0x00007f46bf403f19 in () at /usr/lib/libglib-2.0.so.0
#10 0x00007f46bf4622b7 in () at /usr/lib/libglib-2.0.so.0
#11 0x00007f46bf402112 in g_main_context_iteration () at /usr/lib/libglib-2.0.so.0
#12 0x00007f46bf637af6 in g_application_run () at /usr/lib/libgio-2.0.so.0
#13 0x0000559c420ee843 in main ()
quit
#0 0x00005632f596b480 in ()
#1 0xfffffffffffffe48 in ()
#2 0x0000000000000000 in ()
#0 0x00005632f596b480 in ()
#1 0xfffffffffffffe48 in ()
#2 0x0000000000000000 in ()
quit
#0 0x00007f11d2e8e83c in () at /usr/lib/libc.so.6
#1 0x00007f11d2e3e668 in raise () at /usr/lib/libc.so.6
#2 0x00007f11d2e264b8 in abort () at /usr/lib/libc.so.6
#3 0x00007f11d2e27390 in () at /usr/lib/libc.so.6
#4 0x00007f11d2e987b7 in () at /usr/lib/libc.so.6
#5 0x00007f11d2e9babc in () at /usr/lib/libc.so.6
#6 0x00007f11d2e9dd08 in calloc () at /usr/lib/libc.so.6
#7 0x00007f11d317e26b in g_malloc0 () at /usr/lib/libglib-2.0.so.0
#8 0x00007f11c3f08f99 in bd_nvme_get_namespace_info () at /usr/lib/libbd_nvme.so.3
#9 0x000055c3c61a3c78 in udisks_linux_device_reprobe_sync ()
#10 0x000055c3c61a3fe6 in udisks_linux_device_new_sync ()
#11 0x000055c3c616e229 in ()
#12 0x000055c3c6170750 in ()
#13 0x000055c3c616c678 in ()
#14 0x00007f11d328ae63 in () at /usr/lib/libgobject-2.0.so.0
#15 0x00007f11d328cf0b in g_object_new_valist () at /usr/lib/libgobject-2.0.so.0
#16 0x00007f11d328d29e in g_object_new () at /usr/lib/libgobject-2.0.so.0
#17 0x000055c3c6169de6 in udisks_daemon_new ()
#18 0x000055c3c6169e47 in ()
#19 0x00007f11d33d87f8 in () at /usr/lib/libgio-2.0.so.0
#20 0x00007f11d3371ce4 in () at /usr/lib/libgio-2.0.so.0
#21 0x00007f11d3375bfd in () at /usr/lib/libgio-2.0.so.0
#22 0x00007f11d33d3e63 in () at /usr/lib/libgio-2.0.so.0
#23 0x00007f11d3371ce4 in () at /usr/lib/libgio-2.0.so.0
#24 0x00007f11d3371d1d in () at /usr/lib/libgio-2.0.so.0
#25 0x00007f11d3173f19 in () at /usr/lib/libglib-2.0.so.0
#26 0x00007f11d31d22b7 in () at /usr/lib/libglib-2.0.so.0
#27 0x00007f11d3174b47 in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#28 0x000055c3c616847d in main ()
quit
Comment by Toolybird (Toolybird) - Saturday, 23 September 2023, 01:31 GMT
You're still not there yet... No debugging info is present. Unless you can see source code line numbers in the trace, it's completely useless.

What does this output?

$ echo $DEBUGINFOD_URLS
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 01:44 GMT
lt:~ echo $DEBUGINFOD_URLS
https://debuginfod.archlinux.org
Comment by Toolybird (Toolybird) - Saturday, 23 September 2023, 01:53 GMT
That looks fine. But it appears something is screwed up in your gdb config because:

> Debuginfod has been disabled

Hopefully you can sort out whatever is wrong with your system because we cannot help otherwise. Maybe try the support forum? Because I give up at this stage.
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 03:32 GMT
@Toolybird, I disagree with changing this to low priority as I suspect it affects many Arch users (or will soon?). There is nothing wrong with my mostly stock system. After research/experimentation, I will quote here what I found for your future reference.

On Arch both root and users by default have DEBUGINFOD_URLS='https://debuginfod.archlinux.org' set in their environment. If I just do `sudo coredumpctl gdb` or `sudo -E coredumpctl gdb` then coredumpctl does not see DEBUGINFOD_URLS set so says "Debuginfod has been disabled". BTW, I don't understand why -E does not pass it but this is my observation. It works if you either log in as root and run `coredumpctl gdb`, or you do `sudo --preserve-env=DEBUGINFOD_URLS coredumpctl gdb`.

So here is the output with debuginfod:

lt:~ sudo --preserve-env=DEBUGINFOD_URLS coredumpctl gdb
PID: 1340 (udisksd)
UID: 0 (root)
GID: 0 (root)
Signal: 6 (ABRT)
Timestamp: Sat 2023-09-23 12:22:59 AEST (55min ago)
Command Line: /usr/lib/udisks2/udisksd
Executable: /usr/lib/udisks2/udisksd
Control Group: /system.slice/udisks2.service
Unit: udisks2.service
Slice: system.slice
Boot ID: 2b1057e712a842f0ba4c93b2e1a9ac63
Machine ID: 9eabba1a60b9408c970e3e88cfc74c46
Hostname: lt
Storage: /var/lib/systemd/coredump/core.udisksd.0.2b1057e712a842f0ba4c93b2e1a9ac63.1340.1695435779000000.zst (present)
Size on Disk: 424.5K
Message: Process 1340 (udisksd) of user 0 dumped core.

Stack trace of thread 1340:
#0 0x00007fae3fa8e83c n/a (libc.so.6 + 0x8e83c)
#1 0x00007fae3fa3e668 raise (libc.so.6 + 0x3e668)
#2 0x00007fae3fa264b8 abort (libc.so.6 + 0x264b8)
#3 0x00007fae3fa27390 n/a (libc.so.6 + 0x27390)
#4 0x00007fae3fb1f17b __fortify_fail (libc.so.6 + 0x11f17b)
#5 0x00007fae3fb20486 __stack_chk_fail (libc.so.6 + 0x120486)
#6 0x00007fae3fb0ca19 ioctl (libc.so.6 + 0x10ca19)

Stack trace of thread 1352:
#0 0x00007fae3fb0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007fae3fdcf247 g_cond_wait (libglib-2.0.so.0 + 0xb3247)
#2 0x00007fae3fd411b4 n/a (libglib-2.0.so.0 + 0x251b4)
#3 0x00007fae3fda9a2e n/a (libglib-2.0.so.0 + 0x8da2e)
#4 0x00007fae3fda79a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007fae3fa8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007fae3fb10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1353:
#0 0x00007fae3fb0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007fae3fdcfc23 g_cond_wait_until (libglib-2.0.so.0 + 0xb3c23)
#2 0x00007fae3fd41185 n/a (libglib-2.0.so.0 + 0x25185)
#3 0x00007fae3fdaa4db n/a (libglib-2.0.so.0 + 0x8e4db)
#4 0x00007fae3fda79a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007fae3fa8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007fae3fb10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1350:
#0 0x00007fae3fb0359f __poll (libc.so.6 + 0x10359f)
#1 0x00007fae3fdd4206 n/a (libglib-2.0.so.0 + 0xb8206)
#2 0x00007fae3fd74112 g_main_context_iteration (libglib-2.0.so.0 + 0x58112)
#3 0x00007fae3fd74162 n/a (libglib-2.0.so.0 + 0x58162)
#4 0x00007fae3fda79a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007fae3fa8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007fae3fb10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1354:
#0 0x00007fae3fb0359f __poll (libc.so.6 + 0x10359f)
#1 0x00007fae3fdd4206 n/a (libglib-2.0.so.0 + 0xb8206)
#2 0x00007fae3fd76b47 g_main_loop_run (libglib-2.0.so.0 + 0x5ab47)
#3 0x00007fae3ffdc0bc n/a (libgio-2.0.so.0 + 0x1120bc)
#4 0x00007fae3fda79a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007fae3fa8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007fae3fb10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 1410:
#0 0x00007fae3fb0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007fae3fdcf247 g_cond_wait (libglib-2.0.so.0 + 0xb3247)
#2 0x00007fae3fd411b4 n/a (libglib-2.0.so.0 + 0x251b4)
#3 0x00007fae3fd4121c g_async_queue_pop (libglib-2.0.so.0 + 0x2521c)
#4 0x000055fd4ad7408b probe_request_thread_func (udisksd + 0x2108b)
#5 0x00007fae3fda79a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#6 0x00007fae3fa8c9eb n/a (libc.so.6 + 0x8c9eb)
#7 0x00007fae3fb10dfc n/a (libc.so.6 + 0x110dfc)
ELF object binary architecture: AMD x86-64

GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/udisks2/udisksd...

This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Reading symbols from /root/.cache/debuginfod_client/51779c7b83d7877d37eaa08cc40664ec26683ad2/debuginfo...
[New LWP 1340]
[New LWP 1352]
[New LWP 1353]
[New LWP 1350]
[New LWP 1354]
[New LWP 1410]

warning: Corrupted shared library list: 0x7fae3fc4d000 != 0x0
Downloading separate debug info for /lib64/ld-linux-x86-64.so.2
Downloading separate debug info for /usr/lib/libcap.so.2
Downloading separate debug info for /usr/lib/libgcrypt.so.20
Downloading separate debug info for /usr/lib/liblz4.so.1
Downloading separate debug info for /usr/lib/liblzma.so.5
Downloading separate debug info for /usr/lib/libzstd.so.1
Downloading separate debug info for /usr/lib/libz.so.1
Downloading separate debug info for /usr/lib/libffi.so.8
Downloading separate debug info for /usr/lib/libpcre2-8.so.0
Downloading separate debug info for /usr/lib/libcrypto.so.3
--Type <RET> for more, q to quit, c to continue without paging--c
Downloading separate debug info for /usr/lib/libgpg-error.so.0
Downloading separate debug info for system-supplied DSO at 0x7fff1d506000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/lib/udisks2/udisksd'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
Downloading source file /usr/src/debug/glibc/glibc/nptl/pthread_kill.c
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
[Current thread is 1 (Thread 0x7fae3f712880 (LWP 1340))]
warning: File "/home/mark/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /home/mark/.gdbinit
line to your configuration file "/root/.config/gdb/gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
(gdb) set logging enabled
Copying output to gdb.txt.
Copying debug output to gdb.txt.
(gdb) bt full
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = <optimized out>
old_mask = {__val = {0}}
ret = <optimized out>
#1 0x00007fae3fa8e8a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007fae3fa3e668 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#3 0x00007fae3fa264b8 in __GI_abort () at abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
#4 0x00007fae3fa27390 in __libc_message (fmt=fmt@entry=0x7fae3fba32ef "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7fff1d44d2d0, reg_save_area = 0x7fff1d44d260}}
fd = 2
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
#5 0x00007fae3fb1f17b in __GI___fortify_fail (msg=msg@entry=0x7fae3fba3307 "stack smashing detected") at fortify_fail.c:24
#6 0x00007fae3fb20486 in __stack_chk_fail () at stack_chk_fail.c:24
#7 0x00007fae3fb0ca19 in __GI___ioctl (fd=<optimized out>, request=<optimized out>) at ../sysdeps/unix/sysv/linux/ioctl.c:43
args = {{gp_offset = 0, fp_offset = 0, overflow_arg_area = 0x0, reg_save_area = 0x0}}
r = <optimized out>
#8 0x0000000000000000 in ()
(gdb)

Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 03:44 GMT
To follow up on why `sudo -E` does not work here, apparently /etc/profile.d/* are only sourced for login shells so you have to do `sudo -i coredumpctl gdb`.
Comment by loqs (loqs) - Saturday, 23 September 2023, 11:18 GMT Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 23:16 GMT
@loqs, sorry but I just noticed the following "stack smashing detected" errors are what is causing this core dump. That is a gcc added protection against buffer overflows. Here is the log from my last boot with those new problematic packages:

lt:~ journalctl -b-2 -u udisks2
Sep 23 13:58:14 lt systemd[1]: Starting Disk Manager...
Sep 23 13:58:14 lt udisksd[1005]: udisks daemon version 2.10.1 starting
Sep 23 13:58:14 lt udisksd[1005]: *** stack smashing detected ***: terminated
Sep 23 13:58:14 lt systemd[1]: udisks2.service: Main process exited, code=dumped, status=6/ABRT
Sep 23 13:58:14 lt systemd[1]: udisks2.service: Failed with result 'core-dump'.
Sep 23 13:58:14 lt systemd[1]: Failed to start Disk Manager.
Sep 23 13:58:39 lt systemd[1]: Starting Disk Manager...
Sep 23 13:58:39 lt udisksd[1360]: udisks daemon version 2.10.1 starting
Sep 23 13:58:39 lt udisksd[1360]: *** stack smashing detected ***: terminated
Sep 23 13:58:40 lt systemd[1]: udisks2.service: Main process exited, code=dumped, status=6/ABRT
Sep 23 13:58:40 lt systemd[1]: udisks2.service: Failed with result 'core-dump'.
Sep 23 13:58:40 lt systemd[1]: Failed to start Disk Manager.

Do you want me to put the `coredumpctl dump` file somewhere for you to look at?
Comment by Mark Blakeney (bulletmark) - Saturday, 23 September 2023, 23:25 GMT
I tested and found I can `sudo coredumpctl dump -o udisksd.dump` then you can `sudo -i gdb /usr/lib/udisks2/udisksd udisksd.dump`.

I put it at https://www.dropbox.com/scl/fi/i7gy93j2qbs8i0znik43t/udisksd.dump.gz?rlkey=kfmyqq5hd6x6hnclmxq2t61jz.
Comment by Mark Blakeney (bulletmark) - Sunday, 24 September 2023, 07:40 GMT
Seth has been corresponding with me about this issue in the forum discussion at https://bbs.archlinux.org/viewtopic.php?pid=2122750.

I listed all the core dumps I have had for this issue:

lt:~ sudo coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE
Thu 2023-09-21 07:27:05 AEST 859 0 0 SIGABRT missing /usr/lib/udisks2/udisksd -
Thu 2023-09-21 07:27:30 AEST 1244 0 0 SIGABRT missing /usr/lib/udisks2/udisksd -
Fri 2023-09-22 07:26:59 AEST 859 0 0 SIGSEGV present /usr/lib/udisks2/udisksd 447.2K
Fri 2023-09-22 07:27:24 AEST 1250 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.4K
Fri 2023-09-22 07:28:50 AEST 859 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.2K
Fri 2023-09-22 07:29:16 AEST 1233 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.2K
Fri 2023-09-22 13:40:58 AEST 975 0 0 SIGABRT present /usr/lib/udisks2/udisksd 381.9K
Fri 2023-09-22 13:41:23 AEST 1377 0 0 SIGABRT present /usr/lib/udisks2/udisksd 444.5K
Fri 2023-09-22 20:57:18 AEST 996 0 0 SIGABRT present /usr/lib/udisks2/udisksd 381.5K
Fri 2023-09-22 20:57:44 AEST 1408 0 0 SIGSEGV present /usr/lib/systemd/systemd 496.0K
Fri 2023-09-22 22:57:29 AEST 859 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.4K
Fri 2023-09-22 22:57:54 AEST 1236 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.8K
Fri 2023-09-22 22:59:30 AEST 1012 0 0 SIGABRT present /usr/lib/udisks2/udisksd 382.3K
Fri 2023-09-22 22:59:55 AEST 1351 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.6K
Fri 2023-09-22 23:00:38 AEST 1066 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.0K
Fri 2023-09-22 23:01:04 AEST 1441 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.3K
Fri 2023-09-22 23:01:43 AEST 1794 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.1K
Fri 2023-09-22 23:02:08 AEST 1763 1000 1000 SIGTRAP present /usr/bin/gnome-disks 1.2M
Fri 2023-09-22 23:04:21 AEST 1890 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.2K
Fri 2023-09-22 23:04:46 AEST 1883 1000 1000 SIGTRAP present /usr/bin/gnome-disks 1.2M
Sat 2023-09-23 07:24:54 AEST 900 0 0 SIGABRT present /usr/lib/udisks2/udisksd 423.4K
Sat 2023-09-23 07:25:19 AEST 1278 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.0K
Sat 2023-09-23 07:29:49 AEST 920 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.6K
Sat 2023-09-23 07:30:14 AEST 1302 0 0 SIGSEGV present /usr/lib/udisks2/udisksd 183.3K
Sat 2023-09-23 07:33:00 AEST 1869 0 0 SIGSEGV present /usr/lib/udisks2/udisksd 183.6K
Sat 2023-09-23 07:33:25 AEST 1839 1000 1000 SIGTRAP present /usr/bin/gnome-disks 1.1M
Sat 2023-09-23 11:02:14 AEST 1004 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.9K
Sat 2023-09-23 11:02:40 AEST 1367 0 0 SIGILL present /usr/lib/udisks2/udisksd 381.6K
Sat 2023-09-23 11:19:48 AEST 1020 0 0 SIGABRT present /usr/lib/udisks2/udisksd 382.6K
Sat 2023-09-23 12:22:34 AEST 1030 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.5K
Sat 2023-09-23 12:22:59 AEST 1340 0 0 SIGABRT present /usr/lib/udisks2/udisksd 424.5K
Sat 2023-09-23 13:58:14 AEST 1005 0 0 SIGABRT present /usr/lib/udisks2/udisksd 425.3K
Sat 2023-09-23 13:58:39 AEST 1360 0 0 SIGABRT present /usr/lib/udisks2/udisksd 426.1K

He requested 2 specific core dump files which I have posted:

udiskd: https://www.dropbox.com/scl/fi/wk92277fd7gq27w2r88ft/udisks-1020.dump.gz?rlkey=879mgjm3sywrnyuar9h0ny0qz
gnome-disks: https://www.dropbox.com/scl/fi/unx0em2y64ewyeij46mos/gnome-disks-1839.dump.gz?rlkey=xcekuaxrs6bew0baxh7wo0kn8
Comment by Toolybird (Toolybird) - Sunday, 24 September 2023, 07:55 GMT
> changing this to low priority

I cannot repro and the issue is not widespread at this stage. BTW, I changed the severity, not the priority, which is a separate thing and not really used in Arch Flyspray. Although, it will likely get more use once we migrate to GitLab issues.

> sudo

If your user account is a member of the "wheel" group then none of this jumping through hoops is required. As long as your system is configured correctly and DEBUGINFOD_URLS exists in your environment, coredumpctl related commands "just work".

> core dump files

The whole point of debuginfod is to do away with coredump files. They are cumbersome to work with and are literally a waste of space.
Comment by Mark Blakeney (bulletmark) - Sunday, 24 September 2023, 08:27 GMT
I have always been in the wheel group.

lt:~ id
uid=1000(mark) gid=1000(mark) groups=1000(mark),108(vboxusers),969(docker),987(uucp),993(input),995(audio),998(wheel),999(adm)
lt:~ coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE
Thu 2023-09-21 07:27:05 AEST 859 0 0 SIGABRT missing /usr/lib/udisks2/udisksd -
Thu 2023-09-21 07:27:30 AEST 1244 0 0 SIGABRT missing /usr/lib/udisks2/udisksd -
Fri 2023-09-22 07:26:59 AEST 859 0 0 SIGSEGV inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 07:27:24 AEST 1250 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 07:28:50 AEST 859 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 07:29:16 AEST 1233 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 13:40:58 AEST 975 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 13:41:23 AEST 1377 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 20:57:18 AEST 996 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 20:57:44 AEST 1408 0 0 SIGSEGV inaccessible /usr/lib/systemd/systemd -
Fri 2023-09-22 22:57:29 AEST 859 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 22:57:54 AEST 1236 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 22:59:30 AEST 1012 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 22:59:55 AEST 1351 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 23:00:38 AEST 1066 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 23:01:04 AEST 1441 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 23:01:43 AEST 1794 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 23:02:08 AEST 1763 1000 1000 SIGTRAP present /usr/bin/gnome-disks 1.2M
Fri 2023-09-22 23:04:21 AEST 1890 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Fri 2023-09-22 23:04:46 AEST 1883 1000 1000 SIGTRAP present /usr/bin/gnome-disks 1.2M
Sat 2023-09-23 07:24:54 AEST 900 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 07:25:19 AEST 1278 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 07:29:49 AEST 920 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 07:30:14 AEST 1302 0 0 SIGSEGV inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 07:33:00 AEST 1869 0 0 SIGSEGV inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 07:33:25 AEST 1839 1000 1000 SIGTRAP present /usr/bin/gnome-disks 1.1M
Sat 2023-09-23 11:02:14 AEST 1004 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 11:02:40 AEST 1367 0 0 SIGILL inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 11:19:48 AEST 1020 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 12:22:34 AEST 1030 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 12:22:59 AEST 1340 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 13:58:14 AEST 1005 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sat 2023-09-23 13:58:39 AEST 1360 0 0 SIGABRT inaccessible /usr/lib/udisks2/udisksd -
Sun 2023-09-24 09:28:39 AEST 2358 1000 1000 SIGSEGV present /usr/lib/xdg-desktop-portal-gtk 2.4M
Comment by Mark Blakeney (bulletmark) - Sunday, 24 September 2023, 08:30 GMT
I agree core dump files are a PITA but the reason I created them was because loqs wanted to interrogate them so I posted them on dropbox.
Comment by Toolybird (Toolybird) - Sunday, 24 September 2023, 08:39 GMT
> I have always been in the wheel group

Ok, good. I could well be mistaken here. It's just been my experience..but maybe it depends on the circumstances...
Comment by Toolybird (Toolybird) - Sunday, 24 September 2023, 08:42 GMT
This looks suss: https://github.com/linux-nvme/libnvme/issues/684

Edit: According to that ticket, this patch [1] fixes problems. Wanna give it a try?

[1] https://github.com/linux-nvme/libnvme/commit/c56910f807795528fff7ba6b81f8efcdb4babe98
Comment by Mark Blakeney (bulletmark) - Sunday, 24 September 2023, 09:58 GMT
Built new libnvme with that patch and it does not fix the problem.
Comment by Norbert (Sandwich) - Sunday, 24 September 2023, 13:36 GMT
Hello, my friend has the same issue, this is the error:
Stack trace of thread 740:
#0 0x00007f10d1ffc553 g_log_writer_default (libglib-2.0.so.0 + 0x67553)
#1 0x00007f10d1ff61b5 g_log_structured_array (libglib-2.0.so.0 + 0x611b5)
#2 0x00007f10d1ff684f g_log_structured (libglib-2.0.so.0 + 0x6184f)
#3 0x0000564023575d2c udisks_log (udisksd + 0x4ed2c)
#4 0x0000564023546802 n/a (udisksd + 0x1f802)
#5 0x00007f10d2105e63 n/a (libgobject-2.0.so.0 + 0x24e63)
#6 0x00007f10d2107f0b g_object_new_valist (libgobject-2.0.so.0 + 0x26f0b)
#7 0x00007f10d210829e g_object_new (libgobject-2.0.so.0 + 0x2729e)
#8 0x0000564023543de6 udisks_daemon_new (udisksd + 0x1cde6)
#9 0x0000564023543e47 n/a (udisksd + 0x1ce47)
#10 0x00007f10d22537f8 n/a (libgio-2.0.so.0 + 0x1107f8)
#11 0x00007f10d21ecce4 n/a (libgio-2.0.so.0 + 0xa9ce4)
#12 0x00007f10d21f0bfd n/a (libgio-2.0.so.0 + 0xadbfd)
#13 0x00007f10d224ee63 n/a (libgio-2.0.so.0 + 0x10be63)
#14 0x00007f10d21ecce4 n/a (libgio-2.0.so.0 + 0xa9ce4)
#15 0x00007f10d21ecd1d n/a (libgio-2.0.so.0 + 0xa9d1d)
#16 0x00007f10d1feef19 n/a (libglib-2.0.so.0 + 0x59f19)
#17 0x00007f10d204d2b7 n/a (libglib-2.0.so.0 + 0xb82b7)
#18 0x00007f10d1fefb47 g_main_loop_run (libglib-2.0.so.0 + 0x5ab47)
#19 0x000056402354247d main (udisksd + 0x1b47d)
#20 0x00007f10d1c27cd0 n/a (libc.so.6 + 0x27cd0)
#21 0x00007f10d1c27d8a __libc_start_main (libc.so.6 + 0x27d8a)
#22 0x00005640235425c5 _start (udisksd + 0x1b5c5)

Stack trace of thread 744:
#0 0x00007f10d1d0359f __poll (libc.so.6 + 0x10359f)
#1 0x00007f10d204d206 n/a (libglib-2.0.so.0 + 0xb8206)
#2 0x00007f10d1fed112 g_main_context_iteration (libglib-2.0.so.0 + 0x58112)
#3 0x00007f10d1fed162 n/a (libglib-2.0.so.0 + 0x58162)
#4 0x00007f10d20209a5 n/a (libglib-2.0.so.0 + 0x8b9a5)
#5 0x00007f10d1c8c9eb n/a (libc.so.6 + 0x8c9eb)
#6 0x00007f10d1d10dfc n/a (libc.so.6 + 0x110dfc)

Stack trace of thread 745:
#0 0x00007f10d1d0ed6d syscall (libc.so.6 + 0x10ed6d)
#1 0x00007f10d2048247 g_cond_wait (libglib-2.0.so.0 + 0xb3247)
#2 0x00007f10d1fba1b4 n/a (libglib-2.0.so.0 + 0x251b4)

udisks daemon version 2.10.1 starting
Sep 22 21:44:31 Arch-AV udisksd[872]: failed to load module mdraid: libbytesize.so.1: cannot open shared object file: No such file or directory
Sep 22 21:44:31 Arch-AV udisksd[872]: Failed to load the 'mdraid' libblockdev plugin
Comment by Daniel Bermond (Bermond) - Monday, 25 September 2023, 18:45 GMT Comment by loqs (loqs) - Tuesday, 26 September 2023, 12:58 GMT
@Bermond @Sandwich does using the git HEAD of libnvme which includes payload alignment fixes [1] have any impact? Source package attached. You could also try the kernel parameter swiotlb=force so all accesses go through a bounce buffer which will deal with alignment but may cause other issues. This is to try and determine if everyone has the same issue caused by misaligned command payload.

[1] https://github.com/linux-nvme/nvme-cli/commit/1a8bd305c61d7bcc57bbf0eee8a2c4389820527c
Comment by Norbert (Sandwich) - Tuesday, 26 September 2023, 13:08 GMT
Hello @loqs, thank you for providing the Information. I have compiled and forwarded the info and the package to my friend. Hopefully will hear from him soon.
Comment by xsmile (xsmile) - Tuesday, 26 September 2023, 13:19 GMT
Hello, same issue here. Using the libnvme package provided by @loqs does not help.
Comment by loqs (loqs) - Tuesday, 26 September 2023, 13:43 GMT
@xsmile does you issue look like one of the existing upstream udisks issues? Meaning similar back trace or stack smashing detected or out of memory for example. If not please consider opening a new one.
Comment by xsmile (xsmile) - Tuesday, 26 September 2023, 13:50 GMT
@loqs: It looks the same. I can provide a trace later if needed.
Comment by xsmile (xsmile) - Tuesday, 26 September 2023, 16:37 GMT
Trace with debuginfo: https://privatebin.net/?b03559eb24861686#DRfWGRTi3TWJZUm2etgjwmQ3Jv7srVvhzbjVrEy77QZR

Running the command `nvme sanitize-log -H /dev/nvme0` multiple times as per https://github.com/storaged-project/udisks/issues/1152 did not produce a stack smashing error.
Comment by Norbert (Sandwich) - Tuesday, 26 September 2023, 17:04 GMT
@loqs, this patch with the latest git HEAD did not help my friend either.
Comment by loqs (loqs) - Tuesday, 26 September 2023, 18:43 GMT
The attached package returns from bd_nvme_get_namespace_info and bd_nvme_get_controller_info before calling ioctl which will disable NVME probing.
Comment by xsmile (xsmile) - Tuesday, 26 September 2023, 19:09 GMT
In my case most of the time udisks fails to run and produces a stack smashing error. Sometimes it runs successfully though.

The issue remains with the patched libblockdev package by @loqs. On a successful run it additionally produces a warning: "Error performing housekeeping for drive org/freedesktop/UDisks2/drives/SKHynix_HFS001TD9TNI_L2B0B_NS04N638710704H1Q: Error updating Health Information: No probed controller info available (udisks-error-quark, 0)"
Comment by loqs (loqs) - Friday, 06 October 2023, 10:10 GMT
libblockdev with [1] applied. src PKGBUILD attached, built package linked below. Combined with current libnvme (1.6-1) is intended to fix stack smashing.

https://drive.google.com/file/d/1GkIYYHv9MLwqh_qDlKoMAkK6qwVp-x9p/view?usp=sharing libblockdev-3.0.3-4.2-x86_64.pkg.tar.zst

[1] https://github.com/storaged-project/libblockdev/pull/969/commits/2ae0d949eb87142b0212e5953a0e5ad1a146ed6b
Comment by xsmile (xsmile) - Friday, 06 October 2023, 10:56 GMT
Tested with current versions of libnvme, udisks2 and the prebuilt libblockdev package by @loqs - the stack smashing error is still present.
Comment by loqs (loqs) - Friday, 06 October 2023, 11:07 GMT
@xsmile thank you for the prompt testing. Can you please report that on either [1] or [2].
Edit:
Please also include an updated backtrace.

[1] https://github.com/storaged-project/libblockdev/pull/969
[2] https://github.com/storaged-project/udisks/issues/1152
Comment by Mark Blakeney (bulletmark) - Saturday, 07 October 2023, 03:00 GMT
I can also confirm that testing with current libnvme, udisks2, and my own build of libblockdev with above [1] patch applied does not fix the problem.
Comment by xsmile (xsmile) - Sunday, 08 October 2023, 13:39 GMT
I created a new issue with some more details. See https://github.com/storaged-project/udisks/issues/1198.
Comment by loqs (loqs) - Sunday, 08 October 2023, 14:22 GMT
What if you increase [1] to 4096 might also need to be allocated by posix_memalign for alignment.

[1] https://github.com/linux-nvme/libnvme/blob/v1.6/src/nvme/tree.c#L2408
Comment by xsmile (xsmile) - Sunday, 08 October 2023, 18:14 GMT
As far as I can tell, NVME_IDENTIFY_DATA_SIZE is already 4096 by default. Just increasing it did not help.
Comment by loqs (loqs) - Sunday, 08 October 2023, 19:01 GMT
> As far as I can tell, NVME_IDENTIFY_DATA_SIZE is already 4096 by default.
My mistake.

The attached patch attempts to perform the alignment but it is probably not worth testing.
Comment by Mark Blakeney (bulletmark) - Sunday, 08 October 2023, 23:49 GMT
@loqs: thanks for pursuing this but that test.patch does not fix it.
Comment by Daniel Bermond (Bermond) - Monday, 09 October 2023, 18:03 GMT
I'm affected by an infinite loop that leads to OOM killing (https://github.com/storaged-project/udisks/issues/1152).

This is now fixed in libblockdev git master branch, and can be solved by applying the following libblockdev commit on top of version 3.0.3:

https://github.com/storaged-project/libblockdev/commit/2ae0d949eb87142b0212e5953a0e5ad1a146ed6b

@loqs Sorry for not replying / testing new libnvme stuff. My issue was not related to libnvme anyway.
Comment by loqs (loqs) - Monday, 09 October 2023, 18:27 GMT Comment by xsmile (xsmile) - Tuesday, 10 October 2023, 17:56 GMT
A fixed libblockdev package is already in Extra-Testing. Additionally, the libnvme package requires the latest PR. See [1].

1: https://github.com/storaged-project/udisks/issues/1198#issuecomment-1755795662
Comment by Kevin (prurigro) - Tuesday, 10 October 2023, 18:09 GMT
Chiming in here to say that I'm getting the same issue on one of my boxes and https://github.com/storaged-project/libblockdev/commit/2ae0d949eb87142b0212e5953a0e5ad1a146ed6b applied to the libblockdev package in testing doesn't fix it.
Comment by xsmile (xsmile) - Tuesday, 10 October 2023, 18:15 GMT
@prurigro Please test it together with the recent libnvme change at https://github.com/linux-nvme/libnvme/pull/727 .
Comment by Kevin (prurigro) - Tuesday, 10 October 2023, 18:18 GMT
@xsmile: apologies- I'd had the page open for a few minutes and hadn't seen your message. I just tested with https://github.com/linux-nvme/libnvme/pull/727 and unfortunately the issue is still present.
Comment by loqs (loqs) - Tuesday, 10 October 2023, 18:23 GMT
@prurigro there could be other calls that need fixing in libvme [1]. Has the backtrace changed with the updates to libblockdev and libnvme? What does it show now?

[1] https://github.com/linux-nvme/libnvme/pull/727#issuecomment-1755896246
Comment by Kevin (prurigro) - Tuesday, 10 October 2023, 19:05 GMT
@xsmile: So I decided to rebuild the patched packages just in case I'd made a mistake the first time, and after two successful test boots, it seems like I'm not hitting the error anymore! Sorry for the noise and thanks for getting this fix organized :)
Comment by Mark Blakeney (bulletmark) - Tuesday, 10 October 2023, 23:04 GMT
I applied storaged-project/libblockdev#969 (available now in extra-testing libblockdev 3.0.3-5) and linux-nvme/libnvme#727 and it does fix the issue. Running with just one or the other still causes the issue. I need both patched libblockdev and libnvme.
Comment by Kevin (prurigro) - Thursday, 12 October 2023, 16:09 GMT
It might be worth bumping up the severity of this a bit as it seems to randomly take out systemd in a way that prevents you from restarting it, interacting with services or even restarting the computer (without forcing shutdown).
Comment by David Runge (dvzrv) - Friday, 13 October 2023, 08:53 GMT
The updated libblockdev (3.0.4) is now in [extra].
I have applied the relevant patch for libnvme, which is now in [extra-testing].

Please provide feedback, if that works now as intended!
Comment by Mark Blakeney (bulletmark) - Friday, 13 October 2023, 12:08 GMT
@dvzrv, I updated my system which updated libblockdev to 3.0.4-1 and I pulled in libnvme 1.6-2 from extra-testing and confirm that the problem does not occur. Thanks for the updates.
Comment by David Runge (dvzrv) - Friday, 13 October 2023, 15:59 GMT
@bulletmark: Thanks for the confirmation! :)

Loading...