As a workaround, I replaced /usr/bin/osquery with the 4.6.0 pre-compiled binary from https://osquery.io/downloads/official/4.6.0. This seems to work for now.

Christian I think it is related to your recent lvm2 update

Oh damn...
Is the system suffering this actively using lvm2?

I don't believe I am using LVM. I'm not familiar with it's usage, but running "pvs" and "lvs" shows nothing, and I'm pretty sure I didn't use it when setting up this machine.

Andrey, osquery crashes at the start time probably because it failed to initialize complied-in lvm module.

Hi Anatol,

I just want to clarify, are you saying this is an issue on my end? I'm not sure that it is, since the binary available from osquery.io seems to be working fine. I'm happy to work with you to solve this problem, but might need hand holding, since I'm not terribly familiar with the osquery application myself. All I've done with osquery is instal and configure it with a specific host and secret. This is to fulfill security requirements at my workplace for folks running Linux laptops on corporate laptops. At least one other person, also running Arch, has the same segfault issue.

@Andrey, no, the issue is with the Arch's osquery PKGBUILD that pulls and statically compiles LVM library. There must be some configuration problem with it.

Cool, that was my understanding too. I've built a PKGBUILD or two in the past (but am no expert), and available and happy to help testing any changes on my machine.

Would you consider dropping lvm2 functionality as an alternative until the functionality can be replaced?
The attached patch has been built tested only.

Ioqs, yep it makes sense to disable LVM for now. It is better to have osquery without LVM then do not have osquery at all. I will look at the patch.

WRT osquery its build system is a mess with a bunch of hardcoded versions dependencies. About a year ago I've done some work on osquery de-vendorization to follow Arch standard practices. I was hoping that upstream will move forward towards this direction. Unfortunately after a brief spike of interest to this idea they went radio silent. Porting this huge de-vendorization effort to newer major osquery releases is a pain for me. And that is the reason why osquery 4.6.0 is not pushed to [testing] yet. The patch slows down the release process on my side.

So I am thinking about dropping this de-vendorization effort and switch back to the huge static build with libraries checked-out and built and by osquery itself. It is far from ideal but at least it let us get 4.6.0 sooner.

I have 4.6.0 building with the devendorized patches. Added cmake system lib support for dbus, libcap and expat.
Currently a very messy work in progress. Correction smartmon is still using the downloaded snapshot.

@Ioqs, it sounds great.

Actually I looked at returning back to vendorized form of the package yesterday. I have a 4.6.0 package that I am testing right now.

Ioqs, do you want to take over this de-vendorized patchset and continue working with upstream? I would be great to see:
1) decoupling from libc++. osquery should work with glib as well.
2) use system library (the de-vendorization itself)

Unfortunately my build fails with following error:

Mar 07 16:22:13 wolf.lan osqueryd[279175]: E0307 16:22:13.413367 279175 shutdown.cpp:69] Cannot activate tls logger plugin: No node key, TLS logging disabled.
Mar 07 16:22:13 wolf.lan osqueryd[279175]: Cannot activate tls logger plugin: No node key, TLS logging disabled.
Mar 07 16:22:13 wolf.lan audit[279175]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=279175 comm="osqueryd" exe="/usr/bin/osqueryd" sig=11 res=1
Mar 07 16:22:13 wolf.lan kernel: osqueryd[279175]: segfault at 98 ip 0000561178da8175 sp 00007ffd6e3511a0 error 4 in osqueryd[561178d24000+1140000]
Mar 07 16:22:13 wolf.lan kernel: Code: c3 e8 ef be f7 ff 55 41 56 53 48 83 ec 10 41 89 d6 48 89 f5 48 89 fb 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 e8 cb 7e fb ff <0f> b6 55 00 f6 c2 01 74 0a 48 8b 55 08 48 8b 6d 10 eb 06 48 d1 ea
Mar 07 16:22:13 wolf.lan kernel: audit: type=1701 audit(1615162933.412:996): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=279175 comm="osqueryd" exe="/usr/bin/osqueryd" sig=11 res=1
Mar 07 16:22:13 wolf.lan audit: BPF prog-id=261 op=LOAD
Mar 07 16:22:13 wolf.lan audit: BPF prog-id=262 op=LOAD
Mar 07 16:22:13 wolf.lan kernel: audit: type=1334 audit(1615162933.442:997): prog-id=261 op=LOAD
Mar 07 16:22:13 wolf.lan kernel: audit: type=1334 audit(1615162933.442:998): prog-id=262 op=LOAD
Mar 07 16:22:13 wolf.lan systemd[1]: Started Process Core Dump (PID 279518/UID 0).
Mar 07 16:22:13 wolf.lan audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@102-279518-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 07 16:22:13 wolf.lan kernel: audit: type=1130 audit(1615162933.445:999): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@102-279518-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 07 16:22:14 wolf.lan systemd-coredump[279519]: Process 279175 (osqueryd) of user 0 dumped core.

Stack trace of thread 279175:
#0 0x0000561178da8175 _ZNSt3__1plIcNS_11char_traitsIcEENS_9allocatorIcEEEENS_12basic_stringIT_T0_T1_EERKS9_S6_ (osqueryd + 0x34b175)
#1 0x0000561178de4a4a _ZN7osquery20BufferedLogForwarder14genIndexPrefixEb (osqueryd + 0x387a4a)
#2 0x0000561178de56c6 _ZN7osquery20BufferedLogForwarder8genIndexEbm (osqueryd + 0x3886c6)
#3 0x0000561178de558a _ZN7osquery20BufferedLogForwarder14genStatusIndexEm (osqueryd + 0x38858a)
#4 0x0000561178de5163 _ZN7osquery20BufferedLogForwarder9logStatusERKNSt3__16vectorINS_13StatusLogLineENS1_9allocatorIS3_EEEEm (osqueryd + 0x388163)
#5 0x0000561178de9a02 _ZN7osquery15TLSLoggerPlugin9logStatusERKNSt3__16vectorINS_13StatusLogLineENS1_9allocatorIS3_EEEE (osqueryd + 0x38ca02)
#6 0x0000561179b4e016 _ZN7osquery12LoggerPlugin4callERKNSt3__13mapINS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES8_NS1_4lessIS8_EENS6_INS1_4pairIKS8_S8_EEEEEERNS1_6vectorISF_NS6_ISF_EEEE (osqueryd + 0x10f1016)
#7 0x0000561179b25e56 _ZN7osquery17RegistryInterface4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c8e56)
#8 0x0000561179b24229 _ZN7osquery15RegistryFactory4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES9_RKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c7229)
#9 0x0000561179ab30b0 _ZZN7osquery15relayStatusLogsEbENK3$_0clEv (osqueryd + 0x10560b0)
#10 0x0000561179ab1aae _ZN7osquery15relayStatusLogsEb (osqueryd + 0x1054aae)
#11 0x0000561179ab189f _ZN7osquery10initLoggerERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE (osqueryd + 0x105489f)
#12 0x000056117997f424 _ZNK7osquery11Initializer5startEv (osqueryd + 0xf22424)
#13 0x0000561178e994d9 _ZN7osquery11startDaemonERNS_11InitializerE (osqueryd + 0x43c4d9)
#14 0x0000561178e99a76 _ZN7osquery12startOsqueryEiPPc (osqueryd + 0x43ca76)
#15 0x0000561178e99158 main (osqueryd + 0x43c158)
#16 0x00007f41d68beb25 __libc_start_main (libc.so.6 + 0x27b25)
#17 0x0000561178d5e20e _start (osqueryd + 0x30120e)

Stack trace of thread 279177:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73260 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf260)
#2 0x00007f41d6c2bb90 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1 + 0x45b90)
#3 0x00005611790649f5 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (osqueryd + 0x6079f5)
#4 0x0000561179064cb4 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (osqueryd + 0x607cb4)
#5 0x0000561179065dbb _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEPFvPvEPN7rocksdb16BGThreadMetadataEEEEEES7_S7_ (osqueryd + 0x608dbb)
#6 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#7 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279181:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73260 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf260)
#2 0x00007f41d6c2bb90 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1 + 0x45b90)
#3 0x00005611790649f5 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (osqueryd + 0x6079f5)
#4 0x0000561179064cb4 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (osqueryd + 0x607cb4)
#5 0x0000561179065dbb _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEPFvPvEPN7rocksdb16BGThreadMetadataEEEEEES7_S7_ (osqueryd + 0x608dbb)
#6 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#7 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279178:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73260 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf260)
#2 0x00007f41d6c2bb90 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1 + 0x45b90)
#3 0x00005611790649f5 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (osqueryd + 0x6079f5)
#4 0x0000561179064cb4 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (osqueryd + 0x607cb4)
#5 0x0000561179065dbb _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEPFvPvEPN7rocksdb16BGThreadMetadataEEEEEES7_S7_ (osqueryd + 0x608dbb)
#6 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#7 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279179:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73260 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf260)
#2 0x00007f41d6c2bb90 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1 + 0x45b90)
#3 0x00005611790649f5 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (osqueryd + 0x6079f5)
#4 0x0000561179064cb4 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (osqueryd + 0x607cb4)
#5 0x0000561179065dbb _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEPFvPvEPN7rocksdb16BGThreadMetadataEEEEEES7_S7_ (osqueryd + 0x608dbb)
#6 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#7 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279274:
#0 0x00007f41d698b37f __poll (libc.so.6 + 0xf437f)
#1 0x0000561178ee3055 _ZN6apache6thrift9transport13TServerSocket10acceptImplEv (osqueryd + 0x486055)
#2 0x0000561178eeb43d _ZN6apache6thrift9transport16TServerTransport6acceptEv (osqueryd + 0x48e43d)
#3 0x0000561178eeabc4 _ZN6apache6thrift6server16TServerFramework5serveEv (osqueryd + 0x48dbc4)
#4 0x0000561178ee5506 _ZN6apache6thrift6server15TThreadedServer5serveEv (osqueryd + 0x488506)
#5 0x0000561178dc6d4c _ZN7osquery24ExtensionRunnerInterface5serveEv (osqueryd + 0x369d4c)
#6 0x0000561179b34d15 _ZN7osquery19ExtensionRunnerCore11startServerEv (osqueryd + 0x10d7d15)
#7 0x0000561179b352dd _ZN7osquery22ExtensionManagerRunner5startEv (osqueryd + 0x10d82dd)
#8 0x0000561179b20563 _ZN7osquery16InternalRunnable3runEv (osqueryd + 0x10c3563)
#9 0x0000561179b215b5 _ZNSt3__18__invokeIRMN7osquery16InternalRunnableEFvvERPS2_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS8_DpOS9_ (osqueryd + 0x10c45b5)
#10 0x0000561179b21575 _ZNSt3__16__bindIMN7osquery16InternalRunnableEFvvEJPS2_EEclIJEEENS_13__bind_returnIS4_NS_5tupleIJS5_EEENS9_IJDpOT_EEEXsr22__is_valid_bind_returnIS4_SA_SE_EE5valueEE4typeESD_ (osqueryd + 0x10c4575)
#11 0x0000561179b214f9 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN7osquery16InternalRunnableEFvvEJPS9_EEEEEEEEPvSF_ (osqueryd + 0x10c44f9)
#12 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#13 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279180:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73260 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf260)
#2 0x00007f41d6c2bb90 _ZNSt3__118condition_variable4waitERNS_11unique_lockINS_5mutexEEE (libc++.so.1 + 0x45b90)
#3 0x00005611790649f5 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (osqueryd + 0x6079f5)
#4 0x0000561179064cb4 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (osqueryd + 0x607cb4)
#5 0x0000561179065dbb _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEPFvPvEPN7rocksdb16BGThreadMetadataEEEEEES7_S7_ (osqueryd + 0x608dbb)
#6 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#7 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279176:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73852 pthread_cond_clockwait (libpthread.so.0 + 0xf852)
#2 0x0000561178e4cd6c _ZNSt3__118condition_variable15__do_timed_waitERNS_11unique_lockINS_5mutexEEENS_6chrono10time_pointINS5_12steady_clockENS5_8durationIxNS_5ratioILl1ELl1000000000EEEEEEE (osqueryd + 0x3efd6c)
#3 0x0000561179b204c5 _ZNSt3__118condition_variable8wait_forIxNS_5ratioILl1ELl1000EEEEENS_9cv_statusERNS_11unique_lockINS_5mutexEEERKNS_6chrono8durationIT_T0_EE (osqueryd + 0x10c34c5)
#4 0x0000561179b20425 _ZN7osquery21InterruptableRunnable5pauseENSt3__16chrono8durationIxNS1_5ratioILl1ELl1000EEEEE (osqueryd + 0x10c3425)
#5 0x0000561179982f92 _ZN7osquery20WatcherWatcherRunner5startEv (osqueryd + 0xf25f92)
#6 0x0000561179b20563 _ZN7osquery16InternalRunnable3runEv (osqueryd + 0x10c3563)
#7 0x0000561179b215b5 _ZNSt3__18__invokeIRMN7osquery16InternalRunnableEFvvERPS2_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS8_DpOS9_ (osqueryd + 0x10c45b5)
#8 0x0000561179b21575 _ZNSt3__16__bindIMN7osquery16InternalRunnableEFvvEJPS2_EEclIJEEENS_13__bind_returnIS4_NS_5tupleIJS5_EEENS9_IJDpOT_EEEXsr22__is_valid_bind_returnIS4_SA_SE_EE5valueEE4typeESD_ (osqueryd + 0x10c4575)
#9 0x0000561179b214f9 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN7osquery16InternalRunnableEFvvEJPS9_EEEEEEEEPvSF_ (osqueryd + 0x10c44f9)
#10 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#11 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279272:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73574 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf574)
#2 0x000056117907a68b _ZN7rocksdb4port7CondVar9TimedWaitEm (osqueryd + 0x61d68b)
#3 0x0000561178fe73ff _ZN7rocksdb19InstrumentedCondVar9TimedWaitEm (osqueryd + 0x58a3ff)
#4 0x0000561178f80011 _ZN7rocksdb5Timer3RunEv (osqueryd + 0x523011)
#5 0x0000561178f80ec7 _ZNSt3__18__invokeIMN7rocksdb5TimerEFvvEPS2_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS6_DpOS7_ (osqueryd + 0x523ec7)
#6 0x0000561178f80e87 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN7rocksdb5TimerEFvvEJPS7_EJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE (osqueryd + 0x523e87)
#7 0x0000561178f80e05 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN7rocksdb5TimerEFvvEPS8_EEEEEPvSD_ (osqueryd + 0x523e05)
#8 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#9 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279279:
#0 0x00007f41d695e125 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xc7125)
#1 0x00007f41d6963357 __nanosleep (libc.so.6 + 0xcc357)
#2 0x00007f41d6c7e2d9 _ZNSt3__111this_thread9sleep_forERKNS_6chrono8durationIxNS_5ratioILl1ELl1000000000EEEEE (libc++.so.1 + 0x982d9)
#3 0x0000561178d881f4 _ZNSt3__111this_thread9sleep_forIxNS_5ratioILl1ELl1000EEEEEvRKNS_6chrono8durationIT_T0_EE (osqueryd + 0x32b1f4)
#4 0x0000561178d87f9c _ZN7osquery8sleepForEm (osqueryd + 0x32af9c)
#5 0x0000561178db5ecb _ZN7osquery15TLSEnrollPlugin6enrollEv (osqueryd + 0x358ecb)
#6 0x0000561178db8f7b _ZN7osquery12EnrollPlugin4callERKNSt3__13mapINS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES8_NS1_4lessIS8_EENS6_INS1_4pairIKS8_S8_EEEEEERNS1_6vectorISF_NS6_ISF_EEEE (osqueryd + 0x35bf7b)
#7 0x0000561179b25e56 _ZN7osquery17RegistryInterface4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c8e56)
#8 0x0000561179b24229 _ZN7osquery15RegistryFactory4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES9_RKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c7229)
#9 0x0000561178db8a27 _ZN7osquery10getNodeKeyERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE (osqueryd + 0x35ba27)
#10 0x0000561178ddb355 _ZN7osquery16TLSRequestHelper2goINS_14JSONSerializerEEENS_6StatusERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEERNS_4JSONESE_ (osqueryd + 0x37e355)
#11 0x0000561178ddb246 _ZN7osquery16TLSRequestHelper2goINS_14JSONSerializerEEENS_6StatusERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEERNS_4JSONERSA_ (osqueryd + 0x37e246)
#12 0x0000561178ddaa2c _ZN7osquery16TLSRequestHelper2goINS_14JSONSerializerEEENS_6StatusERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEERNS_4JSONERSA_m (osqueryd + 0x37da2c)
#13 0x0000561178dda61b _ZN7osquery15TLSConfigPlugin9genConfigERNSt3__13mapINS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES8_NS1_4lessIS8_EENS6_INS1_4pairIKS8_S8_EEEEEE (osqueryd + 0x37d61b)
#14 0x0000561178dab20f _ZN7osquery12ConfigPlugin4callERKNSt3__13mapINS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES8_NS1_4lessIS8_EENS6_INS1_4pairIKS8_S8_EEEEEERNS1_6vectorISF_NS6_ISF_EEEE (osqueryd + 0x34e20f)
#15 0x0000561179b25e56 _ZN7osquery17RegistryInterface4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c8e56)
#16 0x0000561179b24229 _ZN7osquery15RegistryFactory4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES9_RKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c7229)
#17 0x0000561179b24650 _ZN7osquery15RegistryFactory4callERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEERNS1_6vectorISG_NS5_ISG_EEEE (osqueryd + 0x10c7650)
#18 0x0000561178da6d59 _ZN7osquery6Config7refreshEv (osqueryd + 0x349d59)
#19 0x0000561178daba99 _ZN7osquery19ConfigRefreshRunner5startEv (osqueryd + 0x34ea99)
#20 0x0000561179b20563 _ZN7osquery16InternalRunnable3runEv (osqueryd + 0x10c3563)
#21 0x0000561179b215b5 _ZNSt3__18__invokeIRMN7osquery16InternalRunnableEFvvERPS2_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS8_DpOS9_ (osqueryd + 0x10c45b5)
#22 0x0000561179b21575 _ZNSt3__16__bindIMN7osquery16InternalRunnableEFvvEJPS2_EEclIJEEENS_13__bind_returnIS4_NS_5tupleIJS5_EEENS9_IJDpOT_EEEXsr22__is_valid_bind_returnIS4_SA_SE_EE5valueEE4typeESD_ (osqueryd + 0x10c4575)
#23 0x0000561179b214f9 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN7osquery16InternalRunnableEFvvEJPS9_EEEEEEEEPvSF_ (osqueryd + 0x10c44f9)
#24 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#25 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Stack trace of thread 279273:
#0 0x00007f41d6a799ba __futex_abstimed_wait_common64 (libpthread.so.0 + 0x159ba)
#1 0x00007f41d6a73852 pthread_cond_clockwait (libpthread.so.0 + 0xf852)
#2 0x0000561178e4cd6c _ZNSt3__118condition_variable15__do_timed_waitERNS_11unique_lockINS_5mutexEEENS_6chrono10time_pointINS5_12steady_clockENS5_8durationIxNS_5ratioILl1ELl1000000000EEEEEEE (osqueryd + 0x3efd6c)
#3 0x0000561179b204c5 _ZNSt3__118condition_variable8wait_forIxNS_5ratioILl1ELl1000EEEEENS_9cv_statusERNS_11unique_lockINS_5mutexEEERKNS_6chrono8durationIT_T0_EE (osqueryd + 0x10c34c5)
#4 0x0000561179b20425 _ZN7osquery21InterruptableRunnable5pauseENSt3__16chrono8durationIxNS1_5ratioILl1ELl1000EEEEE (osqueryd + 0x10c3425)
#5 0x0000561178d9d618 _ZN7osquery23ExtensionManagerWatcher5startEv (osqueryd + 0x340618)
#6 0x0000561179b20563 _ZN7osquery16InternalRunnable3runEv (osqueryd + 0x10c3563)
#7 0x0000561179b215b5 _ZNSt3__18__invokeIRMN7osquery16InternalRunnableEFvvERPS2_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS8_DpOS9_ (osqueryd + 0x10c45b5)
#8 0x0000561179b21575 _ZNSt3__16__bindIMN7osquery16InternalRunnableEFvvEJPS2_EEclIJEEENS_13__bind_returnIS4_NS_5tupleIJS5_EEENS9_IJDpOT_EEEXsr22__is_valid_bind_returnIS4_SA_SE_EE5valueEE4typeESD_ (osqueryd + 0x10c4575)
#9 0x0000561179b214f9 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN7osquery16InternalRunnableEFvvEJPS9_EEEEEEEEPvSF_ (osqueryd + 0x10c44f9)
#10 0x00007f41d6a6d299 start_thread (libpthread.so.0 + 0x9299)
#11 0x00007f41d6996053 __clone (libc.so.6 + 0xff053)

Here is the PKGBUILD file

@Ioqs, and with your patch the osquery fails for me

Mar 07 17:11:12 wolf.lan kernel: osqueryd[331713]: segfault at c0 ip 000055e9e9462a0d sp 00007ffefb608320 error 6 in osqueryd[55e9e931e000+5a0000]
Mar 07 17:11:12 wolf.lan kernel: Code: 89 d3 48 81 ec e8 00 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 d8 00 00 00 31 c0 48 85 c9 0f 84 80 05 00 00 bd 01 00 00 00 <f0> 49 0f c1 af c0 00 00 00 48 83 c5 01 48 83 fd 09 0f 86 ee 06 00
Mar 07 17:11:12 wolf.lan kernel: audit: type=1701 audit(1615165872.075:1787): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=331713 comm="osqueryd" exe="/usr/bin/osqueryd" sig=11 res=1
Mar 07 17:11:12 wolf.lan audit: BPF prog-id=461 op=LOAD
Mar 07 17:11:12 wolf.lan kernel: audit: type=1334 audit(1615165872.105:1788): prog-id=461 op=LOAD
Mar 07 17:11:12 wolf.lan kernel: audit: type=1334 audit(1615165872.105:1789): prog-id=462 op=LOAD
Mar 07 17:11:12 wolf.lan audit: BPF prog-id=462 op=LOAD
Mar 07 17:11:12 wolf.lan systemd[1]: Started Process Core Dump (PID 331936/UID 0).
Mar 07 17:11:12 wolf.lan audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@192-331936-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 07 17:11:12 wolf.lan kernel: audit: type=1130 audit(1615165872.109:1790): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@192-331936-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 07 17:11:13 wolf.lan systemd-coredump[331937]: [🡕] Process 331713 (osqueryd) of user 0 dumped core.

Stack trace of thread 331713:
#0 0x000055e9e9462a0d _ZN7osquery20BufferedLogForwarder8genIndexB5cxx11Ebm (osqueryd + 0x1d4a0d)
#1 0x000055e9e9467975 _ZN7osquery20BufferedLogForwarder9logStatusERKSt6vectorINS_13StatusLogLineESaIS2_EEm (osqueryd + 0x1d9975)
#2 0x000055e9e946cf25 _ZN7osquery15TLSLoggerPlugin9logStatusERKSt6vectorINS_13StatusLogLineESaIS2_EE (osqueryd + 0x1def25)
#3 0x000055e9e98a9461 _ZN7osquery12LoggerPlugin4callERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4lessIS7_ESaISt4pairIKS7_S7_EEERSt6vectorISE_SaISE_EE (osqueryd + 0x61b461)
#4 0x000055e9e9817ec2 _ZN7osquery17RegistryInterface4callERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt3mapIS6_S6_St4lessIS6_ESaISt4pairIS7_S6_EEERSt6vectorISF_SaISF_EE (osqueryd + 0x589ec2)
#5 0x000055e9e980ebc2 _ZN7osquery15RegistryFactory4callERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_RKSt3mapIS6_S6_St4lessIS6_ESaISt4pairIS7_S6_EEERSt6vectorISF_SaISF_EE (osqueryd + 0x580bc2)
#6 0x000055e9e97c43f1 _ZZN7osquery15relayStatusLogsEbENKUlvE_clEv.constprop.0 (osqueryd + 0x5363f1)

Andrey, and others. Here is an AUR package that installs the upstream binary. It will unblock you until the mess with osquery is clarified

https://aur.archlinux.org/packages/osquery-bin/

Heck, even the prebuilt upstream binary crashes for me.

I probably hit this bug https://github.com/osquery/osquery/issues/6887

@anatolik I can not reproduce that issue using osquery 4.5.1-8, this is with a default config with osquery-monitoring and hardware-monitoring packs enabled.

@Ioqs 4.5.1-8 already contains the fix for my issue https://github.com/osquery/osquery/commit/d69380cb7713dd5f7f48849a028a7723e7a619bf

It also contains your nolvm2 patch to disable LVM2 functionality. Hopefully it fixes the original issue. @Andrey please check it.

The work on 4.6.0 should continue. Either with your (@Ioqs) devendarization work or returning back to the upstream strategy of building all the dependencies from sources.

This is the patch set for 4.6.0 and the matching PKGBUILD.

thank a lot @Ioqs. Is there any chance you can put it to a github branch? It will help to fork/add new patches on top of this work.

@Anatol, I can confirm that 4.5.1-8 no longer segfaults here. Thanks for the quick fix!

Hopefully I created the branch [1] and tag [2] correctly.

[1] https://github.com/loqs/osquery/tree/devendor
[2] https://github.com/loqs/osquery/tree/4.6.0-a1

@Andrey, thank you for the verification. Hence I am closing the ticket as fixed.

@loqs thanks a lot for you work! I just pushed 4.6.0 with your de-verndorization work to [community-testing]. Please test it and file tickets if you see any issues with it.