FS#66688 - [linux-hardened] takes all memory allocated to virtual machine
Attached to Project:
Arch Linux
Opened by Amin Vakil (aminvakil) - Sunday, 17 May 2020, 15:12 GMT
Last edited by Levente Polyak (anthraxx) - Sunday, 17 May 2020, 20:12 GMT
Opened by Amin Vakil (aminvakil) - Sunday, 17 May 2020, 15:12 GMT
Last edited by Levente Polyak (anthraxx) - Sunday, 17 May 2020, 20:12 GMT
|
Details
I have a 48GB RAM server with KVM hypervisor (Proxmox)
installed on it and I've created a vm installing archlinux
on it. I can allocate as much ram as I want to vm and
hypervisor doesn't care as long as it doesn't go
out-of-memory, if it does will kill the process of virtual
machine to free memory.
When I install linux kernel and boot my arch on it, it's fine and my arch will forcefully shut down if I use too much memory and hypervisor goes out-of-memory. But when I boot my arch with linux-hardened it doesn't boot up, it will stuck on "loading initial ramdisk" for a few seconds and then hypervisor forcefully shuts it down as it was using too much memory. I've allocated 40GB (to this vm and started it, dmesg on hypervisor after a few seconds stucking on "loading initial ramdisk": [463934.050187] kvm invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [463934.050273] CPU: 1 PID: 26756 Comm: kvm Tainted: P IO 5.3.18-3-pve #1 [463934.050349] Hardware name: XXX [463934.050426] Call Trace: [463934.050485] dump_stack+0x6d/0x9a [463934.050541] dump_header+0x4f/0x1e1 [463934.050597] oom_kill_process.cold.34+0xb/0x10 [463934.050654] out_of_memory+0x1ad/0x490 [463934.050711] __alloc_pages_slowpath+0xa9d/0xe40 [463934.050769] __alloc_pages_nodemask+0x2df/0x330 [463934.050828] alloc_pages_current+0x81/0xe0 [463934.050885] __page_cache_alloc+0x6a/0xa0 [463934.050941] pagecache_get_page+0xaf/0x2d0 [463934.050997] filemap_fault+0x74e/0xa40 [463934.051054] ? unlock_page_memcg+0x12/0x20 [463934.051112] ? page_add_file_rmap+0x119/0x160 [463934.051169] ? xas_load+0xc/0x80 [463934.051222] ? xas_find+0x17e/0x1b0 [463934.051277] ? filemap_map_pages+0x1ae/0x380 [463934.051336] ext4_filemap_fault+0x31/0x44 [463934.051393] __do_fault+0x3c/0x130 [463934.051448] __handle_mm_fault+0xe66/0x1280 [463934.051505] ? set_fd_set.part.6+0x50/0x50 [463934.051562] handle_mm_fault+0xc5/0x1e0 [463934.051619] __do_page_fault+0x233/0x4c0 [463934.051708] ? kvm_on_user_return+0x6f/0xa0 [kvm] [463934.051766] do_page_fault+0x2c/0xe0 [463934.051821] page_fault+0x34/0x40 [463934.051875] RIP: 0033:0x7fc80c1748e0 [463934.051933] Code: Bad RIP value. [463934.051986] RSP: 002b:00007ffe7f2c0c18 EFLAGS: 00010202 [463934.052043] RAX: 00007fc7fd31ea00 RBX: 00007fc7fd31ea00 RCX: 0000000000000000 [463934.052118] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007fc7fd31ea00 [463934.052193] RBP: 00007ffe7f2c0c50 R08: 00007ffe7f2c0b98 R09: 00007ffe7f2c0b90 [463934.052269] R10: 0000000000071439 R11: 00007ffe7f2dd090 R12: 00007fc7fd31ea00 [463934.052343] R13: 00007ffe7f2c0dc0 R14: 00007ffe7f2c0c4c R15: 0000000000000000 [463934.052420] Mem-Info: [463934.052476] active_anon:12045597 inactive_anon:10968 isolated_anon:41 active_file:187 inactive_file:100 isolated_file:1 unevictable:1330 dirty:0 writeback:0 unstable:0 slab_reclaimable:61724 slab_unreclaimable:39525 mapped:11980 shmem:11196 pagetables:26077 bounce:0 free:65999 free_pcp:149 free_cma:0 [463934.052661] Node 0 active_anon:48182388kB inactive_anon:43872kB active_file:748kB inactive_file:400kB unevictable:5320kB isolated(anon):164kB isolated(file):4kB mapped:47920kB dirty:0kB writeback:0kB shmem:44784kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 45887488kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes [463934.052819] Node 0 DMA free:15896kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [463934.052960] lowmem_reserve[]: 0 2944 48124 48124 48124 [463934.053018] Node 0 DMA32 free:184840kB min:4132kB low:7144kB high:10156kB active_anon:2860688kB inactive_anon:0kB active_file:68kB inactive_file:320kB unevictable:0kB writepending:0kB present:3120640kB managed:3055104kB mlocked:0kB kernel_stack:80kB pagetables:104kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [463934.053178] lowmem_reserve[]: 0 0 45180 45180 45180 [463934.054591] Node 0 Normal free:63496kB min:63428kB low:109692kB high:155956kB active_anon:45321244kB inactive_anon:43872kB active_file:844kB inactive_file:192kB unevictable:5320kB writepending:0kB present:47185920kB managed:46272640kB mlocked:5320kB kernel_stack:4880kB pagetables:104204kB bounce:0kB free_pcp:640kB local_pcp:84kB free_cma:0kB [463934.054752] lowmem_reserve[]: 0 0 0 0 0 [463934.054805] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB [463934.054894] Node 0 DMA32: 106*4kB (UE) 132*8kB (UE) 277*16kB (UME) 276*32kB (UME) 293*64kB (UME) 229*128kB (UME) 150*256kB (UME) 89*512kB (UME) 37*1024kB (UME) 0*2048kB 0*4096kB = 184664kB [463934.055004] Node 0 Normal: 895*4kB (UME) 281*8kB (UME) 451*16kB (UME) 174*32kB (UME) 71*64kB (UME) 172*128kB (UME) 60*256kB (U) 4*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 63604kB [463934.055113] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [463934.055188] 12540 total pagecache pages [463934.055240] 0 pages in swap cache [463934.055291] Swap cache stats: add 0, delete 0, find 0/0 [463934.055346] Free swap = 0kB [463934.055396] Total swap = 0kB [463934.055446] 12580635 pages RAM [463934.055497] 0 pages HighMem/MovableOnly [463934.055549] 244725 pages reserved [463934.055600] 0 pages cma reserved [463934.055650] 0 pages hwpoisoned [463934.055701] Tasks state (memory values in pages): [463934.055754] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [463934.055834] [ 358] 0 358 35198 1090 311296 0 0 systemd-journal [463934.055912] [ 385] 0 385 5805 755 69632 0 -1000 systemd-udevd [463934.055988] [ 543] 106 543 1705 432 53248 0 0 rpcbind [463934.056064] [ 544] 101 544 23269 233 81920 0 0 systemd-timesyn [463934.056140] [ 557] 0 557 4876 412 77824 0 0 systemd-logind [463934.056216] [ 558] 0 558 111585 451 98304 0 0 lxcfs [463934.056291] [ 559] 104 559 2220 474 57344 0 -900 dbus-daemon [463934.056367] [ 560] 0 560 2958 590 65536 0 0 smartd [463934.056442] [ 563] 0 563 25304 531 77824 0 0 zed [463934.056517] [ 564] 0 564 56455 439 77824 0 0 rsyslogd [463934.056594] [ 565] 0 565 535 150 36864 0 -1000 watchdog-mux [463934.056674] [ 566] 0 566 68958 51 81920 0 0 pve-lxc-syscall [463934.056755] [ 618] 0 618 953 81 45056 0 0 lxc-monitord [463934.056834] [ 644] 0 644 1722 61 53248 0 0 iscsid [463934.056909] [ 645] 0 645 1848 1245 53248 0 -17 iscsid [463934.056985] [ 671] 0 671 568 186 45056 0 0 none [463934.057059] [ 706] 0 706 1487 301 49152 0 0 agetty [463934.057135] [ 769] 0 769 220054 359 204800 0 0 rrdcached [463934.057210] [ 810] 0 810 595233 14216 569344 0 0 pmxcfs [463934.057285] [ 838] 0 838 10867 584 77824 0 0 master [463934.057360] [ 845] 0 845 2210 513 53248 0 0 cron [463934.057435] [ 854] 0 854 74880 20443 303104 0 0 pve-firewall [463934.057511] [ 855] 0 855 72532 19885 303104 0 0 pvestatd [463934.057586] [ 907] 0 907 87312 29486 421888 0 0 pvedaemon [463934.057669] [ 915] 0 915 82645 22864 364544 0 0 pve-ha-crm [463934.057745] [ 916] 33 916 87716 29874 430080 0 0 pveproxy [463934.057820] [ 922] 33 922 16912 12090 172032 0 0 spiceproxy [463934.057896] [ 924] 0 924 82534 22696 352256 0 0 pve-ha-lrm [463934.057972] [ 24098] 0 24098 329233 138161 2330624 0 0 kvm [463934.058048] [ 32592] 0 32592 586 385 45056 0 0 qmeventd [463934.058123] [ 19389] 108 19389 10890 550 77824 0 0 qmgr [463934.058198] [ 22457] 0 22457 2327569 1165877 10600448 0 0 kvm [463934.058273] [ 26756] 0 26756 246281 23067 671744 0 0 kvm [463934.058348] [ 25251] 0 25251 21543 53 65536 0 0 pvefw-logger [463934.058424] [ 25271] 33 25271 17023 12051 172032 0 0 spiceproxy work [463934.058501] [ 10928] 0 10928 705830 496279 5001216 0 0 kvm [463934.058576] [ 31928] 0 31928 3962 460 65536 0 -1000 sshd [463934.058650] [ 24561] 0 24561 89522 29800 421888 0 0 pvedaemon worke [463934.058727] [ 25993] 0 25993 89521 29979 421888 0 0 pvedaemon worke [463934.058804] [ 4064] 33 4064 90999 31538 450560 0 0 pveproxy worker [463934.058880] [ 4244] 33 4244 91223 31647 450560 0 0 pveproxy worker [463934.058957] [ 11339] 108 11339 10877 508 77824 0 0 pickup [463934.059033] [ 29288] 0 29288 89474 29793 421888 0 0 pvedaemon worke [463934.059110] [ 30485] 33 30485 90988 31228 450560 0 0 pveproxy worker [463934.059186] [ 31629] 0 31629 4236 897 73728 0 0 sshd [463934.059261] [ 31640] 0 31640 5350 938 86016 0 0 systemd [463934.059337] [ 31641] 0 31641 43009 793 98304 0 0 (sd-pam) [463934.059412] [ 31657] 0 31657 2030 711 53248 0 0 bash [463934.059487] [ 31765] 0 31765 1508 76 49152 0 0 dmesg [463934.059562] [ 32140] 0 32140 10603125 10011755 80732160 0 0 kvm [463934.059637] [ 32220] 0 32220 89522 29764 405504 0 0 task UPID:srv1: [463934.059713] [ 32251] 0 32251 80497 22847 372736 0 0 qm [463934.059788] [ 32252] 0 32252 5805 400 61440 0 0 systemd-udevd [463934.059865] [ 32253] 0 32253 5805 611 61440 0 0 systemd-udevd [463934.059941] [ 32254] 0 32254 5805 399 61440 0 0 systemd-udevd [463934.060017] [ 32255] 0 32255 5805 611 61440 0 0 systemd-udevd [463934.060093] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/qemu.slice/101.scope,task=kvm,pid=32140,uid=0 [463934.060196] Out of memory: Killed process 32140 (kvm) total-vm:42412500kB, anon-rss:40045844kB, file-rss:1156kB, shmem-rss:20kB [463934.428837] oom_reaper: reaped process 32140 (kvm), now anon-rss:0kB, file-rss:16kB, shmem-rss:20kB Kernels version: Linux hostname 5.6.13.a-1-hardened #1 SMP PREEMPT Fri, 15 May 2020 15:25:34 +0000 x86_64 GNU/Linux Linux hostname 5.6.13-arch1-1 #1 SMP PREEMPT Thu, 14 May 2020 06:52:53 +0000 x86_64 GNU/Linux Additional information and things I've tried that doesn't fixed the problem: I've tried both udev and systemd in /etc/mkinitcpio.conf and executed mkinitcpio -P && grub-mkconfig -o /boot/grub/grub.cfg I'm using grub. I've tried changing cpu type to host from kvm64 in proxmox vm configuration. |
This task depends upon
sudo pacman -Suy asp
cd ~ && mkdir build && cd build
asp update linux && asp export linux
asp update linux-hardened && asp export linux-hardened
cd linux
rm config
cp ../linux-hardened/config .
updpkgsums
makepkg -s
Let's see how it behaves with that config :)
After previous commands I've executed these:
sudo pacman -U linux-5.6.13.arch1-1-x86_64.pkg.tar.xz && sudo grub-mkconfig -o /boot/grub/grub.cfg
Then I shutted the vm down, changed ram to 40GB and start the vm, but neither linux-hardened nor linux (custom kernel) boots up. Here is the log from dmesg:
Linux custom kernel:
Linux hostname 5.6.13-arch1-1 #1 SMP PREEMPT Sun, 17 May 2020 16:39:42 +0000 x86_64 GNU/Linux
dmesg booting linux-hardened:
[476172.303635] kvm invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[476172.303719] CPU: 5 PID: 26756 Comm: kvm Tainted: P IO 5.3.18-3-pve #1
[476172.303795] Hardware name: XXX
[476172.303871] Call Trace:
[476172.303928] dump_stack+0x6d/0x9a
[476172.303984] dump_header+0x4f/0x1e1
[476172.304039] oom_kill_process.cold.34+0xb/0x10
[476172.304096] out_of_memory+0x1ad/0x490
[476172.304152] __alloc_pages_slowpath+0xa9d/0xe40
[476172.304210] __alloc_pages_nodemask+0x2df/0x330
[476172.304267] alloc_pages_current+0x81/0xe0
[476172.304324] __page_cache_alloc+0x6a/0xa0
[476172.304381] pagecache_get_page+0xaf/0x2d0
[476172.304437] filemap_fault+0x74e/0xa40
[476172.304494] ? unlock_page_memcg+0x12/0x20
[476172.304550] ? page_add_file_rmap+0x119/0x160
[476172.304608] ? xas_load+0xc/0x80
[476172.304661] ? xas_find+0x17e/0x1b0
[476172.304716] ? filemap_map_pages+0x1ae/0x380
[476172.304774] ext4_filemap_fault+0x31/0x44
[476172.304831] __do_fault+0x3c/0x130
[476172.304886] __handle_mm_fault+0xe66/0x1280
[476172.304943] ? set_fd_set.part.6+0x50/0x50
[476172.305000] handle_mm_fault+0xc5/0x1e0
[476172.305056] __do_page_fault+0x233/0x4c0
[476172.305111] do_page_fault+0x2c/0xe0
[476172.305166] page_fault+0x34/0x40
[476172.305220] RIP: 0033:0x563590043820
[476172.305278] Code: Bad RIP value.
[476172.305331] RSP: 002b:00007ffe7f2c0bd8 EFLAGS: 00010246
[476172.305388] RAX: 0000000000000000 RBX: 00005635909928c0 RCX: 0000000000000000
[476172.305462] RDX: 00000000000000f0 RSI: 000056359062ef41 RDI: 00005635909928c0
[476172.305537] RBP: 000056359062ef41 R08: 0000000000000000 R09: 0000000000000000
[476172.305611] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000000f0
[476172.305685] R13: 00007fc7fd31ea00 R14: 00007ffe7f2c0c4c R15: 0000000000000000
[476172.305761] Mem-Info:
[476172.305817] active_anon:12041090 inactive_anon:10971 isolated_anon:6
active_file:191 inactive_file:161 isolated_file:0
unevictable:1330 dirty:0 writeback:0 unstable:0
slab_reclaimable:62715 slab_unreclaimable:40872
mapped:12005 shmem:11198 pagetables:26074 bounce:0
free:65880 free_pcp:194 free_cma:0
[476172.306031] Node 0 active_anon:48164360kB inactive_anon:43884kB active_file:764kB inactive_file:644kB unevictable:5320kB isolated(anon):24kB isolated(file):0kB mapped:48020kB dirty:0kB writeback:0kB shmem:44792kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 42917888kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[476172.306189] Node 0 DMA free:15896kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[476172.307697] lowmem_reserve[]: 0 2944 48124 48124 48124
[476172.307756] Node 0 DMA32 free:184476kB min:4132kB low:7144kB high:10156kB active_anon:2852944kB inactive_anon:3120kB active_file:296kB inactive_file:112kB unevictable:0kB writepending:0kB present:3120640kB managed:3055104kB mlocked:0kB kernel_stack:176kB pagetables:628kB bounce:0kB free_pcp:92kB local_pcp:16kB free_cma:0kB
[476172.307917] lowmem_reserve[]: 0 0 45180 45180 45180
[476172.307974] Node 0 Normal free:63148kB min:63428kB low:109692kB high:155956kB active_anon:45311236kB inactive_anon:40764kB active_file:864kB inactive_file:964kB unevictable:5320kB writepending:0kB present:47185920kB managed:46272640kB mlocked:5320kB kernel_stack:5088kB pagetables:103668kB bounce:0kB free_pcp:804kB local_pcp:84kB free_cma:0kB
[476172.308137] lowmem_reserve[]: 0 0 0 0 0
[476172.308192] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
[476172.308287] Node 0 DMA32: 348*4kB (UME) 470*8kB (UME) 572*16kB (UME) 424*32kB (UME) 256*64kB (UME) 108*128kB (UE) 94*256kB (ME) 102*512kB (UME) 50*1024kB (UME) 0*2048kB 0*4096kB = 185568kB
[476172.308403] Node 0 Normal: 1199*4kB (UME) 567*8kB (UME) 755*16kB (UME) 353*32kB (UME) 151*64kB (UME) 57*128kB (UE) 62*256kB (UME) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65540kB
[476172.308516] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[476172.308592] 12611 total pagecache pages
[476172.308647] 0 pages in swap cache
[476172.308700] Swap cache stats: add 0, delete 0, find 0/0
[476172.308756] Free swap = 0kB
[476172.308808] Total swap = 0kB
[476172.308860] 12580635 pages RAM
[476172.308912] 0 pages HighMem/MovableOnly
[476172.308966] 244725 pages reserved
[476172.309018] 0 pages cma reserved
[476172.309070] 0 pages hwpoisoned
[476172.309123] Tasks state (memory values in pages):
[476172.309178] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[476172.309261] [ 358] 0 358 37246 1098 327680 0 0 systemd-journal
[476172.309341] [ 385] 0 385 5805 772 69632 0 -1000 systemd-udevd
[476172.309420] [ 543] 106 543 1705 425 53248 0 0 rpcbind
[476172.309498] [ 544] 101 544 23269 644 81920 0 0 systemd-timesyn
[476172.309578] [ 557] 0 557 4876 652 77824 0 0 systemd-logind
[476172.309656] [ 558] 0 558 111585 451 98304 0 0 lxcfs
[476172.309734] [ 559] 104 559 2220 298 57344 0 -900 dbus-daemon
[476172.309812] [ 560] 0 560 2958 492 65536 0 0 smartd
[476172.309890] [ 563] 0 563 25304 531 77824 0 0 zed
[476172.309973] [ 564] 0 564 56455 439 77824 0 0 rsyslogd
[476172.310051] [ 565] 0 565 535 148 36864 0 -1000 watchdog-mux
[476172.310131] [ 566] 0 566 68958 51 81920 0 0 pve-lxc-syscall
[476172.310210] [ 618] 0 618 953 81 45056 0 0 lxc-monitord
[476172.310289] [ 644] 0 644 1722 61 53248 0 0 iscsid
[476172.310366] [ 645] 0 645 1848 1245 53248 0 -17 iscsid
[476172.310444] [ 671] 0 671 568 186 45056 0 0 none
[476172.310521] [ 706] 0 706 1487 301 49152 0 0 agetty
[476172.310599] [ 769] 0 769 220054 360 204800 0 0 rrdcached
[476172.310677] [ 810] 0 810 595233 14648 569344 0 0 pmxcfs
[476172.310754] [ 838] 0 838 10867 644 77824 0 0 master
[476172.310832] [ 845] 0 845 2210 522 53248 0 0 cron
[476172.310910] [ 854] 0 854 74880 20443 303104 0 0 pve-firewall
[476172.310989] [ 855] 0 855 72532 19888 303104 0 0 pvestatd
[476172.311067] [ 907] 0 907 87312 29278 421888 0 0 pvedaemon
[476172.311145] [ 915] 0 915 82645 22542 364544 0 0 pve-ha-crm
[476172.311224] [ 916] 33 916 87716 29874 430080 0 0 pveproxy
[476172.311302] [ 922] 33 922 16912 11981 172032 0 0 spiceproxy
[476172.311380] [ 924] 0 924 82534 22446 352256 0 0 pve-ha-lrm
[476172.311459] [ 24098] 0 24098 329233 137908 2322432 0 0 kvm
[476172.311537] [ 32592] 0 32592 586 385 45056 0 0 qmeventd
[476172.311615] [ 19389] 108 19389 10890 454 77824 0 0 qmgr
[476172.311692] [ 22457] 0 22457 2331667 1165892 10608640 0 0 kvm
[476172.311770] [ 26756] 0 26756 246281 22994 671744 0 0 kvm
[476172.311847] [ 25251] 0 25251 21543 53 65536 0 0 pvefw-logger
[476172.311926] [ 25271] 33 25271 17023 12051 172032 0 0 spiceproxy work
[476172.312005] [ 10928] 0 10928 710713 496463 5058560 0 0 kvm
[476172.312084] [ 31928] 0 31928 3962 613 65536 0 -1000 sshd
[476172.312161] [ 24561] 0 24561 89522 29923 421888 0 0 pvedaemon worke
[476172.312240] [ 25993] 0 25993 89521 30011 421888 0 0 pvedaemon worke
[476172.312319] [ 29288] 0 29288 89474 29844 421888 0 0 pvedaemon worke
[476172.312399] [ 30485] 33 30485 90988 31528 450560 0 0 pveproxy worker
[476172.312478] [ 25254] 108 25254 10877 570 81920 0 0 pickup
[476172.312556] [ 30505] 33 30505 89859 30207 434176 0 0 pveproxy worker
[476172.312635] [ 32060] 33 32060 89897 29967 434176 0 0 pveproxy worker
[476172.312716] [ 14020] 0 14020 4150 767 69632 0 0 sshd
[476172.312794] [ 14029] 0 14029 5351 941 73728 0 0 systemd
[476172.312872] [ 14031] 0 14031 43009 793 98304 0 0 (sd-pam)
[476172.312950] [ 14047] 0 14047 2030 713 53248 0 0 bash
[476172.313027] [ 14093] 0 14093 10619918 10009103 80793600 0 0 kvm
[476172.313104] [ 14134] 0 14134 82117 22987 380928 0 0 qm
[476172.313182] [ 14135] 0 14135 1705 171 57344 0 0 socat
[476172.313259] [ 14138] 0 14138 4323 947 69632 0 0 sshd
[476172.313344] [ 14179] 0 14179 2030 720 53248 0 0 bash
[476172.313422] [ 14183] 0 14183 1508 40 49152 0 0 dmesg
[476172.313500] [ 14258] 0 14258 5805 643 61440 0 0 systemd-udevd
[476172.313578] [ 14259] 0 14259 5805 513 61440 0 0 systemd-udevd
[476172.313658] [ 14260] 0 14260 5805 617 61440 0 0 systemd-udevd
[476172.313737] [ 14261] 0 14261 5805 617 61440 0 0 systemd-udevd
[476172.313816] [ 14262] 0 14262 5805 610 61440 0 0 systemd-udevd
[476172.313895] [ 14263] 0 14263 5805 499 61440 0 0 systemd-udevd
[476172.313979] [ 14264] 0 14264 5805 612 61440 0 0 systemd-udevd
[476172.314058] [ 14265] 0 14265 5805 612 61440 0 0 systemd-udevd
[476172.314137] [ 14266] 0 14266 5805 500 61440 0 0 systemd-udevd
[476172.314216] [ 14267] 0 14267 5805 500 61440 0 0 systemd-udevd
[476172.314295] [ 14268] 0 14268 5805 619 61440 0 0 systemd-udevd
[476172.314373] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/qemu.slice/101.scope,task=kvm,pid=14093,uid=0
[476172.314499] Out of memory: Killed process 14093 (kvm) total-vm:42479672kB, anon-rss:40036264kB, file-rss:128kB, shmem-rss:20kB
[476172.620218] oom_reaper: reaped process 14093 (kvm), now anon-rss:0kB, file-rss:40kB, shmem-rss:20kB
dmesg booting linux (custom kernel):
[476202.579421] pveproxy worker invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[476202.579509] CPU: 4 PID: 30505 Comm: pveproxy worker Tainted: P IO 5.3.18-3-pve #1
[476202.579587] Hardware name: MSI MS-7522/MSI X58 Pro (MS-7522) , BIOS V8.14B9 10/11/2018
[476202.579663] Call Trace:
[476202.579722] dump_stack+0x6d/0x9a
[476202.579778] dump_header+0x4f/0x1e1
[476202.579833] oom_kill_process.cold.34+0xb/0x10
[476202.579890] out_of_memory+0x1ad/0x490
[476202.579946] __alloc_pages_slowpath+0xa9d/0xe40
[476202.580004] __alloc_pages_nodemask+0x2df/0x330
[476202.580062] alloc_pages_current+0x81/0xe0
[476202.580120] __page_cache_alloc+0x6a/0xa0
[476202.580177] pagecache_get_page+0xaf/0x2d0
[476202.580233] filemap_fault+0x74e/0xa40
[476202.580290] ? unlock_page_memcg+0x12/0x20
[476202.580347] ? page_add_file_rmap+0x119/0x160
[476202.580404] ? xas_load+0xc/0x80
[476202.580458] ? xas_find+0x17e/0x1b0
[476202.580513] ? filemap_map_pages+0x1ae/0x380
[476202.580571] ext4_filemap_fault+0x31/0x44
[476202.580627] __do_fault+0x3c/0x130
[476202.580683] __handle_mm_fault+0xe66/0x1280
[476202.580740] handle_mm_fault+0xc5/0x1e0
[476202.580797] __do_page_fault+0x233/0x4c0
[476202.580887] ? kvm_on_user_return+0x6f/0xa0 [kvm]
[476202.580945] do_page_fault+0x2c/0xe0
[476202.581000] page_fault+0x34/0x40
[476202.581055] RIP: 0033:0x55cb2061ce40
[476202.581111] Code: Bad RIP value.
[476202.581162] RSP: 002b:00007ffd44f96760 EFLAGS: 00010206
[476202.581218] RAX: 0000000000000061 RBX: 0000000000000000 RCX: 0000000000000002
[476202.581290] RDX: 000055cb248e97b8 RSI: 000055cb28b80d48 RDI: 000055cb21c2b260
[476202.581362] RBP: 000055cb28b52338 R08: 0000000000000001 R09: 000055cb289c6b30
[476202.581434] R10: 000055cb21c59e30 R11: 0000000000000003 R12: 000055cb28b52338
[476202.581507] R13: 000055cb28b48f10 R14: 000055cb21c2b260 R15: 000055cb28bf57e0
[476202.581587] Mem-Info:
[476202.581663] active_anon:12042855 inactive_anon:10970 isolated_anon:0
active_file:227 inactive_file:72 isolated_file:0
unevictable:1330 dirty:0 writeback:0 unstable:0
slab_reclaimable:62780 slab_unreclaimable:40813
mapped:12072 shmem:11198 pagetables:26069 bounce:0
free:65951 free_pcp:121 free_cma:0
[476202.581848] Node 0 active_anon:48171420kB inactive_anon:43880kB active_file:908kB inactive_file:288kB unevictable:5320kB isolated(anon):0kB isolated(file):0kB mapped:48288kB dirty:0kB writeback:0kB shmem:44792kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 45799424kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[476202.582005] Node 0 DMA free:15896kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[476202.582146] lowmem_reserve[]: 0 2944 48124 48124 48124
[476202.582205] Node 0 DMA32 free:184824kB min:4132kB low:7144kB high:10156kB active_anon:2854312kB inactive_anon:3120kB active_file:288kB inactive_file:0kB unevictable:0kB writepending:0kB present:3120640kB managed:3055104kB mlocked:0kB kernel_stack:160kB pagetables:220kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[476202.582372] lowmem_reserve[]: 0 0 45180 45180 45180
[476202.582430] Node 0 Normal free:63084kB min:63428kB low:109692kB high:155956kB active_anon:45316928kB inactive_anon:40760kB active_file:428kB inactive_file:1112kB unevictable:5320kB writepending:0kB present:47185920kB managed:46272640kB mlocked:5320kB kernel_stack:5136kB pagetables:104056kB bounce:0kB free_pcp:504kB local_pcp:20kB free_cma:0kB
[476202.582594] lowmem_reserve[]: 0 0 0 0 0
[476202.582650] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
[476202.582744] Node 0 DMA32: 361*4kB (UME) 383*8kB (UME) 515*16kB (UME) 435*32kB (UME) 362*64kB (UME) 200*128kB (UME) 96*256kB (UME) 86*512kB (UME) 41*1024kB (UME) 0*2048kB 0*4096kB = 186028kB
[476202.582859] Node 0 Normal: 1252*4kB (UME) 578*8kB (UME) 696*16kB (UME) 316*32kB (UME) 136*64kB (UE) 65*128kB (UME) 66*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64800kB
[476202.582973] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[476202.583049] 12612 total pagecache pages
[476202.583103] 0 pages in swap cache
[476202.583156] Swap cache stats: add 0, delete 0, find 0/0
[476202.583213] Free swap = 0kB
[476202.583266] Total swap = 0kB
[476202.583318] 12580635 pages RAM
[476202.583369] 0 pages HighMem/MovableOnly
[476202.583421] 244725 pages reserved
[476202.583471] 0 pages cma reserved
[476202.583522] 0 pages hwpoisoned
[476202.583574] Tasks state (memory values in pages):
[476202.583628] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[476202.583708] [ 358] 0 358 37246 1099 327680 0 0 systemd-journal
[476202.583785] [ 385] 0 385 5805 844 69632 0 -1000 systemd-udevd
[476202.583862] [ 543] 106 543 1705 418 53248 0 0 rpcbind
[476202.583937] [ 544] 101 544 23269 644 81920 0 0 systemd-timesyn
[476202.584014] [ 557] 0 557 4876 351 77824 0 0 systemd-logind
[476202.584090] [ 558] 0 558 111585 451 98304 0 0 lxcfs
[476202.584166] [ 559] 104 559 2220 521 57344 0 -900 dbus-daemon
[476202.584242] [ 560] 0 560 2958 492 65536 0 0 smartd
[476202.584317] [ 563] 0 563 25304 531 77824 0 0 zed
[476202.584392] [ 564] 0 564 56455 530 77824 0 0 rsyslogd
[476202.584468] [ 565] 0 565 535 330 36864 0 -1000 watchdog-mux
[476202.584581] [ 566] 0 566 68958 51 81920 0 0 pve-lxc-syscall
[476202.584659] [ 618] 0 618 953 81 45056 0 0 lxc-monitord
[476202.584735] [ 644] 0 644 1722 61 53248 0 0 iscsid
[476202.584810] [ 645] 0 645 1848 1245 53248 0 -17 iscsid
[476202.584886] [ 671] 0 671 568 186 45056 0 0 none
[476202.584966] [ 706] 0 706 1487 301 49152 0 0 agetty
[476202.585041] [ 769] 0 769 220054 360 204800 0 0 rrdcached
[476202.585122] [ 810] 0 810 595233 14310 569344 0 0 pmxcfs
[476202.585197] [ 838] 0 838 10867 644 77824 0 0 master
[476202.585273] [ 845] 0 845 2210 522 53248 0 0 cron
[476202.585385] [ 854] 0 854 74880 20443 303104 0 0 pve-firewall
[476202.585497] [ 855] 0 855 72532 19888 303104 0 0 pvestatd
[476202.585572] [ 907] 0 907 87312 29438 421888 0 0 pvedaemon
[476202.585647] [ 915] 0 915 82645 22542 364544 0 0 pve-ha-crm
[476202.585723] [ 916] 33 916 87716 29660 430080 0 0 pveproxy
[476202.585799] [ 922] 33 922 16912 11981 172032 0 0 spiceproxy
[476202.585874] [ 924] 0 924 82534 22764 352256 0 0 pve-ha-lrm
[476202.585950] [ 24098] 0 24098 329233 138160 2326528 0 0 kvm
[476202.586025] [ 32592] 0 32592 586 374 45056 0 0 qmeventd
[476202.586101] [ 19389] 108 19389 10890 454 77824 0 0 qmgr
[476202.586175] [ 22457] 0 22457 2364451 1166020 10694656 0 0 kvm
[476202.586259] [ 26756] 0 26756 246281 22994 671744 0 0 kvm
[476202.586338] [ 25251] 0 25251 21543 53 65536 0 0 pvefw-logger
[476202.586436] [ 25271] 33 25271 17023 12399 172032 0 0 spiceproxy work
[476202.587786] [ 10928] 0 10928 710713 496460 5058560 0 0 kvm
[476202.587865] [ 31928] 0 31928 3962 613 65536 0 -1000 sshd
[476202.587942] [ 24561] 0 24561 89522 29766 421888 0 0 pvedaemon worke
[476202.588022] [ 25993] 0 25993 89521 29754 421888 0 0 pvedaemon worke
[476202.588102] [ 29288] 0 29288 89474 29586 421888 0 0 pvedaemon worke
[476202.588182] [ 30485] 33 30485 90988 31296 450560 0 0 pveproxy worker
[476202.588261] [ 25254] 108 25254 10877 570 81920 0 0 pickup
[476202.588338] [ 30505] 33 30505 89909 30238 434176 0 0 pveproxy worker
[476202.588418] [ 32060] 33 32060 89897 30247 434176 0 0 pveproxy worker
[476202.588498] [ 14020] 0 14020 4150 672 69632 0 0 sshd
[476202.588576] [ 14029] 0 14029 5351 868 73728 0 0 systemd
[476202.588655] [ 14031] 0 14031 43009 793 98304 0 0 (sd-pam)
[476202.588733] [ 14047] 0 14047 2030 716 53248 0 0 bash
[476202.588811] [ 14138] 0 14138 4323 853 69632 0 0 sshd
[476202.588888] [ 14179] 0 14179 2030 720 53248 0 0 bash
[476202.588966] [ 14183] 0 14183 1508 28 49152 0 0 dmesg
[476202.589044] [ 14380] 0 14380 10620814 10012097 80814080 0 0 kvm
[476202.589121] [ 14420] 0 14420 82108 22983 393216 0 0 qm
[476202.589198] [ 14421] 0 14421 1705 155 57344 0 0 socat
[476202.589274] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/qemu.slice/101.scope,task=kvm,pid=14380,uid=0
[476202.589411] Out of memory: Killed process 14380 (kvm) total-vm:42483256kB, anon-rss:40046620kB, file-rss:1748kB, shmem-rss:20kB
[476202.750685] oom_reaper: reaped process 14380 (kvm), now anon-rss:0kB, file-rss:40kB, shmem-rss:20kB
CONFIG_INIT_ON_FREE_DEFAULT_ON=y
which could be tweaked in grub using init_on_free=0.
So right now I can successfully boot both extra/linux-hardened and custom compiled linux kernel with hardened config after I've added init_on_free=0 to /etc/default/grub and generating the main configuration again.
https://patchwork.kernel.org/patch/11017569/
Although I haven't compiled the kernel disabling flag on config file yet, because I'm not going to compile my own kernel and will change grub default commandline if linux-hardened package doesn't change the config file.
Also I don't know if it's practical and correct to change the flag on linux-hardened config file or not because of this problem, but I'm willing to open a pull request on github repo if that's the correct way to do it.
Please let me know what would happen from here too, feel free to close this bug report and thanks.