Arch Linux

> it does not occur on 5.15 linux kernel

Seeing as you're already this far down the rabbit hole, the next step is git bisect [1][2] to hopefully find the upstream commit that caused the regression. Kernel config could also be a factor. Have you compared configs with Ubuntu?

[1] https://wiki.archlinux.org/title/Bisecting_bugs_with_Git
[2] https://docs.kernel.org/admin-guide/bug-bisect.html

Hi, thanks for a quick response. Ubuntu is using a fairly recent linux and that is working fine so I think kernel config if more likely. I'll diff them tomorrow and report back.

There is a huge amount of config parameters. I will start with a diff between *NVME* config params in arch 5.15 and arch (current 6.4):

diff <(grep NVME < linux-5.15.130-1-lts515-kernel-config.txt) <(grep NVME linux-6.4.12-arch1-1-kernel-config.txt)
=============================================================================
1c1
< CONFIG_BLK_DEV_NVME=y
---
> CONFIG_BLK_DEV_NVME=m
2a3,5
> CONFIG_NVMEM_LAYOUT_ONIE_TLV=m
> CONFIG_NVMEM_LAYOUT_SL28_VPD=m
> CONFIG_NVMEM_RAVE_SP_EEPROM=m
5c8,10
< CONFIG_NVME_CORE=y
---
> CONFIG_NVME_AUTH=y
> CONFIG_NVME_COMMON=m
> CONFIG_NVME_CORE=m
11a17
> CONFIG_NVME_TARGET_AUTH=y
18a25
> CONFIG_NVME_VERBOSE_ERRORS=y
=============================================================================
Here are the changes between ubuntu (6.2) and arch 6.4:

diff <(grep NVME < linux-6.2.0-31-generic-kernel-config.txt) <(grep NVME linux-6.4.12-arch1-1-kernel-config.txt)
=============================================================================
2a3,4
> CONFIG_NVMEM_LAYOUT_ONIE_TLV=m
> CONFIG_NVMEM_LAYOUT_SL28_VPD=m
5d6
< CONFIG_NVMEM_SPMI_SDAM=m
17a19
> CONFIG_NVME_TARGET_FCLOOP=m
22a25
> CONFIG_NVME_VERBOSE_ERRORS=y
=============================================================================

For completeness below is analysis of all differences (All kernel configs will be attached):

Since we know that both arch 5.15 and ubuntu 6.2 are working fine, I looked at the common differences between:

1) arch 5.15 and arch current (6.4)
2) ubuntu 6.2 and arch current (6.4)

This was computed in python like this:

config_files = glob.glob('./linux-*.txt')

d = {p: pathlib.Path(p) for p in config_files}

d2 = {s: set(p.read_text().splitlines()) for s, p in d.items()}

diff_between_arch_and_ubuntu = d2['./linux-6.2.0-31-generic-kernel-config.txt'].symmetric_difference(d2['./linux-6.4.12-arch1-1-kernel-config.txt'])

diff_between_arch_5_and_arch_current = d2['./linux-5.15.130-1-lts515-kernel-config.txt'].symmetric_difference(d2['./linux-6.4.12-arch1-1-kernel-config.txt'])

print(dumps(sorted({i.split('=')[0] for i in diff_between_arch_and_ubuntu} & {i.split('=')[0] for i in diff_between_arch_5_and_arch_current}), indent=2))

[
"CONFIG_ADDRESS_MASKING",
"CONFIG_ADE7854",
"CONFIG_ADE7854_I2C",
"CONFIG_ADE7854_SPI",
"CONFIG_ARCH_SUPPORTS_PER_VMA_LOCK",
"CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP",
"CONFIG_AS_GFNI",
"CONFIG_ATH12K",
"CONFIG_ATH12K_DEBUG",
"CONFIG_ATH12K_TRACING",
"CONFIG_BACKLIGHT_KTZ8866",
"CONFIG_BLKDEV_UBLK_LEGACY_OPCODES",
"CONFIG_BLK_CGROUP_PUNT_BIO",
"CONFIG_BLK_DEV_UBLK",
"CONFIG_BLK_MQ_RDMA",
"CONFIG_BLOCK_COMPAT",
"CONFIG_BT_NXPUART",
"CONFIG_CACHEFILES_ONDEMAND",
"CONFIG_CARDMAN_4000",
"CONFIG_CARDMAN_4040",
"CONFIG_CHARGER_RT9467",
"CONFIG_CHARGER_RT9471",
"CONFIG_CONTEXT_TRACKING_USER",
"CONFIG_CROS_EC_UART",
"CONFIG_CRYPTO_ARIA_AESNI_AVX2_X86_64",
"CONFIG_CRYPTO_ARIA_GFNI_AVX512_X86_64",
"CONFIG_CRYPTO_LIB_GF128MUL",
"CONFIG_DAMON_LRU_SORT",
"CONFIG_DAMON_PADDR",
"CONFIG_DAMON_RECLAIM",
"CONFIG_DAMON_SYSFS",
"CONFIG_DEV_DAX_CXL",
"CONFIG_DRM_ACCEL_HABANALABS",
"CONFIG_DRM_ACCEL_IVPU",
"CONFIG_DRM_ACCEL_QAIC",
"CONFIG_DRM_AMD_DC_DCN",
"CONFIG_DRM_AMD_DC_FP",
"CONFIG_DRM_AMD_DC_HDCP",
"CONFIG_DRM_PANEL_AUO_A030JTN01",
"CONFIG_DRM_PANEL_ORISETECH_OTA5601A",
"CONFIG_DRM_SUBALLOC_HELPER",
"CONFIG_DRM_VIRTIO_GPU_KMS",
"CONFIG_EROFS_FS_ONDEMAND",
"CONFIG_EROFS_FS_PCPU_KTHREAD",
"CONFIG_EROFS_FS_PCPU_KTHREAD_HIPRI",
"CONFIG_EROFS_FS_ZIP_LZMA",
"CONFIG_FB_CMDLINE",
"CONFIG_FB_MODE_HELPERS",
"CONFIG_FB_TILEBLITTING",
"CONFIG_FIRMWARE_EDID",
"CONFIG_FPGA_M10_BMC_SEC_UPDATE",
"CONFIG_FW_LOADER_DEBUG",
"CONFIG_FW_UPLOAD",
"CONFIG_GOOGLE_CBMEM",
"CONFIG_GPIO_ELKHARTLAKE",
"CONFIG_GPIO_FXL6408",
"CONFIG_GPIO_REGMAP",
"CONFIG_GPIO_TANGIER",
"CONFIG_GPIO_UCB1400",
"CONFIG_HABANA_AI",
"CONFIG_HAS_IOPORT",
"CONFIG_HID_BPF",
"CONFIG_HID_EVISION",
"CONFIG_HID_SUPPORT",
"CONFIG_HTC_PASIC3",
"CONFIG_HZ",
"CONFIG_HZ_300",
"CONFIG_I2C_CCGX_UCSI",
"CONFIG_I2C_HID",
"CONFIG_I2C_HID_OF",
"CONFIG_IIO_GTS_HELPER",
"CONFIG_INPUT_IBM_PANEL",
"CONFIG_INPUT_VIVALDIFMAP",
"CONFIG_INTEL_BYTCRC_PWRSRC",
"CONFIG_INTEL_IOMMU_PERF_EVENTS",
"CONFIG_INTEL_MENLOW",
"CONFIG_INTEL_SPEED_SELECT_TPMI",
"CONFIG_INTEL_TCC",
"CONFIG_INTEL_TPMI",
"CONFIG_IOASID",
"CONFIG_IPMI_IPMB",
"CONFIG_IP_NF_TARGET_CLUSTERIP",
"CONFIG_IXGB",
"CONFIG_KFENCE_DEFERRABLE",
"CONFIG_KVM_GENERIC_HARDWARE_ENABLING",
"CONFIG_LEDS_BD2606MVV",
"CONFIG_LEDS_MT6370_FLASH",
"CONFIG_LEDS_MT6370_RGB",
"CONFIG_LEDS_TRIGGER_GPIO",
"CONFIG_LEGACY_DIRECT_IO",
"CONFIG_LENOVO_YMC",
"CONFIG_LRU_GEN_ENABLED",
"CONFIG_MAX5522",
"CONFIG_MAX_SKB_FRAGS",
"CONFIG_MCTP_FLOWS",
"CONFIG_MCTP_TRANSPORT_I2C",
"CONFIG_MFD_INTEL_M10_BMC",
"CONFIG_MFD_INTEL_M10_BMC_CORE",
"CONFIG_MFD_INTEL_M10_BMC_PMCI",
"CONFIG_MFD_INTEL_M10_BMC_SPI",
"CONFIG_MICROCHIP_T1S_PHY",
"CONFIG_MLX5_EN_MACSEC",
"CONFIG_MMU_LAZY_TLB_REFCOUNT",
"CONFIG_MODULE_DEBUGFS",
"CONFIG_MODULE_DECOMPRESS",
"CONFIG_MODULE_UNLOAD_TAINT_TRACKING",
"CONFIG_MSI_EC",
"CONFIG_NCN26000_PHY",
"CONFIG_NETFILTER_BPF_LINK",
"CONFIG_NET_CLS_RSVP",
"CONFIG_NET_CLS_RSVP6",
"CONFIG_NET_DSA_MICROCHIP_KSZ_PTP",
"CONFIG_NET_DSA_MSCC_FELIX_DSA_LIB",
"CONFIG_NET_DSA_MSCC_OCELOT_EXT",
"CONFIG_NET_DSA_MT7530_MDIO",
"CONFIG_NET_DSA_MT7530_MMIO",
"CONFIG_NET_DSA_QCA8K_LEDS_SUPPORT",
"CONFIG_NET_HANDSHAKE",
"CONFIG_NET_SCH_ATM",
"CONFIG_NET_SCH_CBQ",
"CONFIG_NET_SCH_DSMARK",
"CONFIG_NET_SCH_MQPRIO_LIB",
"CONFIG_NF_CONNTRACK_OVS",
"CONFIG_NF_FLOW_TABLE_PROCFS",
"CONFIG_NO_HZ_FULL",
"CONFIG_NO_HZ_IDLE",
"CONFIG_NVMEM_LAYOUT_ONIE_TLV",
"CONFIG_NVMEM_LAYOUT_SL28_VPD",
"CONFIG_NVME_VERBOSE_ERRORS",
"CONFIG_NXP_CBTX_PHY",
"CONFIG_PAGE_POOL_STATS",
"CONFIG_PARPORT_AX88796",
"CONFIG_PATA_PARPORT",
"CONFIG_PATA_PARPORT_ATEN",
"CONFIG_PATA_PARPORT_BPCK",
"CONFIG_PATA_PARPORT_BPCK6",
"CONFIG_PATA_PARPORT_COMM",
"CONFIG_PATA_PARPORT_DSTR",
"CONFIG_PATA_PARPORT_EPAT",
"CONFIG_PATA_PARPORT_EPATC8",
"CONFIG_PATA_PARPORT_EPIA",
"CONFIG_PATA_PARPORT_FIT2",
"CONFIG_PATA_PARPORT_FIT3",
"CONFIG_PATA_PARPORT_FRIQ",
"CONFIG_PATA_PARPORT_FRPW",
"CONFIG_PATA_PARPORT_KBIC",
"CONFIG_PATA_PARPORT_KTTI",
"CONFIG_PATA_PARPORT_ON20",
"CONFIG_PATA_PARPORT_ON26",
"CONFIG_PCS_MTK_LYNXI",
"CONFIG_PDA_POWER",
"CONFIG_PDS_CORE",
"CONFIG_PEAQ_WMI",
"CONFIG_PER_VMA_LOCK",
"CONFIG_PINCTRL_METEORLAKE",
"CONFIG_PREEMPT",
"CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT",
"CONFIG_PTP_DFL_TOD",
"CONFIG_R8188EU",
"CONFIG_RCU_BOOST",
"CONFIG_RCU_BOOST_DELAY",
"CONFIG_RCU_LAZY",
"CONFIG_RCU_NOCB_CPU",
"CONFIG_REGULATOR_MAX20411",
"CONFIG_REGULATOR_RT4803",
"CONFIG_REGULATOR_RT5739",
"CONFIG_REMOTE_TARGET",
"CONFIG_ROHM_BU27034",
"CONFIG_RPCSEC_GSS_KRB5_CRYPTOSYSTEM",
"CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1",
"CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2",
"CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_CAMELLIA",
"CONFIG_RTC_DRV_V3020",
"CONFIG_RTW88_8821CS",
"CONFIG_RTW88_8822BS",
"CONFIG_RTW88_8822CS",
"CONFIG_RTW88_SDIO",
"CONFIG_SCHED_MM_CID",
"CONFIG_SCR24X",
"CONFIG_SCSI_UFS_HWMON",
"CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE",
"CONFIG_SENSORS_ACBEL_FSG032",
"CONFIG_SENSORS_MC34VR500",
"CONFIG_SENSORS_MPQ7932",
"CONFIG_SENSORS_MPQ7932_REGULATOR",
"CONFIG_SENSORS_TDA38640",
"CONFIG_SENSORS_TDA38640_REGULATOR",
"CONFIG_SERIAL_8250_DFL",
"CONFIG_SERIAL_8250_PCI1XXXX",
"CONFIG_SERIAL_8250_PCILIB",
"CONFIG_SERIAL_8250_PERICOM",
"CONFIG_SHRINKER_DEBUG",
"CONFIG_SMBFS",
"CONFIG_SMBFS_COMMON",
"CONFIG_SND_AC97_POWER_SAVE_DEFAULT",
"CONFIG_SND_CTL_FAST_LOOKUP",
"CONFIG_SND_CTL_INPUT_VALIDATION",
"CONFIG_SND_HDA_POWER_SAVE_DEFAULT",
"CONFIG_SND_SOC_AW88395",
"CONFIG_SND_SOC_AW88395_LIB",
"CONFIG_SND_SOC_CS35L56",
"CONFIG_SND_SOC_CS35L56_I2C",
"CONFIG_SND_SOC_CS35L56_SDW",
"CONFIG_SND_SOC_CS35L56_SHARED",
"CONFIG_SND_SOC_CS35L56_SPI",
"CONFIG_SND_SOC_CS42L42_SDW",
"CONFIG_SND_SOC_IDT821034",
"CONFIG_SND_SOC_MAX98363",
"CONFIG_SND_SOC_PEB2466",
"CONFIG_SND_SOC_RT712_SDCA_DMIC_SDW",
"CONFIG_SND_SOC_RT712_SDCA_SDW",
"CONFIG_SND_SOC_SMA1303",
"CONFIG_SND_SOC_SOF_HDA_MLINK",
"CONFIG_SNET_VDPA",
"CONFIG_SOC_BUS",
"CONFIG_SOUNDWIRE_AMD",
"CONFIG_SPI_AX88796C_COMPRESSION",
"CONFIG_SPI_NXP_FLEXSPI",
"CONFIG_SRCU",
"CONFIG_SSIF_IPMI_BMC",
"CONFIG_STACK_VALIDATION",
"CONFIG_STEAM_FF",
"CONFIG_SUNRPC_DISABLE_INSECURE_ENCTYPES",
"CONFIG_SYNCLINK_CS",
"CONFIG_SYSFB_SIMPLEFB",
"CONFIG_SYSTEM_BLACKLIST_AUTH_UPDATE",
"CONFIG_THERMAL_ACPI",
"CONFIG_TICK_CPU_ACCOUNTING",
"CONFIG_TI_ADS1100",
"CONFIG_TI_ADS7924",
"CONFIG_TI_LMP92064",
"CONFIG_TI_TMAG5273",
"CONFIG_TOOLS_SUPPORT_RELR",
"CONFIG_TOUCHSCREEN_NOVATEK_NVT_TS",
"CONFIG_TOUCHSCREEN_UCB1400",
"CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS",
"CONFIG_TRANSPARENT_HUGEPAGE_MADVISE",
"CONFIG_TRUSTED_KEYS_TEE",
"CONFIG_TYPEC_MUX_GPIO_SBU",
"CONFIG_UCB1400_CORE",
"CONFIG_USB_FTDI_ELAN",
"CONFIG_USB_U132_HCD",
"CONFIG_USER_EVENTS",
"CONFIG_UVC_COMMON",
"CONFIG_VHOST_TASK",
"CONFIG_VIDEOBUF_VMALLOC",
"CONFIG_VIDEO_CMDLINE",
"CONFIG_VIDEO_IMX296",
"CONFIG_VIDEO_M5MOLS",
"CONFIG_VIDEO_MT9M032",
"CONFIG_VIDEO_MT9T001",
"CONFIG_VIDEO_NOON010PC30",
"CONFIG_VIDEO_OV8858",
"CONFIG_VIDEO_S5K6AA",
"CONFIG_VIDEO_SR030PC30",
"CONFIG_VIDEO_VS6624",
"CONFIG_VIRTIO_PCI_LIB_LEGACY",
"CONFIG_VIRT_CPU_ACCOUNTING",
"CONFIG_VIRT_CPU_ACCOUNTING_GEN",
"CONFIG_VMGENID",
"CONFIG_W1_MASTER_DS1WM",
"CONFIG_WPCM450_SOC",
"CONFIG_X86_AMD_PSTATE_UT",
"CONFIG_X86_KERNEL_IBT",
"CONFIG_XFS_DRAIN_INTENTS",
"CONFIG_XFS_SUPPORT_ASCII_CI",
"CONFIG_XILINX_XDMA",
"CONFIG_ZRAM_DEF_COMP",
"CONFIG_ZRAM_DEF_COMP_LZORLE",
"CONFIG_ZRAM_DEF_COMP_ZSTD",
"CONFIG_ZRAM_MULTI_COMP",
"CONFIG_ZSMALLOC_CHAIN_SIZE",
"CONFIG_ZSMALLOC_STAT",
"CONFIG_ZSWAP_COMPRESSOR_DEFAULT",
"CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD",
"CONFIG_ZSWAP_ZPOOL_DEFAULT",
"CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC"
]

Comparing different kernels is not useful. Too many variables. Can you specify the fs as well? And have you tested with incoming 6.5 ?

I am seeing the exact same issues on https://aur.archlinux.org/packages/linux-mainline (linux 6.5), benchmark will be attached.

Regarding fs, I am using EXT4 on arch. All the benchmarks provided so far have been run on root EXT4 volume.

On ubuntu I am using BTRFS (for root volume as well).

Strangely the performance of the BTRFS volume is *very* poor (around 1/2 of what I was expecting) but *consistent* across ubuntu and arch. I am attaching 3 benchmarks run on different kernels on the BTRFS root volume used by ubuntu. All of them have 1200-1300MB/s speed. The LVM LV used by ubuntu is a simple linear volume. The performance should be similar to running the benchmark on the underlying device (maybe with some overhead of running LVM and BTRFS) but it's not. I am getting 2500-3000MB/s when running the benchmark directly on the PV.

I don't think this type of comparison is distibution specific but more an academic one for ML of kernel. Certainly not a bug until you prove that with same kernel version and different config (this vs that) that Arch uses you see degrated performance - otherwise this is not a bug for Archlinux. At least my 2 cents.

I don't know whether it is a bug or a configuration thing. It's beyond the scope of my expertise. It is entirely possible that this is not a bug in arch linux. I am raising an issue in performance that I have observed. The performance degradation is real and it is measurable and it doesn't occur in every arch kernel (5.15 doesn't have this issue). So I am not sure if this can be labelled as academic. But of course it is entirely up to you whether this issue will be acknowledged or not.