FS#51970 - [linux] Writing is very slow on a F2FS root partition

Attached to Project: Arch Linux
Opened by UnicornDarkness (Xorg) - Saturday, 26 November 2016, 18:21 GMT
Last edited by Toolybird (Toolybird) - Sunday, 13 November 2022, 20:20 GMT
Task Type Bug Report
Category Kernel
Status Closed
Assigned To Tobias Powalowski (tpowa)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

Description:
When I run 'pacman -Syu', it takes a very long time to update packages: the progression bar seems frozen.
This issue appears when Arch Linux is installed on a F2FS partition. On Ext4, no troubles.

For comparison:
- updating a kernel takes about 30 seconds on my VM (the virtual drive is located on a Ext4 partition on a HDD).
- updating a kernel takes more than 3 minutes on my host system (F2FS partition on a SSD).
VM on a HDD is faster than host on a SSD. *Cough*


Additional info:
* pacman 5.0.1
* linux-ck-sandybridge 4.8.11
* linux-lts 4.4.34
* f2fs-tools 1.7.0
* F2FS.txt and EXT4.txt files are attached for comparison (having two installed kernels on host isn't performance bottleneck)


Steps to reproduce:
Updating packages by using Pacman.
   F2FS.txt (2.8 KiB)
   EXT4.txt (2 KiB)
This task depends upon

Closed by  Toolybird (Toolybird)
Sunday, 13 November 2022, 20:20 GMT
Reason for closing:  Fixed
Additional comments about closing:  Reporter says "Old bug, cannot reproduce it with another computer."
Comment by Doug Newgard (Scimmia) - Sunday, 27 November 2016, 16:51 GMT
Without telling us what is taking the time, this is kind of useless.

FWIW, I have no problem with this with a F2FS USB installation.
Comment by UnicornDarkness (Xorg) - Sunday, 27 November 2016, 16:55 GMT
The update process take time, on lines like that:
(1/2) upgrading linux-ck-sandybridge [---------------------------] 100%

The package extraction is very slow, the progress bar is frozen. After time, it progresses a bit, and then look like still frozen.
Comment by UnicornDarkness (Xorg) - Sunday, 27 November 2016, 17:32 GMT
When using the --debug option, the slowdown appears on 'debug: extracting [...] lines'.

I'm looking for tuning F2FS mount options, like the garbage collector.
Comment by Doug Newgard (Scimmia) - Sunday, 27 November 2016, 17:38 GMT
If you just take the package file and extract it with bsdtar, does it take the same amount of time?
Comment by UnicornDarkness (Xorg) - Sunday, 27 November 2016, 17:55 GMT
Extraction into Ext4 partition (HDD): tar xf /var/cache/pacman/pkg/linux-ck-sandybridge-4.8.11-1-x86_64.pkg.tar.xz 3,90s user 0,34s system 85% cpu 4,983 total
Extraction into Ext4 partition (SSD): tar xf /var/cache/pacman/pkg/linux-ck-sandybridge-4.8.11-1-x86_64.pkg.tar.xz 3,00s user 0,35s system 106% cpu 3,139 total
Extraction into F2FS partition (SSD): tar xf /var/cache/pacman/pkg/linux-ck-sandybridge-4.8.11-1-x86_64.pkg.tar.xz 4,79s user 0,43s system 13% cpu 37,773 total

The /var/cache is located on the same SSD. I think it is a write performance issue with F2FS.
As you can see, on the SSD, the extraction time is ~12 higher on the F2FS partition.

Maybe mount options?
- type f2fs (rw,noatime,background_gc=off,user_xattr,acl,inline_data,extent_cache,active_logs=6)
- type ext4 (rw,relatime,commit=60,data=writeback)
Comment by Doug Newgard (Scimmia) - Sunday, 27 November 2016, 18:02 GMT
Well I specifically said bsdtar because it uses libarchive, same as pacman. Your results, though, prove that it not only has nothing to do with pacman but nothing to do with libarchive, so yeah, looking like a f2fs issue.
Comment by UnicornDarkness (Xorg) - Sunday, 27 November 2016, 18:38 GMT
Yes, that's true, it's more probably a kernel/F2FS issue. That's because only Pacman does write on my root partition. :$

Extraction into Ext4 partition (SSD): bsdtar -x -J -f /var/cache/pacman/pkg/linux-ck-sandybridge-4.8.11-1-x86_64.pkg.tar.xz 3,07s user 0,07s system 94% cpu 3,339 total
Extraction into F2FS partition (SSD): bsdtar -x -J -f /var/cache/pacman/pkg/linux-ck-sandybridge-4.8.11-1-x86_64.pkg.tar.xz 4,45s user 0,16s system 15% cpu 30,526 total

This is with background_gc=on. So the garbage collector is not the source of this issue.
Comment by UnicornDarkness (Xorg) - Wednesday, 30 November 2016, 19:08 GMT
So I switch back my root partition to Ext4. Now updates are really faster. :)
Comment by Gonzo (Gonzo_2028) - Sunday, 23 August 2020, 06:25 GMT
This bug still exists and it makes f2f2 pretty unusable as root-partition-fylesystem.

A simple update by pacman with 50 packages takes 30-60 Minutes while the system becomes unusable and hangs.
doing "sudo fstrim -v /" while that immediately eliminates the "hang" but the pacman stuff stays very slow.

2TB NVME PCIE 4 drive with 5GB/sec.

Are there any mount options i could try to get around this?
Beside of this bug, the system and all the apps are way faster and quicker in launch times. I like it and i really want to use it.
Comment by UnicornDarkness (Xorg) - Sunday, 13 November 2022, 14:28 GMT
Since I changed computer with a new SSD (ADATA SX8200PNP 1TB), I cannot reproduce this issue.

FYI, here are my mount options in /etc/fstab:
LABEL=ArchLinux / f2fs rw,nodiscard,relatime,lazytime,background_gc=on,no_heap,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,fsync_mode=posix 0 1

Loading...