FS#47787 - [systemd] USerTaskMax in logind.conf has a too small value

Attached to Project: Arch Linux
Opened by vnoel (kaouete) - Saturday, 16 January 2016, 15:56 GMT
Last edited by Dave Reisner (falconindy) - Saturday, 13 February 2016, 13:28 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Thomas Bächler (brain0)
Dave Reisner (falconindy)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Hi, the symptoms of this problem have been reported around here in #47662 and #47303 but in these bug report, the problem was the settings of some systemd units.

The problem also manifests directly for the user if he needs to run applications using many threads.
I reported the problem on the forum (https://bbs.archlinux.org/viewtopic.php?id=207255) and it led me to the previously cited bug report.

I'm wondering if the default value for logind.conf's UserTaskMax of 4096 is not too small.
I propose to up it a bit, I don't know what is the perfect value, but I think that even for a normal user that is also a developer for example, you can run into this limit quite rapidly. I myself experienced it during development when running tests on Java code that were using many threads.
Also, I don't see the reason for limiting too much this value except in multi-user settings, and it would make sense that the default behaviour is 1) for a one-user installation, which I think is one of the main use of archlinux, and 2) closer to the previous behaviour before this problem appeared.

In that case, something like 32000 could be a good value since it is close to the default max number of user processes according to ulimit (-u).

Thanks :)
This task depends upon

Closed by  Dave Reisner (falconindy)
Saturday, 13 February 2016, 13:28 GMT
Reason for closing:  Fixed
Additional comments about closing:  Systemd 229
Comment by vnoel (kaouete) - Thursday, 21 January 2016, 16:24 GMT
The description of this issue is:
- unclear: my problem is not about systemd unit that are misconfigured as in  FS#47303 , but a problem with the user session itself.
- incorrect: my problem is not caused by a misconfiguration (as I previously thought, hence the incorrect description) but by a bug in systemd that was fixed upstream (https://github.com/systemd/systemd/pull/1987) but not yet released. The result is that the user session is limited to 512 threads instead of the configured limit of 4096.

Hence, I really think it would be easier to deal with that in a separate issue instead of drowning my comments in the middle of another discussion in  FS#47303 . The source and the solution of the problem has nothing to do with this other bug :)

Thanks!
Comment by BoBeR182 (BoBeR182) - Friday, 22 January 2016, 18:42 GMT Comment by vnoel (kaouete) - Saturday, 23 January 2016, 15:40 GMT
Yes but there is also a bug as explained in my first comment that makes UserTaskMax not being honoured (we are currently limited to 512 thread for a user!!) and that could be fixed in arch for now. The fix is the pull request cited in my first comment.
Comment by Candid Dauth (cdauth) - Saturday, 23 January 2016, 17:20 GMT
kaouete: I cannot confirm that UserTaskMax is not being honoured. Running the program from http://stackoverflow.com/questions/3521485/nptl-caps-maximum-threads-at-65528 lets me create 2925 threads from a user session, and "systemctl status user-1001.slice" shows "Tasks: 1158 (limit: 4096)", so both values about add up.

Like I already wrote before, I think the pull request you are talking about refers to services started using "systemctl --user" (see "systemctl status user@1001.service" if your uid is 1001). Right now, there is a per-user limit of 512 for such services, which is removed in the pull request.
Comment by vnoel (kaouete) - Saturday, 23 January 2016, 17:24 GMT
OK, thanks, I must be completely confused by systemd :)
So we can simply wait for upstream then, I will test this script too on Monday to double check!
Comment by vnoel (kaouete) - Sunday, 24 January 2016, 21:27 GMT
That is strange, I just tested and I could create only 385 threads and "systemctl status user-1001.slice" shows Tasks: 279 (limit: 4096)
Comment by Candid Dauth (cdauth) - Sunday, 24 January 2016, 21:32 GMT
Did you verify that your user has the id 1001? Did you check if there are any restrictive ulimits in place?
Comment by vnoel (kaouete) - Monday, 25 January 2016, 08:29 GMT
So, yes I checked for the right user, and ulimit returns:
➜ ~ ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 31461
-n: file descriptors 1024
-l: locked-in-memory size (kbytes) 64
-v: address space (kbytes) unlimited
-x: file locks unlimited
-i: pending signals 31461
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0
-N 15: unlimited

I understand your explanation about the commit I referenced being for user services units, systemd is quite complex so I easily get confused with it :)
Still, you agree the results I'm seeing right now are strange… I don't think I have touched anything…
Comment by vnoel (kaouete) - Monday, 25 January 2016, 08:32 GMT
And by the way, using linux-lts and changing nothing else, I can create up to 31101, which is coherent with ulimit output I think...
Comment by Dan Ziemba (zman0900) - Monday, 25 January 2016, 18:44 GMT
This commit seems to be the one to fix the problem for me: https://github.com/systemd/systemd/commit/6098bb0dc31b1c472ab868088d04234352c44b75

I manually added "TasksMax=infinity" to the end of my /usr/lib/systemd/system/user@.service file and after a reboot everything seems to be working normally. The code for creating threads from stackoverflow now allows me to create about 2700 threads when `systemctl status user-1000.slice` shows about 1400 tasks. Previously I could only create about 300 threads in this situation, which was leading to issues running unit tests for my java code and forking failures when trying to compile a kernel.

Maybe this commit could be added to the current systemd package until the next upstream release is ready?
Comment by Candid Dauth (cdauth) - Monday, 25 January 2016, 18:56 GMT
kaouete and zman0900: When you run systemctl status without any arguments, you can see the whole tree of services. For me, the hierarchy looks like this:
user.slice (no limit)
-> user-1001.service (limit 4096)
---> session-c6.scope (no limit)
-----> (Here are all the programs that I am running in my session)
---> user@1001.service (limit 512)
-----> (Here are all the services that are started in user scope, such as dbus and pulseaudio)

Does the tree look differently for any of you? Are the programs that you start listed under the session scope?

kaouete: In the ulimits that you post, it seems like the number of file descriptors is quite low. Some people say that this limit also applies to the number of threads that can be opened, although I cannot find any reliable sources on this. Maybe you could try if increasing it changes anything?
Comment by vnoel (kaouete) - Monday, 25 January 2016, 19:18 GMT
cdauth, I have the same tree BUT there is indeed something strange: the terminal I'm running (as well as the systemctl status call) is listed under dbus.service which itself is listed under user@1000.service!

That is maybe caused by gnome and the way it launch applications then?
That's really strange because for example, I launch firefox in the same way (using the normal gnome launcher), and it is listed under session-c2.scope...
Comment by Dan Ziemba (zman0900) - Monday, 25 January 2016, 20:59 GMT
Mine doesn't show the limits in the output like that. I am running gnome too, and I noticed that things I have started from gnome-terminal are all running under dbus.service. That's interesting since everything I've had problems with has been ran from gnome-terminal.

/
├─init.scope
│ └─1 /sbin/init
├─machine.slice
│ └─machine-qemu\x2dgames.scope
│ └─1524 /usr/sbin/qemu-system-x86_64 -name games .....
├─system.slice
....
└─user.slice
├─user-1000.slice
│ ├─user@1000.service
│ │ ├─dbus.service
│ │ │ ├─ 5433 systemctl status
│ │ │ ├─ 5434 bash /usr/bin/vimpager
Comment by Candid Dauth (cdauth) - Monday, 25 January 2016, 22:47 GMT
Wow, interesting, that certainly explains the issue. I am running KDE myself.

I wonder if it should be considered a bug that Gnome runs the terminal in that scope, but I guess with the pull request from https://github.com/systemd/systemd/pull/1987, it doesn’t actually make a difference anymore.

Regarding the issue: In upstream, the default per-user limit has been increased to 12288 (see https://github.com/systemd/systemd/issues/2388).
Comment by vnoel (kaouete) - Tuesday, 26 January 2016, 08:29 GMT
I reported the bug to gnome in case it has other implications :)

https://bugzilla.gnome.org/show_bug.cgi?id=761121
Comment by Llewelyn Trahaearn (llewelyn) - Sunday, 31 January 2016, 04:07 GMT
I've assembled a build of systemd 228 including the proposed changes to the defualt user limits from upstream using the following as this issue was giving me more than a little grief. https://gist.github.com/WoefulDerelict/647ba2d3656d6dfc167a
Comment by Catalin Hritcu (catalin_hritcu) - Sunday, 31 January 2016, 14:15 GMT
Llewelyn's package makes this problem go away for me too. Given the huge impact of this problem, is there any chance that the arch package can include these patches before an upstream release?
Comment by Aldrik (N30N) - Sunday, 07 February 2016, 00:37 GMT
Thanks for the fix Llewelyn. This is quite a major issue, hope it get included in core.
Comment by Llewelyn Trahaearn (llewelyn) - Sunday, 07 February 2016, 00:46 GMT
No problem. I'll do my best to keep the PKGBUILD linked here up to date with the 228 revisions in core until those fixes are included there or systemd 229 with the updated code is released. I've already mirrored the VirtualBox fix in 228-4.

Loading...