Community Packages

Please read this before reporting a bug:
http://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in Unsupported. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#47303 - [lxc] [systemd] service file does not specify TasksMax which leads to quick exhaustion of pids.

Attached to Project: Community Packages
Opened by Jakub Sztandera (Kubuxu) - Tuesday, 08 December 2015, 22:00 GMT
Last edited by Sergej Pupykin (sergej) - Wednesday, 09 March 2016, 19:31 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Thomas Bächler (brain0)
Sergej Pupykin (sergej)
Dave Reisner (falconindy)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 9
Private No

Details

Description:
If LXC is run from systemd service then it runs in its cgroup.
Default limit of 512 pids if too low for most use-cases.
Fix is to assing:

[Service]
TasksMax=infinity

in service files.

Additional info:
* systemd 228-3
* lxc 1:1.1.5-1


Steps to reproduce:
1. Start container.
2. Start many threads: for i in {1..1000}; do sleep 1000& done;
3. Look and quirks in the console:

bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable
bash: let: detached=: syntax error: operand expected (error token is "=")
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable
bash: let: detached+=: syntax error: operand expected (error token is "+=")
bash: fork: retry: No child processes
bash: fork: retry: No child processes
This task depends upon

Closed by  Sergej Pupykin (sergej)
Wednesday, 09 March 2016, 19:31 GMT
Reason for closing:  Fixed
Comment by Jakub Sztandera (Kubuxu) - Tuesday, 08 December 2015, 22:37 GMT
Or not, it is slice/socping problem.
Comment by Jakub Sztandera (Kubuxu) - Wednesday, 09 December 2015, 01:06 GMT
LXC uses always "init.scope" which breaks systemd.

https://github.com/lxc/lxc/blob/master/src/lxc/cgroup.c#L198
Comment by Jakub Sztandera (Kubuxu) - Thursday, 10 December 2015, 09:11 GMT Comment by Candid Dauth (cdauth) - Friday, 15 January 2016, 19:34 GMT
This applies to docker.service. See  FS#47662  and http://unix.stackexchange.com/q/253903/59955.
Comment by vnoel (kaouete) - Saturday, 16 January 2016, 16:17 GMT
Hi, this problem also manifests directly for the user if he needs to run applications using many threads (i.e., the solution is not to change the service file but the value of UserTaskMax in logind.conf).
I reported the problem on the forum https://bbs.archlinux.org/viewtopic.php?id=207255 and then at  FS#47787  but was asked to report it here instead.

I'm wondering if the default value for logind.conf's UserTaskMax of 4096 is not too small.
I propose to up it a bit, I don't know what is the perfect value, but I think that even for a normal user that is also a developer, for example, you can run into this limit quite rapidly. I myself experienced it during development when running tests on Java code that were using many threads.
Also, I don't see the reason for limiting too much this value except in multi-user settings, and it would make sense that the default behaviour is 1) for a one-user installation, which I think is one of the main use of archlinux, and 2) closer to the system's behaviour before this problem appeared (before linux 4.3 apparently).

In that case, something like 32000 could be a good value since it is close to the default max number of user processes according to ulimit (-u).
Comment by Candid Dauth (cdauth) - Sunday, 17 January 2016, 17:48 GMT
I have opened a pull request for the example systemd files provided with docker: https://github.com/docker/docker/pull/19391
I guess the systemd files of the Arch Linux package need to be adjusted.
Comment by zless (roentgen) - Sunday, 17 January 2016, 17:56 GMT
This is not related to the service file. I start the containers manually and still hit the limit. As someone else said above "LXC uses always "init.scope" which breaks systemd."
Comment by Candid Dauth (cdauth) - Sunday, 17 January 2016, 18:29 GMT
roentgen: systemd also applies a limit of 4096 to login shells, using UserTasksMax in logind.conf.

I don’t know what the thing about the “init.scope” is supposed to mean. At least for docker containers, it looks like the systemd task limit is applied correctly.
Comment by BoBeR182 (BoBeR182) - Wednesday, 20 January 2016, 08:48 GMT
Bug also effects many java apps, including Freenet https://aur.archlinux.org/packages/freenet/

This should be tagged under linux kernel?
Comment by Candid Dauth (cdauth) - Wednesday, 20 January 2016, 14:30 GMT
BoBeR182: I wouldn’t say that this issue is related to the linux kernel. It is an issue of the individual packages that provide a systemd service file, but do not yet set the TasksMax attribute yet despite needing more than 512 processes/threads. If people consider the limit of 4096 tasks per use too low, I think this is an issue that should be fixed in upstream systemd.
Comment by Jakub Sztandera (Kubuxu) - Wednesday, 20 January 2016, 15:35 GMT
Problem is that LXC (and possibly others) are inserted into cgroups tree two times. Once under their designated slice (that max threads can be expanded) and once under init.scope (which is limited to 512 IIRC).
Comment by vnoel (kaouete) - Wednesday, 20 January 2016, 15:41 GMT
FYI, the problem with logind as a user is fixed in the development version of systemd (see https://github.com/systemd/systemd/pull/1987) but until it is released, the user is limited to 512 processes apparently...
Comment by Candid Dauth (cdauth) - Wednesday, 20 January 2016, 20:53 GMT
kaouete: As far as I understand this, it applies to services started in the user scope. So logind has the limit of 4096, and services started with systemctl --user have a default limit of 512, just like any other services. It can be modified by using the TasksMax directive. In the development version, user scope services don't have a default task limit anymore, as the number of tasks is limited per user anyways.

Kubuxu: I don't know how lxc and docker relate to each other, as far as I know docker uses lxc. At least docker doesn't seem to have the problem you describe.
Comment by vnoel (kaouete) - Thursday, 21 January 2016, 08:39 GMT
cdauth: that's not what I am talking about: I'm talking about running programs by hand in a terminal in my user session!

The thing is: by default the limit is 512 because there is a bug in systemd that was fixed but not released (see https://github.com/systemd/systemd/pull/1987).
As for the default when this bug is fixed (which is 4096), I agree it should either be configured by myself or fixed upstream.

I tried to create another issue ( FS#47787 ) but it was refused to me even though I think it does not make sense to report these problems here.

Comment by vnoel (kaouete) - Thursday, 21 January 2016, 08:44 GMT
FYI, I've asked for reopening  FS#47787  in order not to pollute this issue with a different problem.
Comment by BoBeR182 (BoBeR182) - Friday, 22 January 2016, 18:43 GMT Comment by Candid Dauth (cdauth) - Saturday, 23 January 2016, 15:34 GMT
So, to summarize, there are 4 issues being discussed about here:

1. The systemd service file provided with the Arch lxc package should have TasksMax increased. This is what this issue is about.
2. The TasksMax attribute does not work properly for lxc, because lxc containers are inserted into the init.scope cgroup in addition to their designated cgroup. This is being dealt with in lxc upstream: https://github.com/lxc/lxc/issues/713
3. The UserTasksMax default limit of 4096 is too low for some use cases. This is being dealt with in systemd upstream and is being increased to 12288: https://github.com/systemd/systemd/issues/2388. It is additionally being discussed in  FS#47787 .
4. The systemd service file provided with the Arch docker package should have TasksMax increased. In docker upstream the limit has been increased in the example service files: https://github.com/docker/docker/pull/19391. I have opened a separate bug report  FS#47872  to increase the value in the Arch package.
Comment by Sergej Pupykin (sergej) - Wednesday, 09 March 2016, 19:30 GMT
I've added TasksMax=32768 into lxc{,-auto}.service
Other issues were fixed or upstream as I understand

Loading...