FS#59855 - [apache] low task limit

Attached to Project: Arch Linux
Opened by Mark Nottingham (mnot) - Wednesday, 29 August 2018, 05:16 GMT
Last edited by Doug Newgard (Scimmia) - Thursday, 06 September 2018, 04:00 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To No-one
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

httpd sometimes wedges for me, with the following in error.log:

[Wed Aug 29 02:00:06.143339 2018] [mpm_event:alert] [pid 20043:tid 139622617827072] (11)Resource temporarily unavailable: AH03104: apr_thread_create: unable to create worker thread

Doing some looking around, I suspect it's related to a task limit as per:
https://unix.stackexchange.com/a/255603

... and indeed, my httpd service right after startup shows:

# systemctl status httpd
httpd.service - Apache Web Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-08-13 23:30:07 UTC; 2 weeks 1 days ago
Main PID: 18540 (httpd)
Tasks: 511 (limit: 560)
Memory: 87.3M

... which seems quite close to the limit. Could we get this bumped up or unlimited in httpd.service? I can modify it myself, of course, but I suspect many people are going to run with more processes + threads.

Additional info:
* apache 2.4.34-1

Steps to reproduce:
This task depends upon

Closed by  Doug Newgard (Scimmia)
Thursday, 06 September 2018, 04:00 GMT
Reason for closing:  Not a bug
Comment by loqs (loqs) - Wednesday, 29 August 2018, 13:12 GMT
What is the output of `systemctl show -p TasksMax httpd.service` ?
Comment by Mark Nottingham (mnot) - Wednesday, 29 August 2018, 23:24 GMT
TasksMax=560
Comment by loqs (loqs) - Wednesday, 29 August 2018, 23:37 GMT
As the unit does not set TasksMax I would expect it to be inheriting the value from DefaultTaskmax
Which can be set in /etc/systemd/system.conf
systemctl show -p DefaultTasksMax
From the man 5 systemd-system.conf
DefaultTasksMax=
Configure the default value for the per-unit TasksMax= setting. See
systemd.resource-control(5) for details. This setting applies to
all unit types that support resource control settings, with the
exception of slice units. Defaults to 15%, which equals 4915 with
the kernel's defaults on the host, but might be smaller in OS
containers.
Comment by Mark Nottingham (mnot) - Wednesday, 29 August 2018, 23:52 GMT
Fair enough; I've already modified my httpd unit; just thought it might bite other people.

Thanks,
Comment by loqs (loqs) - Thursday, 30 August 2018, 10:31 GMT
I am trying to understand where the 560 value came from rather than 4915 that I expected.
Comment by Mark Nottingham (mnot) - Sunday, 02 September 2018, 00:03 GMT
# systemctl show -p DefaultTasksMax
DefaultTasksMax=560

in /etc/systemd-system.conf it's still the stock:

#DefaultTasksMax=15%

This is on a VM at Rackspace, so it's working as designed, I suppose. Seems like a bit of a footgun to me.
Comment by loqs (loqs) - Sunday, 02 September 2018, 00:35 GMT
Is that a custom kernel or is the service being run in some sort of container in the VM or I would have expected the value to still be 4915.
Comment by Mark Nottingham (mnot) - Sunday, 02 September 2018, 00:38 GMT
It's in a VM at Rackspace

# uname -a
Linux cloud.mnot.net 4.16.11-1-ARCH #1 SMP PREEMPT Tue May 22 21:40:27 UTC 2018
Comment by loqs (loqs) - Sunday, 02 September 2018, 01:47 GMT
You can trace what systemd is doing to set the default from https://github.com/systemd/systemd/blob/de7436b02badc82200dc127ff190b8155769b8e7/src/core/main.c#L1955
Seems to be a Rackspace change that is causing systemd to generate a lower limit.
Edit:
Has sysctl kernel.threads-max been dropped from 126085?
Comment by Mark Nottingham (mnot) - Sunday, 02 September 2018, 02:55 GMT
Working through system_tasks_max <https://github.com/systemd/systemd/blob/be0135c252cc1705ec5f302622f2eaa3ad831a9f/src/basic/util.c#L467>:

- /proc/sys/kernel/pid_max is 32768
- /proc/sys/kernel/threads-max is 3737
- therefore procfs_tasks_get_limit returns 3737
- /sys/fs/cgroup/pids/system.slice/pids.max is "max", so I *believe* cgroups doesn't come into it.
- AIUI the default scaling factor is 15%
- 3737 * .15 = 560.55

dmesg reports 129949 RAM pages, 7465 pages reserved.

My understanding is that /proc/sys/kernel/threads-max is set by default to 1/16 of RAM pages, so that seems low. I don't see anything setting that sysctl, but I'll probably bump it up and reduce my number of threads a bit.

Thanks.

Loading...