FS#56412 - [procmail] out of memory since 3.22-8

Attached to Project: Arch Linux
Opened by Geert Hendrickx (ghen) - Wednesday, 22 November 2017, 19:59 GMT
Last edited by Andreas Radke (AndyRTR) - Tuesday, 28 November 2017, 16:25 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Andreas Radke (AndyRTR)
Levente Polyak (anthraxx)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
The security patch introduced with procmail 3.22-8 ( FS#56398 ) makes procmail go out of memory.

Procmail logs for every mail either:
'Out of memory' or 'Terminating prematurely whilst waiting for memory'

Which caused all my mail to remain stuck in the queue.
Rolled back to procmail 3.22-7 and everything was instantly delivered.



Additional info:
* package version(s)
procmail 3.22-8 (3.22-7 unaffected)

* config and/or log files etc.
From xxx@yyy Wed Nov 22 20:44:48 2017
Folder: **Bounced** 0
procmail: Terminating prematurely whilst waiting for memory
Folder: **Bounced** 0


Steps to reproduce:
This task depends upon

Closed by  Andreas Radke (AndyRTR)
Tuesday, 28 November 2017, 16:25 GMT
Reason for closing:  Fixed
Comment by Andreas Radke (AndyRTR) - Wednesday, 22 November 2017, 21:22 GMT
I'm not sure if I understand the original Debian report right. Sounds to me like only 32bit versions were vulnerable. So it should be safe to revert that patch since we are only 64bit now.
Comment by Levente Polyak (anthraxx) - Wednesday, 22 November 2017, 21:25 GMT
let me investigate the code a bit...
Comment by Santiago Torres (sangy) - Wednesday, 22 November 2017, 21:33 GMT
I can't seem to get an OOM/bounce emails. The original reproducer doesn't crash the application. Ghen, do you have a MWE/reproducer we can use to try this out?
Comment by Levente Polyak (anthraxx) - Wednesday, 22 November 2017, 22:43 GMT
so for now:
- see no reason why this does not affect 64bit, it does and rexl is a int(32) while the other vars are size_t. the underlying problem is pretty much present on 64bit
- can't explain why it would be a error that leads to OOM if you don't have a mail equally huge

i would really appreciate to see an input reproducer that triggers this problem so we can prove it exists and using the reproducer to debug the assumptions
Comment by Geert Hendrickx (ghen) - Thursday, 23 November 2017, 08:28 GMT
The problem seems to be calling an external program from .procmailrc. I use spamprobe from AUR for spamfiltering, but any pipe seems to do. Here's a minimal rule that reproduces it on any e-mail:

VERBOSE=true

:0
MYVAR=|/usr/bin/cat

It takes about a minute before it finally errors out:

procmail: [23960] Thu Nov 23 09:24:47 2017
procmail: Assigning "MYVAR="
procmail: [23960] Thu Nov 23 09:24:47 2017
procmail: Executing "/usr/bin/cat"
<... long delay here ...>
procmail: Out of memory
buffer 0: "/usr/bin/cat"
buffer 1: "/usr/bin/cat"
Folder: **Bounced** 0

real 1m4.005s
user 0m0.000s
sys 0m0.000s

strace shows repeated mmap failures:

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=23995, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
read(0, "R", 1) = 1
read(0, "eturn-Path: <srs0=x4vu=cu=archli"..., 16385) = 2363
read(0, "", 14022) = 0
mmap(NULL, 93843545276416, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0xaab34eb61000) = 0x5559a7878000
mmap(NULL, 93843545411584, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 93843545276416, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0xaab34eb61000) = 0x5559a7878000
mmap(NULL, 93843545411584, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f4e5147e000
Comment by Santiago Torres (sangy) - Monday, 27 November 2017, 21:39 GMT
Hmm, I get the same behavior regardless of the patch using your .rc.

Without patch:
[santiago@LykOS bin]$ zcat ../../../../overflow.822.gz | ./formail -s ./procmail
procmail: [18632] Mon Nov 27 16:34:32 2017
procmail: Assigning "MYVAR="
procmail: [18632] Mon Nov 27 16:34:32 2017
procmail: Executing "/usr/bin/cat"
procmail: Out of memory
buffer 0: "/usr/bin/cat"
buffer 1: "/usr/bin/cat"
procmail: Notified comsat: "santiago@:**Bounced**"
From foo@bar Mon Nov 27 16:34:32 2017
Folder: **Bounced**

With patch:
[santiago@LykOS bin]$ zcat ../../../../overflow.822.gz | ./formail -s ./procmail
procmail: [21776] Mon Nov 27 16:37:26 2017
procmail: Assigning "MYVAR="
procmail: [21776] Mon Nov 27 16:37:26 2017
procmail: Executing "/usr/bin/cat"
procmail: Out of memory
buffer 0: "/usr/bin/cat"
buffer 1: "/usr/bin/cat"
procmail: Notified comsat: "santiago@:**Bounced**"
From foo@bar Mon Nov 27 16:37:26 2017
Folder: **Bounced**

Anything I may be missing to try to reproduce here?
Comment by Geert Hendrickx (ghen) - Monday, 27 November 2017, 21:51 GMT
For me the previous/unpatched 3.22-7 version works fine. I used the 2014 build from the 3.22-7 package though, not a freshly built one. Maybe something else changed, that causes this?
Comment by Geert Hendrickx (ghen) - Tuesday, 28 November 2017, 11:22 GMT
Indeed a newly built procmail 3.22-7 (without the latest patch) fails in the same way.
So something else broke procmail between July 2014 and now. :-(

procmail.old: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=427a4d812afb08578a5cec5369097ab0fbf808b6, stripped
procmail.new: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=8e6ff70dba528e6f5436f389af1a2f67b8f0f673, stripped
Comment by Geert Hendrickx (ghen) - Tuesday, 28 November 2017, 11:32 GMT
The following webpage refers to this bug: http://www.impsec.org/email-tools/sanitizer-download.html (look for "OUT OF MEMORY")
"Procmail 3.22 has a bug in the VARNAME=|command-line construct used by the 1.133 version of the sanitizer. If you are getting OUT OF MEMORY errors, update to the current release of the sanitizer."
(sanitizer is a packaged procmail ruleset - it seems they just stopped using the VARNAME=|command construct)
Comment by Andreas Radke (AndyRTR) - Tuesday, 28 November 2017, 14:03 GMT
http://www.impsec.org/email-tools/sanitizer-changelog.html

" 04/21/2002 (1.134) Customize the MTA command line, to allow for newer sendmail command line options and non-sendmail MTAs: $MTA_FLAGS_CMDLN and $MTA_FLAGS_HDRS. Mangle MIME types in deferred headers if appropriate. Improve encoded-filename handling. Set Errors-To: header. Put the version number in the $NOTIFY message. Fix no-LOGFILE-breaks-UUE-sanitization bug. Defang quotes-in-extension Outlook attack. Add WMA and WMV to mangled executable extensions, per bugtraq. Fix trailing periods in addition to trailing whitespace - Windows drops trailing periods from filenames without warning. Work around memory allocation error in procmail v3.22. Add the OnContextMenu and OnDragStart events to HTML defanger. Improved recipient address parsing for logs and bounce messages. Minor procmail efficiency enhancements."

I guess we need that workaround to be added. Also there's a Debian pkg where they seem to have fixed some memory corruption.
http://metadata.ftp-master.debian.org/changelogs/main/p/procmail/procmail_3.22-26_changelog
Comment by Geert Hendrickx (ghen) - Tuesday, 28 November 2017, 14:21 GMT Comment by Geert Hendrickx (ghen) - Tuesday, 28 November 2017, 14:23 GMT
Yes, the above referenced Debian patch fixes this issue.
Comment by Andreas Radke (AndyRTR) - Tuesday, 28 November 2017, 15:38 GMT
Please confirm 3.22-9 in testing repo.
Comment by Geert Hendrickx (ghen) - Tuesday, 28 November 2017, 15:44 GMT
Yes, confirmed. Thanks!

That being said, I think it's time to start looking for an alternative, as procmail seems to be riddled with such problems (just looking at the Debian patches - we should probably grab them all).
Also procmail's old maintainer declared the code dead and insecure: https://marc.info/?l=openbsd-ports&m=141634350915839&w=2

I've looked into alternatives in the past (mostly Sieve), the problem is they can't pipe mail through external commands, and I love spamprobe for filtering (I use spamprobe not just for spam/ham, but for other categorisation as well).

Comment by Levente Polyak (anthraxx) - Tuesday, 28 November 2017, 15:47 GMT
yeah pulling the other corruption patches would be neat.

Loading...