Arch Linux

I cannot reproduce on x86_64. Can your reproduce this bug with a different mailbox and/or maildir? Can you reproduce on x86_64? Cheers.

Hello Thomas, Gaetan told me yesterday that you have a problem with S-nail? My problem is that i can't reproduce your problem! (I could understand if you could NOT interrupt from an IMAP password prompt -- ;) -- but interrupting via CTRL+C plus signal delivery is unfortunately ;) well supported.)

Note that i'm aware that the code path has a lot of problems per se, but SIGSEGV really shouldn't occur *there*.
Background: the only buffer involved here is the linebuffer (#3 readline_restart.. linebuf=0x80a5b5c <termios_state+60>); it may be reallocated, but this would not happen when read(2)ing. Of course, signals are not blocked (note the I/O system will be rewritten in the forthcoming years, but it'll really take _years_), and therefore a signal could occur during reallocation, causing a jump, pretty surely leaving garbage behind. Yet, if that would be the case here, we'd crash somewhere else (lex.c:commands(), unless i'm mistaken).

See, i'm not euphemising anything here, i *hate* bugs and i'm more than just interested in fixing this, but i need to know what to look for.
Thank you, and ciao

P.S.: that is to say, if you have a configuration which can be used to reproducably cause this, then please be so kind and make it available. The way you've shown above alone is not reproducable (by me).
Ciao,

Hello again,
i still think this bug report is more about criticism on code quality (is it?), i've taken a preliminary step and added some signal blocking allocation functions, a.k.a. cure-by-sledgehammer. I have yet pushed it to the [master] branch, but hope i will someday be able to drop them again.
Does the attached patch against S-nail 14.4.5 stop your problem?
Apply it in the source directory of S-nail via

$ patch -E -p1 < PATH-TO-PATCH

Thank you.
(P.S.: my weekend was terrible.)

Hello,

this bug report is no criticism on code quality... I have applied the patch and the problem is still there. I actually thought this bug was very obvious, so when you said you could not reproduce it, I examined my mail.conf and found out that the problem seems to be related to these two lines, that I use to read html mail on the commandline and gpg encrypted mail (as far as I understand, S/Mime is not very convenient for me, maybe I don't understand enough). Whatever here are the two lines:

set pipe-text/html="html2text -ascii -"
set pipe-text/plain="tee >(gpg2 -d - 2>/dev/null)"

So the problem seems to be related to the piping of text to some other program.

Hope that is more helpful than my other report,
Thomas

P.S.: Sorry, that I made you wait, as I did not have access to internet last weekend, I should have told you before.
P.P.S.: I really do like mailx, best mail program there is, in my opinion, thank you.

|this bug report is no criticism on code quality...

Sure. Raindrops keep falling on my head.. they keep falling.
Just continue complaining, the codebase can only get better due to that.

|set pipe-multipart/encrypted="gpg -d"

Yes, i'm sorry that we do not yet support GPG.
There was a hint on how to handle GPG with the nail codebase on the original nail list [1]; i never tried it, because back in time i once managed to forget anything i need to unencrypt a backup, effectively loosing *a lot* of work and almost going stony. So i have mental deficiencies in respect to GPG.

But i have heard this feature request; starting with S-nail 15.0 (not before 2015) it will be possible to address message parts individually, as in 'p 12.2' (print only 2nd part of message number 12); the codebase rework will then hopefully allow nice integration of GPG.

And the S/MIME feature i haven't tested yet, there are a lot of things to do.

[1] <http://article.gmane.org/gmane.mail.nail.devel/1485/match=gpg>

|P.P.S.: I really do like mailx, best mail program there is, in my opinion, thank you.

No, it's not.
Maybe it will never be.
Ciao,

Ping?
I still don't know what your problem is.

Does your SMTP server require an authentification method that we don't yet support, i.e., GSSAPI?
If so, please mention it, it may become added at a later time. But note it *will* take time, and it definitely won't happen until *after* v15.0, because we need a more flexible implementation for the most important part of the software.
(Also, i think secure transport is more important than secure login, because the latter is automatically secure if the former is, but that only as a side note. DNSSEC is evil.)

Also, if your "tee >(gpg2 -d - 2>/dev/null)" fails then because your shell doesn't support that syntax.
I come to this conclusion because on ArchLinux /bin/sh is (actually a symlink.. to) bash(1), and because S-nail automatically prefers $SHELL from the user environment over that fallback, which would work.
So please either override $SHELL in your shell ('$ SHELL=/usr/bin/bash; export SHELL; mail') or in S-nail itself ('? set shell=/usr/bin/bash').
That should get you going.

I'm open to any other suggestion.
Thank you and ciao,

Hello Steffen,

I can read html mail, I can send mail (authentication works fine), I can read encrypted messages and html mail with my configuration. The problem is just that in some cases I get a segmentation fault, which is something that should not happen. Basic functionality is working.

I send you a script, a mail directory and a configuration file for mailx, and the files SESSION and SESSION2.
What I do (output in the file SESSION):
./test-script
r [to respond mail]
Ctrl-D [to end mail]
Ctrl-C [to end password input]
SEGFAULT


This results in a segfault, too (output in SESSION2):
./test-script
r
Ctrl-C
Ctrl-C [to finally interrupt mail]
Ctrl-C [at mailx prompt ?]
SEGFAULT

I set shell=/usr/bin/bash in the configuration file, nothing changes. This bug is not a problem for basic functionality, just for unexpected behavior. Please tell me if you can reproduce it with this configuration file and this script.

Have a good day,
Thomas

Hello Thomas, i've downloaded it and can indeed reproduce the crash running ArchLinux.
Thank you very much for insisting and producing this test case, i really tried many times on multiple systems, including ArchLinux, but was not able to reproduce that myself! Anywhere else i only see

^DEOT
Password: Interrupt
? x

I'm looking into this!

Hello,

for the sake of practice and convenience of testing, I wrote an expect testing script, which will run my test-script and inputs the commands that lead to the segfault automatically. In case you do not have it on your computer, you find expect in the Archlinux repository

https://www.archlinux.org/packages/extra/x86_64/expect/

Have a nice weekend,
Thomas

Heya, Thomas,
expect(1) i never used, but thanks, i've tried it, see below. (I first thought i was wrong, since i'm a blind-worm. But it's TCL!! Don't mind if you don't understand this :)

First i must say that this is a new VM that i've freshly setup (on a Laptop) with archlinux-2013.10.01-dual.iso last Friday evening, after Gaetan (thank you!) has informed me about your bug report. I've had some trouble on this smallish laptop etc., and i usually only have GPRS internet (also consciously) so until this evening it has had no gdb(1), gcc(1), make(1) etc.

Short: i did a 'pacman -S -v -u -y', *because* pacman(1) failed to install gcc(1) 4.8.2-4 (whatever, but "-4"), except from one .hu mirror (the 42nd or so), but it didn't finish (install it) because there was a NULL message (key verification or so) whatever (i haven't really understood it). It turned out that some packages were outdated (it installed 4.8.2-5 (.. but "-5")) without complaining.

Now this: after performing the 'pacman -S -v -u -y' update i cannot reproduce your error no more:

?127[steffen@arlx s-nail-debug]$ expect s-nail-segfault-expect.test
spawn ./test-script
r
mail version v14.4.5. Type ? for help.
"/home/steffe...c/s-nail-debug/test-mail": 1 message
>A 1 bugs@archlinux.org Thu Dec 5 14:47 103/4430  FS#37950 : [s-nail] 14.4.5
? r
To: bugs@archlinux.org
Subject: Re:  FS#37950 : [s-nail] 14.4.5 segmentation fault

^DEOT
^CInterrupt
? x
?0[steffen@arlx s-nail-debug]$ expect s-nail-segfault2-expect.test
spawn ./test-script
r
mail version v14.4.5. Type ? for help.
"/home/steffe...c/s-nail-debug/test-mail": 1 message
>A 1 bugs@archlinux.org Thu Dec 5 14:47 103/4430  FS#37950 : [s-nail] 14.4.5
? r
To: bugs@archlinux.org
Subject: Re:  FS#37950 : [s-nail] 14.4.5 segmentation fault


(Interrupt -- one more to kill letter)
? Interrupt
? x

Oooops! ,)

That brings me back to my very first statement -- this is not our buffer, if so, it would have crashed at a different place.
I *ADMIT* this is *TERRIBLE* code, and one of the hard things to do will be to get rid of the longjmp()s that are used.
It is very hard to deal with those, especially with todays super-optimizing compilers, thread-enabled (also: TLS) and FORTIFYied libraries, etc.
I personally *never* used jumps regulary, and it was one of the first TODO entries of S-nail to get rid of 'em. But it'll take time.

Dear Thomas, would you be so kind to update your system and try again?
Can you confirm that updating ArchLinux stops S-nail from crashing?
That would be wonderful! :-))
I hope you have had a nice weekend when you read this, ciao,

--steffen

Hello Steffen,

recompiling with my system updated to the newest state, does not change the problem.

I use the source from the Arch-ABS and building with makepkg with options (without stripping the binaries and keeping the debug symbols). I will add my makepkg.conf for the compiler options and the PKGBUILD (where I added your patch that you sent earlier, patch or no patch, no difference). GCC is version 4.8.2-5.

It seems odd to me, that simply recompiling resolves the problem for you, it does not work for me. I was wondering if Gaetan was able to reproduce the bug now, and if yes, if recompiling does the trick for him. And push the new package.

As you said earlier: this is not our buffer. So maybe then it maybe a problem with communication between the mailx and external programs:
I was wondering in what part of the source you handle the communication with external programs like gpg and html2txt and writing to the dead.letter file. It may be some problem with the output of these files.

This is actually my first bug report, thought it would be a bit easier to close a bug...
But I had a nice weekend,
Thomas

Hmm, pretty stormy weekend indeed.

Interesting... Frankly i didn't compile anything, i've just did a 'pacman -S -v -u -y' and tested with the ArchLinux-shipped S-nail a.k.a. mailx(1). I.e., i was in the preparation to build, but the binary worked well after the update. But i can compile v14.4.5 as well as the current [crawl] HEAD, with and without the extended compiler flags (from the S-nail build system), and everything is still just fine.

I think i need to dig more into the ArchLinux pacman/makepkg system - how can i actually build the way you do: can you show the command line that you use, please?
But now i try again with your CFLAGS etc..

Communication problem with other programs? This is a different part of the program. Yet i agree that normally you collect user credentials before you enter something that has to deal with timeouts, but for this to be done easily we need the intermediate "void" box, so that a leave-current <-> credentials of new <-> enter new code flow is possible. That takes even more time (but will be done).

ok, i've did makepkg:

212K -rw-r--r-- 1 steffen steffen 215168 10. Dez 12:51 s-nail-14.4.5-1-x86_64.pkg.tar.xz

but using the binary produced like that doesn't bring us together either.
The only thing that is different is that you are using 32-bit ArchLinux, and Gaetan and i use x86_64.
Nonetheless, given that i actually could reproduce your problem with your specific usecase and configuration on vanilla 2013.10.01 (x86_64), but no longer after doing 'pacman -S -v -u -y', i don't think that that matters. (And i hope that nobody disproves me.)

Now, if you're in doubt why you see what could be seen: it is likely that the problem is only triggered in a very special memory layout situation. This is why i couldn't reproduce it with my local configuration, but only with the one you've thankfully provided. 'Thing is that behind the scenes not few memory allocations and de-allocations occur, which need to be served by the system allocator even further behind the scenes, which in turn allocates and de-allocates large(r) memory areas from the operating system. It makes sense to believe that the problem you see -- most likely in the GNU C library or some of its "satellites", or even the dynamic linker? -- only occurs (or occurred in the buggy version that was in 2013.10.01) in, well, a specific state of that environment. Allocating some further memory chunks (e.g., by having more set'ted options in MAILRC) causes that state to be changed, and the problem disappears.
The assertion you've seen was

Password: *** Error in `mailx': double free or corruption (!prev): 0x08389288 ***

See the '!prev' - when i do

?0[steffen@arlx nail.git]$ git grep -E '!\s*prev'
?1[steffen@arlx nail.git]$ git grep -E '!\s*prev' v14.4.5 --
?1[steffen@arlx nail.git]$

i get nothing in return (note the '?1' is actually the $? from the shell).
(We actually have some 'prev' in auxlily.c, though. Nonetheless, see how far apart the buffer 0x08389288 is from all the dynamic memory that those auxlily.c pointers manage, e.g., linebuf=0x80a5b5c: this is more than 3 megabytes! If you don't use makepkg(1) but simply run 'make _buh' in the S-nail source directory you'll get additional commands, like 'sstats' and 'smemtrace' -- if you use the latter you see all memory allocations that auxlily knows about.)

Sometimes programming is hairy; e.g., i've stumbled over a program crash that ended in OpenSSL session objects, but after compiling OpenSSL as a debug library and linking statically against those debug libraries the problem simply doesn't show up again! Since i-think-more-than-a-month i need special treatment to get at those static libs, and for no benefit at all. It's extremely annoying.

Whee, maybe i should start and try valgrind or so.

P.S.: of course, with randomized memory locations three megabytes doesn't mean a thing, and, trying it out a bit, it's likely to be used here.
Oooops, i get a different segmentation fault:

*** Error in `./s-nail': free(): invalid next size (fast): 0x0000000000cda3e0 ***

What is this?? :-((
I'm looking into this.

For the record..

Program received signal SIGABRT, Aborted.
0x00007ffff73f23d9 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007ffff73f23d9 in raise () from /lib/libc.so.6
#1 0x00007ffff73f37d8 in abort () from /lib/libc.so.6
#2 0x00007ffff742fe64 in __libc_message () from /lib/libc.so.6
#3 0x00007ffff743562e in malloc_printerr () from /lib/libc.so.6
#4 0x00007ffff7436307 in _int_free () from /lib/libc.so.6
#5 0x000000000043e545 in sendpart (zmp=zmp@entry=0x6a1a08, ip=ip@entry=0x6d4d48, obuf=obuf@entry=0x695950, doign=doign@entry=0x6680a0 <ignore>,
qf=qf@entry=0x7fffffffbe80, action=action@entry=SEND_TODISP, stats=<optimized out>, stats@entry=0x7fffffffbf70, level=level@entry=0) at send.c:910
#6 0x000000000043f9c9 in xsend (mp=mp@entry=0x6a1a08, obuf=0x695950, doign=0x6680a0 <ignore>, prefix=prefix@entry=0x0, action=SEND_TODISP,
stats=stats@entry=0x7fffffffbf70) at send.c:499
#7 0x00000000004091e4 in type1 (msgvec=0x694aa0, doign=doign@entry=1, page=page@entry=0, pipe=pipe@entry=0, decode=decode@entry=0, cmd=cmd@entry=0x0,
tstats=tstats@entry=0x0) at cmd1.c:1021
#8 0x000000000040bae0 in type (v=<optimized out>) at cmd1.c:1191
#9 0x0000000000428053 in execute (linebuf=<optimized out>, contxt=contxt@entry=0, linesize=<optimized out>) at lex.c:702
#10 0x0000000000429446 in commands () at lex.c:508
#11 0x000000000042e801 in _rcv_mode (folder=<optimized out>, folder@entry=0x7fffffffe6d8 "/tmp/nail.ar") at main.c:344
#12 0x0000000000405b8d in main (argc=<optimized out>, argv=<optimized out>) at main.c:691
(gdb)

Hello again,
i have the gut feeling that at least what i see is (yet another) bug in the GNU LibC, and i'm practically always right.
I've instrumented the code to get at the above:

(SFREE=0x2507360 . . . 0x2507340)@@!
[6](SFREE=0x25137f0 . . . 0x25137d0)@@!
[7][7.17][7.18][7.19][7.19.6][7.19.7]{1}(SMALLOC=0x2507110 0x2507130)
(SFREE=0x2507130 . . . 0x2507110)@@!
(SMALLOC=0x2507110 0x2507130)
{2=0x2507130}{3}(SMALLOC=0x24fc9f0 0x24fca10)
(SFREE=0x24fca10 . . . 0x24fc9f0)@@!
{8}{9}{10}{11=0x2507130}(SFREE=0x2507130 . . . 0x2507110)@

and it turned out that in auxlily.c:sfree() we crash when calling the LibC free(3). Note that the pointer we pass (0x2507110) is the same as the one we've allocated three lines higher.
Debug-enabled S-nail can change it's allocation strategy on-the-fly when debug is enabled at runtime via 'set debug=1', in which case allocated chunks are not given back to the LibC allocator, but held in a list of to-be-freed chunks unless the command loop ticks (the interactive prompt appears), and when i'm doing this i get:

(SMALLOC=0x35b1f00 0x35b1f20)s-nail: malloc.c:2365: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t)))
- 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.

At the time this happens S-nail consumes less than 17 MB of memory, so i don't think this is the problem.

Conclusion: either GNU LibC malloc is broken and/or the GNU C compiler overoptimizes some piece of code around here, causing this buggy behaviour.

Looking at the piece of cake that produces the 'invalid next size (fast)' abortion message i've shown you above [1],
i do see multiple gcc(1) extensions around (__builtin_expect(), ({}) blocks). But not that this is the cause of it.

[1] <https://github.com/cdepillabout/glibc/blob/e28c88707ef0529593fccedf1a94c3fce3df0ef3/malloc/malloc.c>

All i can do is to reassure you that all other systems i'm using to test S-nail (Mac OS X, FreeBSD, NetBSD, OpenBSD, DragonFly BSD, UnixWare 7.1.4, CRUX 3.0 Linux, Slackware 14.0 Linux) do *NOT* show this error, regardless of the used compiler (several sorts of clang and gcc)!

At the moment i'm out of luck in respect to S-nail. We don't do weird things in the path that fails, which is all i can tell you right now.
Any suggestions? Gaetan??

Sorry, update: i need to correct myself: when i compile the current development HEAD on (up-to-date) Crux 3.0, open my mail archive of the nail mailing list, and type 'p*' (which is what i've done above, by the way :(), then scroll in less(1) via 'G' (ditto :(), then S-nail bails with the same error (freeing a pointer that was allocated just before, in send.c:_print_part_info()).

So, thanks again for insisting, i'm finally convinced there is a weird thing going on in S-nail.
I come back to you when i actually have found out what is wrong.
Thank you.

Sorry Steffen that I haven't been more responsive: I was never able to reproduce that segfault. But I'm glad to read here (and on the s-nail mailing list) that you found an environment where this reproducibly happens. That bug stands no chance now that you've tracked it down! :)

Heh Mr. -- could it be that your problem would be catched if a memory cache / stack whistleblower would do this:

// upper hope
if(*r(ui4*,x.ui1p+u.e->alloc) != _HOPE) {
Log::lock();
Log::write(Log::alert,
"%s(%P): \"upper hope test\" failed.%R"
"\tThis means that you have violated upper bounds.%R"
"\tPointer last seen: file %s, line %hu.%R",
_func, _ptr, u.e->file, u.e->line);

if(!_LAST_HOPE(*r(ui4*,x.ui1p+u.e->alloc)))
Log::write(Log::emerg,
"%s(%P): \"last upper hope test\" failed.%R"
"\tYou've violated upper bound GT 2 bytes.%R"
"\tMay cause problems for upper alloc later on...%R",
_func, _ptr);
else
Log::write(Log::alert,
"%s(%P): \"last upper hope test\" succeeded.%R"
"\tYou've violated upper bound LE 2 bytes.%R",
_func, _ptr);

Log::unlock();
ret = fal0;
}

Anyway (sigh), could it be that this diff makes you again a happy S-nail user?

diff --git a/send.c b/send.c
index f45831d..c10835f 100644
--- a/send.c
+++ b/send.c
@@ -205,8 +205,8 @@ _print_part_info(struct str *out, struct mimepart *mip,
/* Max. 24 */
if (is_ign("content-type", 12, doign)) {
out->s = mip->m_ct_type_plain;
- out->l = strlen(out->s) + 1;
- ct.s = ac_alloc(2 + out->l);
+ out->l = strlen(out->s);
+ ct.s = ac_alloc(out->l + 2 +1);
ct.s[0] = ',';
ct.s[1] = ' ';
ct.l = 2;
@@ -220,7 +220,7 @@ _print_part_info(struct str *out, struct mimepart *mip,
out->l = smin(out->l, 24);
memcpy(ct.s + ct.l, out->s, out->l);
ct.l += out->l;
- ct.s[ct.l] = 0;
+ ct.s[ct.l] = '\0';
}

/* Max. 27 */

If so -- otherwise intermediate only -- many thanks for reporting this bug and your endurance.
The diff above (and attached) is on the [master] branch already.
It applies cleanly to the v14.4.5 ball, too.
If you can confirm that this fixes your problem i'm gonna release a new S-nail version asap.
Thanks!

@gaetan: YEAH! (At least what i hope.)

P.S.: and don't worry, at some date in the future S-nail and myself will be ready to give spiritual care for doomed mutt(1) users!!

Very nice!

I cloned from your git master branch, compiled, et voila:
My test case does not give a segfault anymore. Problem resolved !!!!

Thank you! Keep it up.
Thomas

P.S: I was one of those doomed mutt users. Mutt was too complicated for me, and not very well documented, in my opinion. Discovered mailutils (I think) which was kind of too basic, which led me to (heirloom-)mailx, which was exactly right, not to basic and still simple. And then I went with the flow to s-nail. So again, keep up the good work.

Now that is really great news, Thomas!!

As a result of your report i'll change the S-nail debug memory allocator to detect bound excess on its own, the three different GNU LibC reports we see in this bug ticket were not helpful, even misleading. That will hopefully reduce user inconvenience in the future!
I'll release a new S-nail version pretty soon!

Thanks again for staying responsive, Thomas!


(P.S., for the record: the upper hope test code excerpt from my memory cache, as above, uses misaligned memory accesses, but which was ok, then. It should test bytewise instead to be portable.)

P.S.: i've implemented the memory canaries, and they were already helpful in verifying another bug report on the s-nail-users@ mailing-list. (In fact it seems S-nail was near being unusable on Linux systems all this time, SORRY!)

I will work hard the upcoming days to get the next release shipped, which will also include the necessary fix for this ticket, but i think it is better to do some more testing now that we have memory canaries, so please be patient a bit.
Thank you and Ciao,

--steffen