FS#16363 - pacman inside 32bit chroot segmentation fault

Attached to Project: Pacman
Opened by Emil Micek (pantaril) - Saturday, 26 September 2009, 12:14 GMT
Last edited by Dan McGee (toofishes) - Saturday, 26 September 2009, 15:37 GMT
Task Type Bug Report
Category Backend/Core
Status Closed
Assigned To No-one
Architecture All
Severity Critical
Priority Normal
Reported Version 3.3.0
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Summary and Info:
Hi.

On my 64bit PC i'm running up to date version of 64bit arch AND 32bit version of arch is installed into chrooted enviroment in /opt/arch32.
Few days ago i noticed that the regular updates of the 32bit arch started to segfault.

From the log i can see that last successful update was 23.9.2009.

I tried to change mirrors but it doesn't help.

The debug output of pacman:

[root@CHROOT /]# pacman -Syu --noconfirm --debug
debug: config: attempting to read file /etc/pacman.conf
debug: config: new section 'options'
debug: config: HoldPkg: pacman
debug: config: HoldPkg: glibc
debug: config: logfile: /var/log/pacman_32.log
debug: config: dbpath: /var/lib/pacman_32
debug: option 'cachedir' = /var/cache/pacman_32/
debug: config: cachedir: /var/cache/pacman_32
debug: config: new section 'core'
debug: setlibpaths() called
debug: option 'dbpath' = /var/lib/pacman_32/
debug: option 'lockfile' = /var/lib/pacman_32/db.lck
debug: option 'logfile' = /var/log/pacman_32.log
debug: registering sync database 'core'
debug: config: including /etc/pacman.d/mirrorlist
debug: config: attempting to read file /etc/pacman.d/mirrorlist
debug: adding new server URL to database 'core': ftp://ftp.sh.cvut.cz/MIRRORS/arch/core/os/i686
debug: config: finished parsing /etc/pacman.d/mirrorlist
debug: config: new section 'extra'
debug: registering sync database 'extra'
debug: config: including /etc/pacman.d/mirrorlist
debug: config: attempting to read file /etc/pacman.d/mirrorlist
debug: adding new server URL to database 'extra': ftp://ftp.sh.cvut.cz/MIRRORS/arch/extra/os/i686
debug: config: finished parsing /etc/pacman.d/mirrorlist
debug: config: new section 'community'
debug: registering sync database 'community'
debug: config: including /etc/pacman.d/mirrorlist
debug: config: attempting to read file /etc/pacman.d/mirrorlist
debug: adding new server URL to database 'community': ftp://ftp.sh.cvut.cz/MIRRORS/arch/community/os/i686
debug: config: finished parsing /etc/pacman.d/mirrorlist
debug: config: new section 'archlinuxfr'
debug: registering sync database 'archlinuxfr'
debug: adding new server URL to database 'archlinuxfr': http://repo.archlinux.fr/i686
debug: config: finished parsing /etc/pacman.conf
debug: registering local database
:: Synchronizing package databases...
debug: using 'core.db.tar.gz' for download progress
debug: HTTP_PROXY: (null)
debug: http_proxy: (null)
debug: FTP_PROXY: (null)
debug: ftp_proxy: (null)
debug: connected to ftp.sh.cvut.cz successfully
debug: mtimes are identical, skipping core.db.tar.gz
core is up to date
debug: using 'extra.db.tar.gz' for download progress
debug: HTTP_PROXY: (null)
debug: http_proxy: (null)
debug: FTP_PROXY: (null)
debug: ftp_proxy: (null)
debug: connected to ftp.sh.cvut.cz successfully
debug: mtimes are identical, skipping extra.db.tar.gz
extra is up to date
debug: failed to get lastupdate time for community
debug: using 'community.db.tar.gz' for download progress
debug: HTTP_PROXY: (null)
debug: http_proxy: (null)
debug: FTP_PROXY: (null)
debug: ftp_proxy: (null)
debug: connected to ftp.sh.cvut.cz successfully
downloading community.db.tar.gz...
error: segmentation fault
Internal pacman error: Segmentation fault.
Please submit a full bug report with --debug if appropriate.
[root@CHROOT /]#

Steps to Reproduce:
1. install 32bit arch chroot using this howto: http://wiki.archlinux.org/index.php/Arch64_Install_bundled_32bit_system
2. chroot into the new environment: chroot /opt/arch32
3. run pacman -Syu --noconfirm

On my system, this bug is always reproducible.

See attachment for output of "strace pacman -Syu --noconfirm"

In the strace output you can see that pacman dies with "Too many open files" error. I checked /proc/sys/fs/file-max and it is set to 203504 which looks high enough for me.
Just to be sure i tried to double this value but with no effect.

Any help would be greatly appreciated.

Emil.
(application/x-gzip)    strace.out.gz (47.9 KiB)
This task depends upon

Closed by  Dan McGee (toofishes)
Saturday, 26 September 2009, 15:37 GMT
Reason for closing:  Won't fix
Additional comments about closing:  Looks like a fsck issue, we could probably handle this slightly better but oh well
Comment by Xavier (shining) - Saturday, 26 September 2009, 12:34 GMT
Which package did you update on last successful update 23.9.2009 ?
Which pacman version : 3.3 or 3.3.1 ?

Can you run pacman inside gdb and provide a backtrace ? Though the "too many open files" error already tells us something.
Comment by Emil Micek (pantaril) - Saturday, 26 September 2009, 13:04 GMT
On 23.9. i upgraded those packages:

[2009-09-23 04:00] starting full system upgrade
[2009-09-23 04:01] upgraded bash (4.0.028-1 -> 4.0.033-1)
[2009-09-23 04:01] upgraded sqlite3 (3.6.17-1 -> 3.6.18-1)
[2009-09-23 04:01] upgraded heimdal (1.2.1-5 -> 1.2.1-6)
[2009-09-23 04:01] upgraded libcap (2.16-3 -> 2.17-1)
[2009-09-23 04:01] upgraded libldap (2.3.43-3 -> 2.4.18-1)
[2009-09-23 04:01] upgraded qt (4.5.2-6 -> 4.5.2-7)
[2009-09-23 04:01] upgraded wine (1.1.29-1 -> 1.1.29-2)

[root@CHROOT tmp]# pacman --version

.--. Pacman v3.3.0 - libalpm v4.0.0
/ _.-' .-. .-. .-. Copyright (C) 2006-2009 Pacman Development Team
\ '-. '-' '-' '-' Copyright (C) 2002-2006 Judd Vinet
'--'
This program may be freely redistributed under
the terms of the GNU General Public License.


I'm not very fammiliar with gdb. Is this what you want?:

(gdb) run
Starting program: /usr/bin/pacman -Syu --noconfirm
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
:: Synchronizing package databases...
[New Thread 0xf7bdbab0 (LWP 8364)]
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
core is up to date
extra is up to date
community 366.2K 1052.3K/s 00:00:00 [###########################################################################################################] 100%

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf7bdbab0 (LWP 8364)]
0xf7cd395e in readdir64@@GLIBC_2.2 () from /lib/libc.so.6
(gdb) bt
#0 0xf7cd395e in readdir64@@GLIBC_2.2 () from /lib/libc.so.6
#1 0xf7bdbab0 in ?? ()
#2 0xf7f8d15c in ?? () from /usr/lib/libalpm.so.4
#3 0x0c0264e0 in ?? ()
#4 0xff633d60 in ?? ()
#5 0xff633d28 in ?? ()
#6 0xf7f87ebd in _alpm_rmrf () from /usr/lib/libalpm.so.4
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)


Anyway i examined the strace output little more and from line from line 2017 on it looks like pacman is trapped in some kind of recursion trying to create directories
/var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2
/var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2/*
/var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2/*/*
/var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2/*/*/*
... and so on until it dies.

The endless directory tree is there in /var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2/ after the command ends prematurely.

I checked the /var/lib/pacman_32/community.db.tar.gz tarball and it extracts correctly.
Comment by Xavier (shining) - Saturday, 26 September 2009, 13:18 GMT
Arf sorry, I actually forgot to check the strace file (I tried to open it inside firefox but it did not work with gz, then I forgot :D)
Your gdb trace seems to confirm that this infinite loop likely happens in alpm_rmrf
And strace indicates the /var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2 directory.

What is the output of ls -al /var/lib/pacman_32/sync/community//perl-config-tiny-2.12-2 ?
Comment by Xavier (shining) - Saturday, 26 September 2009, 13:22 GMT
Other questions :
which filesystem are you using on /var ?
Could you check your filesystem with fsck ?
Comment by Dan McGee (toofishes) - Saturday, 26 September 2009, 13:22 GMT
If I was a betting man, I bet we have some sort of recursive symlink here and we are getting tripped up on it.
Comment by Emil Micek (pantaril) - Saturday, 26 September 2009, 14:20 GMT
Thank you guys, fsck.ext3 discovered error in the /var filesystem. (something about inode
/opt/arch32/var/lib/pacman_32/sync/community/perl-config-tiny-2.12-2/* should be /opt/arch32/var/lib/pacman_32/sync/community/perl-config-tiny-2.12-2/.)

After i fixed it, everything works as expected:)

Emil

Loading...