I have also created upstream bug with little more detail.

http://bugs.squid-cache.org/show_bug.cgi?id=4057

I have these lines in /etc/squid/squid.conf

adaptation_send_client_ip on
icap_enable on
icap_preview_enable on
icap_preview_size 1024
icap_service_revival_delay 60
icap_service_failure_limit 10 in 5 seconds
icap_service service_avi respmod_precache icap://127.0.0.1:1344/virus_scan
bypass=1
adaptation_access service_avi allow !novirusscan

In novirusscan I have put ".*" i.e. temporarily disabled scanning.

My ICAP daemon is disabled for some reason. (I dont know if that may be issue
but squid is supposed to work even if ICAP daemon is down and it works fine
with 3.4.2.2)


If I comment adaptation_access line, it squid starts working.


Overall dont know if its upstream bug or arch bug.

Is anybody looking in to this?

I filed bug report on squid bugzilla too but got no response.

Could this be due to compiler bug in GCC 4.9

Maybe nobody uses icap_service?

it works well with this config:

acl localnet src 192.168.0.0/16 # RFC1918 possible internal network

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

http_access allow localhost manager
http_access deny manager

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

#http_access deny to_localhost

http_access allow localnet
http_access allow localhost

http_access deny all

http_port 3128

#cache_dir ufs /var/cache/squid 256 16 256

coredump_dir /var/cache/squid

refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320

Also it should not be arch bug because of we just do configure, make, make install.

Someone using adaptation on debian wheezy says its not crashing for him.

http://bugs.squid-cache.org/show_bug.cgi?id=4057

Can you add these two lines in your squid.conf and check if it crashes?

acl novirusscan dstdom_regex -n -i google.com
adaptation_access service_avi allow !novirusscan

I tried lots of things but it keeps crashing on 3.4.5 and same config works for 3.4.4.2

There is another guy using ARCH who confirmed same bug.
(at squid bug link given in above post)

I have a feeling that this is related GCC compiler optimization issue, the same issue is making BIND (named) server crash.
See here: https://bugs.archlinux.org/task/40304

I see the same thing with the Arch pkg trying to use squidclamav.
These are the lines I have used that cause issue:
icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service service_req reqmod_precache bypass=1
icap://127.0.0.1:1344/squidclamav
icap_service service_resp respmod_precache bypass=1
icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
adaptation_access service_resp allow all

I had to comment all these lines to get squid to start.
With them in this is what I see:
# systemctl start squid
Job for squid.service failed. See 'systemctl status squid.service' and 'journalctl -xn' for details.

# journalctl -xn
-- Logs begin at Wed 2014-02-19 19:33:55 CST, end at Fri 2014-05-30 09:47:26 CDT. --
May 30 09:47:22 area51 systemd[1]: Starting Web Proxy Cache Server...
-- Subject: Unit squid.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit squid.service has begun starting up.
May 30 09:47:22 area51 squid[5321]: assertion failed: mem.cc:282: "size == StrPoolsAttrs[i].obj_size"
May 30 09:47:22 area51 squid[5322]: Squid Parent: will start 1 kids
May 30 09:47:22 area51 squid[5322]: Squid Parent: (squid-1) process 5325 started
May 30 09:47:22 area51 systemd[1]: squid.service: control process exited, code=dumped status=6
May 30 09:47:22 area51 systemd[1]: Failed to start Web Proxy Cache Server.
-- Subject: Unit squid.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit squid.service has failed.
--
-- The result is failed.
May 30 09:47:22 area51 systemd[1]: Unit squid.service entered failed state.
May 30 09:47:22 area51 systemd-coredump[5323]: Process 5321 (squid) dumped core.
-- Subject: Process 5321 (squid) dumped core
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: man:core(5)
--
-- Process 5321 (squid) crashed and dumped core.
--
-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.


# tail cache.log
2014/05/30 09:47:12| Set Current Directory to /var/cache/squid
2014/05/30 09:47:12 kid1| Preparing for shutdown after 112570 requests
2014/05/30 09:47:12 kid1| Waiting 30 seconds for active connections to finish
2014/05/30 09:47:12 kid1| Closing HTTP port 127.0.0.1:3128
2014/05/30 09:47:12 kid1| Closing HTTP port 192.168.0.1:3128
2014/05/30 09:47:12 kid1| Shutdown: NTLM authentication.
2014/05/30 09:47:12 kid1| Shutdown: Negotiate authentication.
2014/05/30 09:47:12 kid1| Shutdown: Digest authentication.
2014/05/30 09:47:12 kid1| Shutdown: Basic authentication.
2014/05/30 09:47:22| assertion failed: mem.cc:282: "size == StrPoolsAttrs[i].obj_size"

I forgot to mention that sometimes I see assertion failed too: mem.cc:282: "size == StrPoolsAttrs[i].obj_size"

And sometimes I see free() invalid pointer error.(as reported in main post above)

can it be reproduced with 3.4.6 ?

Same with 3.4.6:
Jun 26 08:42:53 area51 squid[7510]: assertion failed: mem.cc:282: "size == StrPoolsAttrs[i].obj_size"
Jun 26 08:42:53 area51 squid[7511]: Squid Parent: will start 1 kids
Jun 26 08:42:53 area51 systemd[1]: Failed to start Web Proxy Cache Server.


These are the lines I have in squid.conf:
icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
adaptation_access service_resp allow all

# squid -v
Squid Cache: Version 3.4.6
configure options: '--prefix=/usr' '--sbindir=/usr/bin' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--with-logdir=/var/log/squid' '--with-pidfile=/run/squid.pid' '--enable-auth' '--enable-auth-basic' '--enable-auth-ntlm' '--enable-auth-digest' '--enable-auth-negotiate' '--enable-removal-policies=lru,heap' '--enable-storeio=aufs,ufs,diskd' '--enable-delay-pools' '--enable-ssl' '--enable-snmp' '--enable-linux-netfilter' '--enable-ident-lookups' '--enable-useragent-log' '--enable-cache-digests' '--enable-referer-log' '--enable-htcp' '--enable-carp' '--enable-epoll' '--with-large-files' '--enable-arp-acl' '--with-default-user=proxy' '--enable-async-io' '--enable-truncate' '--enable-icap-client' '--enable-ssl-crtd' '--disable-arch-native' '--disable-strict-error-checking' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4'

If I start squid with:
/usr/bin/squid -sYC

It will start but spits out the "assertion failed" message in journalctl and an "Aborted" message on command line - but squid starts and works.
I can see that requests are going through c-icap in the log as well.
I am not at home to test if virus files are getting picked up or not though.

So I am not sure if systemd is stumbling on the "assertion failed" message and it is really harmless or if it is a real failure.

I also tested with c-icap-modules and a squid config like so and still the assertion message.
icap_enable on
icap_preview_enable off
icap_send_client_ip on
icap_service service_av_req reqmod_precache bypass=0 icap://localhost:1344/srv_clamav
icap_service service_av_resp respmod_precache bypass=1 icap://localhost:1344/srv_clamav
adaptation_access service_av_req allow all
adaptation_access service_av_resp allow all

I get crash with squid-3.4.6, too.

Everytime I get a bit different error. (Got different errors in 3.4.5 too)


1) Jun 27 16:29:51 amish kernel: traps: squid[28056] general protection ip:7418a0 sp:7fff87091ca0 error:0 in squid[400000+479000]

2) Jun 27 16:30:05 amish kernel: squid[28062]: segfault at 0 ip (null) sp 00007fffdddef6c8 error 14 in squid[400000+479000]

3) Jun 27 16:30:59 amish kernel: squid[28101]: segfault at 20 ip 0000000000737fbd sp 00007fff15a8ddd0 error 4 in squid[400000+46e000]

4) Jun 27 16:31:36 amish squid[28112]: assertion failed: mem.cc:282: "size == StrPoolsAttrs[i].obj_size"


Again had to switch back to 3.4.4.2

For me another thing I noticed is that if I have only one reqmod or respmod adaptaion_access line I get a seg fault rather than the assertion failed.
If I have both lines I get the assertion failed.

This is the seg fault if it helps:
kernel: squid[2321]: segfault at 20 ip 000000000074189d sp 00007fff549153c0 error 4 in squid[400000+479000]

UPDATE that I posed to squid bugreport.

Just pasting here in case maintainer of squid can create a temporary patch:

========================
Ok I have some good news! :)

It does not crash (atleast from last 5 minutes or so) with
--disable-optimizations

So the bad news is I can not provide backtrace!


This possibly seems to be similar error that happened with named (BIND) DNS
server with GCC 4.9.1.

See this: https://bugs.archlinux.org/task/40304

BIND used to segfault with optimization but did not segfault with optimization
disabled.

BIND guys reported to GCC but GCC guys denied that it is GCC bug. (Conversation
link can be found in above bug report link)

So eventually BIND guys patched BIND (no idea what they did) and its now
working even with optimizations.

May be something similar needs to be done in squid too?

May be this can be tested by compiling in other distro where GCC 4.9.1 exists?

Hope this helps.

Just for the record.

The attached patch (squid-r13407.patch) fixes the issue.

The fix was provided by squid developer.

I am not sure if they intend to implement this in next version as very few users using ARCH and adaptation_access.

So not sure if they consider the patch as immediate priority.

Request to squid ARCH maintainer to release 3.4.6-2 with the attached patch.

Thanks.

PS: PKGBUILD.diff is just for those who want to rebuild squid on ARCH themselves.