FS#53499 - [couchdb] fails upon starting
Attached to Project:
Community Packages
Opened by Peter (protake) - Wednesday, 29 March 2017, 17:16 GMT
Last edited by Bruno Pagani (ArchangeGabriel) - Thursday, 07 December 2017, 18:45 GMT
Opened by Peter (protake) - Wednesday, 29 March 2017, 17:16 GMT
Last edited by Bruno Pagani (ArchangeGabriel) - Thursday, 07 December 2017, 18:45 GMT
|
Details
$ ./couchdb
[os_mon] memory supervisor port (memsup): Erlang has closed [os_mon] cpu supervisor port (cpu_sup): Erlang has closed {"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"} Kernel pid terminated (application_controller) ({application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,"couch_epi|chttpd_auth|keeper",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_u Maybe related to the openssl update? Or am I doing something wrong? Fresh installation of couchdb. |
This task depends upon
Closed by Bruno Pagani (ArchangeGabriel)
Thursday, 07 December 2017, 18:45 GMT
Reason for closing: Fixed
Additional comments about closing: Was fixed by CouchDB 2.1 release a while ago.
Thursday, 07 December 2017, 18:45 GMT
Reason for closing: Fixed
Additional comments about closing: Was fixed by CouchDB 2.1 release a while ago.
systemctl start couchdb && systemctl status couchdb
does show it as active for a short time, but a second
systemctl status couchdb
leads to
couchdb.service: Failed with result 'exit-code'.
But this was expected not to work in fact (my comment was answered in IRC) because erlang(-nox) is built against openssl. So maybe there is a real issue with couchdb on 1.1 and unless downgrading erlang too this might need proper fixing.
$ sudo npm install -g pouchdb-server
$ pouchdb-server
I've had no issues regarding compatibility (though it should be noted that I don't use advanced CouchDB features) as my design documents, views, and map-reduce scripts worked out-of-the-box.
perl-net-ssleay also produces crashdump when used from AnyEvent::XMPP::IM::Connection perl module.
(gdb) bt
#0 0x00007ffff43f8ea4 in ?? () from /usr/lib64/libcrypto.so.1.1
#1 0x00007ffff47038a2 in ?? () from /usr/lib64/libssl.so.1.1
#2 0x00007ffff470170f in ?? () from /usr/lib64/libssl.so.1.1
#3 0x00007ffff46e7a3a in ?? () from /usr/lib64/libssl.so.1.1
#4 0x00007ffff46f8049 in SSL_write () from /usr/lib64/libssl.so.1.1
#5 0x00007ffff497f496 in XS_Net__SSLeay_write ()
from /usr/lib/perl5/vendor_perl/auto/Net/SSLeay/SSLeay.so
https://git.archlinux.org/svntogit/community.git/commit/trunk?h=packages/erlang&id=11bc36ff9cb4e5c406c174c04a38ee1a24f63ec7
ERL_FLAGS="-couch_ini /usr/lib/couchdb/etc/default.ini /usr/lib/couchdb/etc/datadirs.ini /etc/couchdb/local.ini" /usr/lib/couchdb/bin/couchdb
Both with erlang from [community] and with a rebuilt erlang (just in case it was related to an updated dependency somehow), I get the same error message.
Perhaps this is a bug for upstream couchdb?
Probably couchdb is using something that is removed by the disable-deprecated.patch file.
I'll have a look at erlang package to check if we can re-enable the deprecated features without too much additional porting work.
$ sudo pacman -S erlang
...
Packages (1) erlang-19.3-3
...
$ sudo pacman -S couchdb
...
Packages (1) couchdb-2.0.0-9
...
$ sudo systemctl start couchdb.service
$ sudo systemctl status couchdb.service
...
(same error)
...
https://git.archlinux.org/svntogit/community.git/commit/trunk?h=packages/couchdb&id=be476aacda21008dc3d4a32a0c5d5fbdf248baff
As I understand most meaningful line must be 1st because of cgit shows only 1st line in history.
$ pacman -Q couchdb erlang-nox
couchdb 2.0.0-11
erlang-nox 19.3-3
$ ERL_FLAGS="-couch_ini /usr/lib/couchdb/etc/default.ini /usr/lib/couchdb/etc/datadirs.ini /etc/couchdb/local.ini" /usr/lib/couchdb/bin/couchd
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,"couch_epi|chttpd_auth|keeper",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_u
Crash dump is being written to: erl_crash.dump...
Crash dump is being written to: erl_crash.dump...done
[root@pupykin ~]# pacman -Q erlang
erlang 19.3-3
[root@pupykin ~]# pacman -Q couchdb
couchdb 2.0.0-11
https://www.apache.org/dyn/closer.lua?path=/couchdb/source/2.0.0/apache-couchdb-2.0.0.tar.gz
Then, simply compiled it and replaced /usr/lib/couchdb folder with the compiled artifact (be sure to configure the path for your ini files as well):
$ ./configure && make release
$ pacman -Q openssl erlang
openssl 1.1.0.e-1
erlang 19.3-3
$ sudo systemctl status couchdb.service
● couchdb.service - CouchDB Server
Loaded: loaded (/usr/lib/systemd/system/couchdb.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2017-04-28 13:22:11 BST; 7h ago
Main PID: 3826 (beam.smp)
Tasks: 47 (limit: 4915)
CGroup: /system.slice/couchdb.service
├─3826 /usr/lib/couchdb/bin/../erts-8.3/bin/beam.smp -K true -A 16 -Bd -- -root /usr/lib/couchdb/bin/.. -progname couchdb -- -home /var/lib/couchdb -- -boot /usr/lib/couchdb/bin/../releases/2.0.0/couchd
├─3837 /usr/lib/couchdb/bin/../erts-8.3/bin/epmd -daemon
├─3856 erl_child_setup 1024
├─3868 sh -s disksup
├─3870 /usr/lib/couchdb/bin/../lib/os_mon-2.4.2/priv/bin/memsup
├─3871 /usr/lib/couchdb/bin/../lib/os_mon-2.4.2/priv/bin/cpu_sup
├─4294 ./bin/couchjs ./share/server/main.js
├─4295 ./bin/couchjs ./share/server/main.js
├─4419 ./bin/couchjs ./share/server/main.js
├─4420 ./bin/couchjs ./share/server/main.js
├─4421 ./bin/couchjs ./share/server/main.js
├─4425 ./bin/couchjs ./share/server/main.js
└─4426 ./bin/couchjs ./share/server/main.js
Also, the whole .install file could be removed by making use of systemd-sysusers and systemd-tmpfiles instead. :) See an example in my certbot-user package for instance: https://aur.archlinux.org/cgit/aur.git/tree/?h=certbot-user.
abr 30 13:59:55 arch couchdb[4723]: [info] 2017-04-30T16:59:55.866396Z couchdb@localhost <0.9.0> -------- Application snappy started on node couchdb@localhost
abr 30 14:00:00 arch couchdb[4723]: [notice] 2017-04-30T17:00:00.846661Z couchdb@localhost <0.326.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:327) <= mem3_shards:load_shards_from_disk/1(line:315) <= mem3_shards:load_shards_from_disk/2(line:331) <= mem3_shards:for_docid/3(line:87) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)
abr 30 14:00:00 arch couchdb[4723]: [error] 2017-04-30T17:00:00.846699Z couchdb@localhost emulator -------- Error in process <0.453.0> on node couchdb@localhost with exit value:
abr 30 14:00:00 arch couchdb[4723]: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,327}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,315}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,331}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,87}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,187}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,134}]}]}
abr 30 14:00:05 arch couchdb[4723]: [notice] 2017-04-30T17:00:05.847768Z couchdb@localhost <0.326.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:327) <= mem3_shards:load_shards_from_disk/1(line:315) <= mem3_shards:load_shards_from_disk/2(line:331) <= mem3_shards:for_docid/3(line:87) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)
abr 30 14:00:05 arch couchdb[4723]: [error] 2017-04-30T17:00:05.847831Z couchdb@localhost emulator -------- Error in process <0.550.0> on node couchdb@localhost with exit value:
abr 30 14:00:05 arch couchdb[4723]: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,327}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,315}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,331}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,87}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,187}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,134}]}]}
abr 30 14:00:10 arch couchdb[4723]: [notice] 2017-04-30T17:00:10.848727Z couchdb@localhost <0.326.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:327) <= mem3_shards:load_shards_from_disk/1(line:315) <= mem3_shards:load_shards_from_disk/2(line:331) <= mem3_shards:for_docid/3(line:87) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)
abr 30 14:00:10 arch couchdb[4723]: [error] 2017-04-30T17:00:10.848800Z couchdb@localhost emulator -------- Error in process <0.615.0> on node couchdb@localhost with exit value:
abr 30 14:00:10 arch couchdb[4723]: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,327}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,315}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,331}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,87}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,187}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,134}]}]}
~
ERL_FLAGS="-couch_ini /usr/lib/couchdb/etc/default.ini /usr/lib/couchdb/etc/datadirs.ini /etc/couchdb/local.ini" /usr/lib/couchdb/bin/couchdb
I still get errors. (erl_crash.dump: http://ix.io/t4T)
However, the ssl-related patches and cherrypicks no longer seems to be needed. Please test/confirm.
erlang-nox 20.0rc-1
couchdb 2.0.0-13
And couchdb works fine.
mai 15 02:16:35 arch systemd[1]: Started CouchDB Server.
mai 15 02:16:36 arch couchdb[3711]: [os_mon] memory supervisor port (memsup): Erlang has closed
mai 15 02:16:36 arch couchdb[3711]: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
mai 15 02:16:37 arch couchdb[3711]: {"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttp
mai 15 02:16:37 arch couchdb[3711]: Kernel pid terminated (application_controller) ({application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,"couch_epi|chttpd_
mai 15 02:16:37 arch systemd[1]: couchdb.service: Main process exited, code=exited, status=1/FAILURE
mai 15 02:16:37 arch systemd[1]: couchdb.service: Unit entered failed state.
mai 15 02:16:37 arch systemd[1]: couchdb.service: Failed with result 'exit-code'.
EDIT: Log.
couchdb: [os_mon] memory supervisor port (memsup): Erlang has closed
couchdb: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
couchdb: {"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}
couchdb: Kernel pid terminated (application_controller) ({application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,"couch_epi|chttpd_auth|keeper",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_u
/build/couchdb/src/apache-couchdb-2.0.0/src/couch_epi/src/couch_epi_util.erl:33: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2
And then a bit latter:
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:25: Warning: call to crypto:sha/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:27: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:37: Warning: call to crypto:sha_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:39: Warning: call to crypto:md5_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:50: Warning: call to crypto:sha_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:52: Warning: call to crypto:md5_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:63: Warning: call to crypto:sha_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:65: Warning: call to crypto:md5_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:76: Warning: call to crypto:sha_mac/2 will fail, since it was removed in 20.0; use crypto:hmac/3
What I still don’t understand is why 2.0.0-13 does work. I’m wondering if erlang is just a makedep and not a rundep, and it seems so since I’ve been able to remove erlang-nox from my system and then still having couchdb starting/working OK. Maybe that’s related to CouchDB packing erts? Anyway this is not a good situation since any further rebuild of couchdb required will make it fail.
Alexander, the bug you’ve reported at https://issues.apache.org/jira/browse/COUCHDB-3414 should be updated with the correct error message (thanks to the last change in the PKGBUILD):
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}
And probably add the above compilation messages. If you could do that it would be nice, avoiding me to open an account there.
I can add the error message to the bug report tomorrow (I'm on mobile now). Please remind me if I forget.
I’ve also been able to confirm that in the current state Erlang is just a makedep, because CouchDB embeds the erts it founds at compile time. Apparently the PKGBUILD was made so that CouchDB use system erts instead, but this is currently commented out in the PKGBUILD. I think we should get back to using system erts when this issue will be fixed, but in the meantime build against a patched Erlang19.
I am also stuck at this bug, maybe I can test your fixed package.
In the meantime, you can take a shot at the package: https://framadrop.org/r/K9yL1VFQ_H#Mbgb7vA9xwWXz8rsBfMKo/xTeMOWyvLTDV7XbWJBw78=
But please mind the upgrade message. ;) That’s why I need to test things, to be sure that having the couchdb group owning the files (without having couchdb user in it) won’t causes issues.
The patch is a straightforward replacement of crypto:rand_bytes with crypto:strong_rand_bytes, along with a single addition of:
{module, crypto} = code:ensure_loaded(crypto)
to src/couch_epi/src/couch_epi_util.erl.
I'm not sure if code:ensure_loaded() is also needed elsewhere; I only patched the place causing the startup failure. Upstream will know how to check if the fix is incomplete.
For now I’ll just push a CouchDB version embedding Erlang 19.3. ;) I’m doing the testing right now, should be online soon.