FS#50049 - [elasticsearch] elasticsearch 2.3.4-1 'start-limit-hit' or no pid file?

Attached to Project: Community Packages
Opened by Antoni Kaniowski (amk) - Friday, 15 July 2016, 08:07 GMT
Last edited by Massimiliano Torromeo (mtorromeo) - Tuesday, 31 January 2017, 09:19 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Massimiliano Torromeo (mtorromeo)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Elasticsearch fails to start on 4.6.3-1-ARCH, systemctl status reports:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit-hit) since Fri 2016-07-15 09:58:24 CEST; 3min 16s ago
Docs: http://www.elastic.co
Process: 4201 ExecStart=/usr/bin/elasticsearch -d -p /run/elasticsearch/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.defau
Main PID: 4219 (code=exited, status=1/FAILURE)

Jul 15 09:58:23 m700 systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 15 09:58:23 m700 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jul 15 09:58:24 m700 systemd[1]: elasticsearch.service: Service hold-off time over, scheduling restart.
Jul 15 09:58:24 m700 systemd[1]: Stopped Elasticsearch.
Jul 15 09:58:24 m700 systemd[1]: elasticsearch.service: Start request repeated too quickly.
Jul 15 09:58:24 m700 systemd[1]: Failed to start Elasticsearch.
Jul 15 09:58:24 m700 systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 15 09:58:24 m700 systemd[1]: elasticsearch.service: Failed with result 'start-limit-hit'.
lines 1-15/15 (END)

Steps taken before:

pacman -S elasticsearch
systemctl enable elasticsearch
systemctl start elasticsearch

Attached are the results of "journalctl -au elasticsearch", contents of "/etc/systemd/system/multi-user.target.wants/elasticsearch.service" and "/usr/lib/tmpfiles.d/elasticsearch.conf"

Upon starting the service, there appears to be no file or folder called "elasticsearch" in "/var/run"
This task depends upon

Closed by  Massimiliano Torromeo (mtorromeo)
Tuesday, 31 January 2017, 09:19 GMT
Reason for closing:  Not a bug
Comment by Doug Newgard (Scimmia) - Saturday, 16 July 2016, 02:29 GMT
The tmpfiles entry for this package is a bit of a mess. The dirs that it lists are ones that should be part of the package, and it doesn't list the necessary /run/elasticsearch/.
Comment by Massimiliano Torromeo (mtorromeo) - Saturday, 16 July 2016, 07:00 GMT
The tmpfile doesn't need to list /run/elasticsearch because the service file uses RuntimeDirectory=elasticsearch for that [1].

The directory is created at startup and cleaned up at stop by systemd so it's normal that you don't see when the service is not running.

Also the message "PID file /run/elasticsearch/elasticsearch.pid not readable (yet?) after start: No such file or directory" is misleading and simply means that elasticsearch hasn't yet created the PID file (but will do it in time) and it is not the cause for the service failure.

You should look at logs in /var/log/elasticsearch.

[1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html
Comment by Massimiliano Torromeo (mtorromeo) - Saturday, 16 July 2016, 07:07 GMT
The fact that some directories are listed in tmpfiles doesn't mean that they are not also part of the package. tmpfiles can also be used to change ownership and permissions of file/directories and are a nice solution to avoid having to reserve a fixed uid/gid number in the package.
Comment by Doug Newgard (Scimmia) - Saturday, 16 July 2016, 13:41 GMT
Not really. You now have a bunch of untracked dirs, including in /usr. This isn't what tmpfiles is used for.
Comment by Boka Ilya (bokailya) - Thursday, 28 July 2016, 15:59 GMT
Any news?
I'm work around with ln -s /etc/elasticsearch /usr/share/elasticsearch/config
Comment by Massimiliano Torromeo (mtorromeo) - Thursday, 04 August 2016, 14:38 GMT
@bokailya I don't know what your problem is and neither do I know what is the probably different problem of @amk.
I've waited enough for more feedback and I'm now closing this as elasticsearch works fine for me on a clean installation.
Comment by Marcos Piccinini (nofxx) - Monday, 12 December 2016, 05:20 GMT
Getting same error out of fresh install.
/etc/elasticsearch has wrong permissions?
Comment by Massimiliano Torromeo (mtorromeo) - Monday, 12 December 2016, 08:58 GMT
I just tried again with a clean install and it works fine.
Did you get an error during installation?
Are you asking if the permissions are wrong or did you check?

Can you share the output of `ls -ld /etc/elasticsearch`.

What about the log files in /var/log/elasticsearch? Any relevant error?
Comment by Stef Hej (LaessigerFeuchtpudel) - Friday, 27 January 2017, 02:18 GMT
I've a fresh arch installation with java-runtime-common-2-2 jre8-openjdk-headless-8.u121-1 and elasticsearch-5.1.2-1.
After 'systemctl start elasticsearch' I got this:

Jan 27 00:41:11 linux systemd[1]: Starting Elasticsearch...
Jan 27 00:41:11 linux systemd[1]: elasticsearch.service: PID file /run/elasticsearch/elasticsearch.pid not readable (yet?) after start: No such file or directory
Jan 27 00:41:13 linux systemd[1]: elasticsearch.service: Daemon never wrote its PID file. Failing.
Jan 27 00:41:13 linux systemd[1]: Failed to start Elasticsearch.
Jan 27 00:41:13 linux systemd[1]: elasticsearch.service: Unit entered failed state.
Jan 27 00:41:13 linux systemd[1]: elasticsearch.service: Failed with result 'resources'.
Jan 27 00:41:13 linux systemd[1]: elasticsearch.service: Service hold-off time over, scheduling restart.
Jan 27 00:41:13 linux systemd[1]: Stopped Elasticsearch.
Jan 27 00:41:13 linux systemd[1]: Starting Elasticsearch...
Jan 27 00:41:13 linux systemd[1]: elasticsearch.service: PID file /run/elasticsearch/elasticsearch.pid not readable (yet?) after start: No such file or directory

Mem load increments about 1.5Gib, page file to 100%.
I am not sure if it is due to the lack of resources or the faulty creation of the PID file.
Comment by Massimiliano Torromeo (mtorromeo) - Friday, 27 January 2017, 12:37 GMT
The missing PID file is not the problem. It simply means that elasticsearch did not start and you already found a problem with a lack of memory for your workload.
You have to look at the logs in /var/log/elasticsearch for more details.

Loading...