FS#56393 - [elasticsearch] Not starting

Attached to Project: Community Packages
Opened by Francois (francoism90) - Monday, 20 November 2017, 20:19 GMT
Last edited by Massimiliano Torromeo (mtorromeo) - Wednesday, 29 November 2017, 09:45 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Massimiliano Torromeo (mtorromeo)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:
$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-11-20 21:16:09 CET; 1min 13s ago
Docs: http://www.elastic.co
Process: 15228 ExecStart=/usr/bin/elasticsearch -d -p /run/elasticsearch/elasticsearch.pid (code=exited, status=0/SUCCESS)
Main PID: 15250 (code=exited, status=1/FAILURE)

Nov 20 21:16:09 archie systemd[1]: elasticsearch.service: Service hold-off time over, scheduling restart.
Nov 20 21:16:09 archie systemd[1]: elasticsearch.service: Scheduled restart job, restart counter is at 5.
Nov 20 21:16:09 archie systemd[1]: Stopped Elasticsearch.
Nov 20 21:16:09 archie systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Nov 20 21:16:09 archie systemd[1]: Failed to start Elasticsearch.

Additional info:
* elasticsearch 6.0.0-1
* clean install, removed all files, logs, etc.


Steps to reproduce:
1. Install elasticsearch
2. Start with systemctl
3. Crash after a few seconds
This task depends upon

Closed by  Massimiliano Torromeo (mtorromeo)
Wednesday, 29 November 2017, 09:45 GMT
Reason for closing:  Fixed
Additional comments about closing:  elasticsearch-6.0.0-3
Comment by Francois (francoism90) - Monday, 20 November 2017, 20:21 GMT
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: activating (start) since Mon 2017-11-20 21:20:37 CET; 80ms ago
Docs: http://www.elastic.co
Main PID: 16331 (code=exited, status=1/FAILURE); Control PID: 16356 (elasticsearch)
Tasks: 10 (limit: 4915)
CGroup: /system.slice/elasticsearch.service
├─16356 /bin/bash /usr/bin/elasticsearch -d -p /run/elasticsearch/elasticsearch.pid
└─16357 /usr/lib/jvm/default-runtime/bin/java -cp /usr/share/elasticsearch/lib/* org.elasticsearch.tools.JavaVersionChecker

Nov 20 21:20:37 archie systemd[1]: Starting Elasticsearch...
$ /usr/lib/jvm/default-runtime/bin/java -cp /usr/share/elasticsearch/lib/* org.elasticsearch.tools.JavaVersionChecker
Error: Could not find or load main class .usr.share.elasticsearch.lib.HdrHistogram-2.1.9.jar
Comment by Francois (francoism90) - Monday, 20 November 2017, 20:33 GMT
Downgrade to elasticsearch-5.6.4-1-any.pkg.tar.xz fixes the problem.
Think this is something in the java properties.
Comment by Christian (close2) - Wednesday, 22 November 2017, 12:09 GMT
One of the last commits on the package removed:
- sed -re 's;#\s*(path\.conf:).*$;\1 /etc/elasticsearch;' \
- -e '0,/#\s*(path\.data:).*$/s;;\1 /var/lib/elasticsearch;' \
- -e 's;#\s*(path\.work:).*$;\1 /tmp/elasticsearch;' \
- -e 's;#\s*(path\.logs:).*$;\1 /var/log/elasticsearch;' \
- -i config/elasticsearch.yml

https://git.archlinux.org/svntogit/community.git/commit/trunk/PKGBUILD?h=packages/elasticsearch&id=99a633f618dca8cc7fd99725f19832b80f759e07

As the elasticsearch.service systemd file specifies:
WorkingDirectory=/usr/share/elasticsearch



elasticsearch tries to create log and data files in this directory. However /usr/share/elasticsearch is owned by root (and in any case no the right place to put those files.)
Comment by Levente Polyak (anthraxx) - Wednesday, 22 November 2017, 12:51 GMT
the default config should be restored to be altered according Arch Linux FHS and indeed use /var/log f.e. its totally the wrong place to persist logs in /usr/share/
Comment by Massimiliano Torromeo (mtorromeo) - Thursday, 23 November 2017, 15:19 GMT
Elasticsearch will always try to use /usr/share/elasticsearch/logs if not configured properly so the older package provided a default configuration that explicitly stated that the default logs path should be /var/log/elasticsearch but if you tried to start the service with a different configuration you would end up with logs in /usr/share

Instead of relying on a specific configuration being present I now symlinked the logs directory to var/log so that it will work even if you don't have path.logs configured in elasticsearch.yml.
Comment by Francois (francoism90) - Monday, 27 November 2017, 16:01 GMT
Still crashes after a restart on 6.0.0-2:
$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-11-24 19:30:19 CET; 24ms ago
Docs: http://www.elastic.co
Process: 1488 ExecStart=/usr/bin/elasticsearch -d -p /run/elasticsearch/elasticsearch.pid (code=exited, status=0/SUCCESS)
Main PID: 1742 (code=exited, status=1/FAILURE)

Nov 24 19:30:19 nas1 systemd[1]: elasticsearch.service: Service hold-off time over, scheduling restart.
Nov 24 19:30:19 nas1 systemd[1]: elasticsearch.service: Scheduled restart job, restart counter is at 5.
Nov 24 19:30:19 nas1 systemd[1]: Stopped Elasticsearch.
Nov 24 19:30:19 nas1 systemd[1]: elasticsearch.service: Start request repeated too quickly.
Nov 24 19:30:19 nas1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Nov 24 19:30:19 nas1 systemd[1]: Failed to start Elasticsearch.
Comment by Massimiliano Torromeo (mtorromeo) - Tuesday, 28 November 2017, 10:00 GMT
@francoism90 please check the logs in /var/logs/elasticsearch
Comment by Ian Hunter (webnard) - Tuesday, 28 November 2017, 23:00 GMT
I'm getting the same issue. Elasticsearch is trying to do something with a directory that doesn't exist. I've tried uninstalling and reinstalling, but the package doesn't create the directory

java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.0.0.jar:6.0.0]
Comment by Christian (close2) - Wednesday, 29 November 2017, 09:15 GMT
if the package doesn't want to change the default configuration, at least the data path requires another symbolic link to /var/lib/elasticsearch
As previous config files also configured the path.work a symbolic link to /tmp/elasticsearch might be required as well.

@webnard a quick fix is to change your /etc/elasticsearch/elasticsearch.yaml and change:
path.conf: /etc/elasticsearch
path.data: /var/lib/elasticsearch
path.work: /tmp/elasticsearch
path.logs: /var/log/elasticsearch
Comment by Massimiliano Torromeo (mtorromeo) - Wednesday, 29 November 2017, 09:31 GMT
@close2: that is correct. I'll add the missing symlink.
I don't think path.work is used anymore by elasticsearch though.

Loading...