You may encounter a scenario where Investigator, despite data existing, will show no results and Kibana will state that it cannot find any Indicies. This may be caused by Elasticsearch hitting an Out of Memory condition.
To confirm this, search the Elasticsearch log (/var/log/elasticsearch/<cluster_name>.log for "java.lang.OutOfMemoryError".
This can be caused by the default heap size setting of 1GB. The recommendation is that Elasticsearch have half of the available system memory, but no more than 31GB. For example, on a system with 32GB of RAM Elasticsearch's heap size should be 16GB, on a system with 48GB it should have a 24GB heap, on a system with 96GB of it should have a 31GB heap.
This setting is in the /etc/sysconfig/elasticsearch file, in a variable called ES_HEAP_SIZE. This is commented out by default.
To fix this, perform the following steps on all nodes where Elasticsearch is installed (e.g. Search nodes):
- Open /etc/sysconfig/elasticsearch in an editor
- Uncomment ES_HEAP_SIZE and change the value to half of the system memory, but no more than 31GB (e.g. ES_HEAP_SIZE=8g).
- Restart Elasticsearch (service elasticsearch restart)
IMPORTANT NOTE: This same behaviour will be seen, without the OutOfMemory error, on initial installation of a system. In that scenario, note that data must be ingested and Analytics must be run before you will be able to use Investigator and Search normally.