A situation arose when installing Interset in Google Cloud where clicking on Raw Events resulted in the Kibana tab opening up but it was not connecting to Elasticsearch. The Kibana logo just sat there on the screen but nothing else happened. The following troubleshooting steps were performed:
- An inspection of logs for Kibana, nginx, and Elasticsearch did not reveal anything out of the ordinary.
- Firewall rules in the Google Cloud were opened up to allow all ports open for ingress/egress between all nodes in the cluster.
- Firewalld confirmed to be disabled.
- SE Linux in permissive mode on all nodes.
This was a two node ES cluster. Due to the timing of the POC, goal was to just establish connectivity as root cause isolation was elusive and taking too long. This install was using default proxy settings which proved to be the issue, although specific issue with proxy settings was not fully resolved. As a result, Kibana settings were adjusted to just point directly to one of the Search nodes instead of using the proxy. Here are the settings in /etc/kibana/kibana.yml which were changed in order to do this:
- server.host: "localhost"
- server.name: "<master node FQDN>"
- elasticsearch.url: "http://<Search node FQN>:9200"
Root cause is still unknown, but most likely has something to do with Google Cloud networking implementation, as this is the first time it has been seen. Never been an issue for AWS, physical environment, etc.