Follow

Cannot connect to Elasticsearch

Issue

Analytics fails because it cannot establish a connection to Elasticsearch, this is because Analytics either:

  1. Fails on sanity check and outputs an error message
    • Error message:
      • Checking elasticsearch…
      • Couldn't connect to ElasticSearch host: invalid.es.host.com
      • ElasticSearch Disk Space:
      • Couldn't connect to the given host: invalid.es.host.com
  2. Times out on sync jobs with an error message
    • Error message:
      • NoNodeAvailableException[None of the configured nodes are available: []]

Cause

There are generally a couple of causes for Analytics that fails to connect to Elasticsearch:

  • Elasticsearch node(s) are not running or refusing connection(s)
  • Elasticsearch parameters in the interset.conf is configured incorrectly

Resolution Steps

Validate Elasticsearch cluster

Validate each Elasticsearch node is up and running. Please follow the steps below:

  1. SSH to the SEARCH NODE(s) as the Interset User.
    • ssh interset@<SEARCH_NODE_FQDN>
  2. Type in the following command to see if Elasticsearch responds without errors:
    • curl -ks -X GET http<s>://<SEARCH_NODE_FQDN>:9200/_cluster/health?pretty
  3. If the command output succeeds, it will return basic information about the Elasticsearch cluster, such as the number of nodes and the name of the cluster. Please continue to “Validate interset.conf” section.
  4. If the command output fails, this indicates there are more than one Elasticsearch node(s). Please continue to “Validate Elasticsearch node(s)” section.

Validate Elasticsearch node(s)

Validate each Elasticsearch node is up and running. Please follow the steps below:

  1. SSH to the SEARCH NODE(s) as the Interset User.
  2. Type in the following command to validate the status of Elasticsearch:
    • sudo systemctl status elasticsearch
  3. If Elasticsearch is not running, please type in the following command to start Elasticsearch:
    • sudo systemctl start elasticsearch
  4. Type in the following command to view the interset.log in the Elasticsearch log (/var/log/elasticsearch) directory:
    • sudo less /var/log/elasticsearch/interset.log
  5. In the interset.log hit the follow key combination jump to the end of the log:
    • Shift + G
  6. When a node starts up properly, the following will be outputted in the interset.log file:
    • [20XX-XX-XXT09:39:49,674][INFO ][o.e.h.HttpServer ] [<SEARCH_NODE_FQDN>] publish_address {<SEARCH_NODE_IP>:9200}, bound_addresses {<SEARCH_NODE_IP>:9200}
    • [20XX-XX-XXT09:39:49,674][INFO ][o.e.n.Node ] [<SEARCH_NODE_FQDN>] started
      • NOTE: The Elasticsearch node that is chosen as the master for the Elasticsearch cluster will have the following outputted in the interset.log:
        • [20XX-XX-XXT09:39:49,080][INFO ][o.e.c.s.ClusterService ] [<SEARCH_NODE_FQDN>] master {new {<SEARCH_NODE_FQDN>}{XzM9TSt3Ql62Pp6Y9WHW1A}{uEFlJWWNQDu8b5F-B84eiA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}}, removed {{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{6szmJfWCSAyQqp8nS-SYbw}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300},}, added {{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{yA2X3jZJTECJrrJMC6PSDA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300},}, reason: zen-disco-elected-as-master ([2] nodes joined)[{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{yA2X3jZJTECJrrJMC6PSDA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}, {<SEARCH_NODE_FQDN>}{RYYfDVzETQCcx-cBAsDInw}{77ZlAVlNTfuHYjBgLXToLw}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}]
        • [20XX-XX-XXT09:40:18,357][INFO ][o.e.c.r.a.AllocationService] [<SEARCH_NODE_FQDN>] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[entity_stats_0_20XX-XX-XX_00:51:13][3]] ...]).
  7. After all Elasticsearch nodes are up and running, type in the following command to verify if the cluster is up and running:
    • curl -ks -X GET http<s>://<SEARCH_NODE_FQDN>:9200/_cluster/health?pretty
  8. The output will be similar to the following:
    • "cluster_name" : "interset",
    • "status" : "<green/yellow/red>",
    • "timed_out" : false,
    • "number_of_nodes" : 3,
    • "number_of_data_nodes" : 2,
    • "active_primary_shards" : 96,
    • "active_shards" : 192,
    • "relocating_shards" : 0,
    • "initializing_shards" : 0,
    • "unassigned_shards" : 0,
    • "delayed_unassigned_shards" : 0,
    • "number_of_pending_tasks" : 0,
    • "number_of_in_flight_fetch" : 0,
    • "task_max_waiting_in_queue_millis" : 0,
    • "active_shards_percent_as_number" : 100.0

Validate interset.conf

In the case that the Elasticsearch cluster is functioning properly, the next is to check the contents of the interset.conf file

  1. SSH to the MASTER (ANALYTICS) NODE(s) as the Interset User.
  2. Type in the following command to edit to the Interset Analytics configuration file:
    • vi /opt/interset/analytics/conf
  3. Ensure that the esClusterName value matches the cluster name set in Elasticsearch.
    • NOTE: If the Elasticsearch cluster name is unknown, executing the command from “step 2” of the “Validate Elasticsearch cluster” section which will return the “cluster_name” value that is set.
  4. If the value is incorrect, please modify the esClusterName value accordingly.
  5. Ensure that your esHost value is set the proper SEARCH NODE FQDN. If the value is incorrect, please modify the esHost accordingly
    • NOTE: If multiple esHost need to/are defined, please enter them in a comma-separated list:
      • EXAMPLE: search1.acme.com,search2.acme.com,search3.acme.com
  6. Once the the interset.conf is validated/modified, save and exit the file by typing/hit the following two commands:
    • esc
    • :wq

Applies To

  • Interset 5.4.x or higher
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments