Follow

The following is outputted in the Elasticsearch log (interset.log) "java.net.NoRouteToHostException: No route to host"

Issue

The following error is outputted in the Elasticsearch log (interset.log):

[2018-02-23 13:16:04,980][WARN ][transport.netty] [<SEARCH_NODE_FQDN>] exception caught on transport layer [[id: 0x89a8fb5b]], closing connection
java.net.NoRouteToHostException: No route to host

  • at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) 

Cause

This is most likely caused by a firewall that is blocking connection to other nodes in the Elasticsearch cluster. 

Resolution Steps

The steps below needs to be performed on each Elasticsearch node to ensure the firewall is disabled or firewall rule is opened/allowed.

Verify iptables/ip6tables/firewalld status log (interset.log)

  1. SSH to the SEARCH NODE(s) as the Interset User
  2. Type in the following command to verify if the firewall is running:
    • For EL6
      • sudo service iptables status
      • sudo service ip6tables status
    • For EL7
      • sudo systemctl status firewalld
  3. If the firewall is enabled, please either:
    • Turn off and disable iptables/ip6tables/firewalld
      • For EL6
        • sudo chkconfig iptables off
        • sudo chkconfig ip6tables off
        • sudo service iptables stop
        • sudo service ip6tables stop
      • For EL7
        • sudo systemctl stop firewalld
        • sudo systemctl disable firewalld
    • Modify the iptables/ip6tables/firewalld to allow inbound/outbound to the following tcp port range:
      • TCP Port 9200-9400 (inclusive)
  4. Once Step 3 is completed, it is prudent to restart each Elasticsearch service that is outputting the NoRouteToHostException. Type in the following command restart each the Elasticsearch service:
    • For EL6
      • sudo service elasticsearch stop
      • sudo service elasticsearch start
    • For EL7 
      • sudo systemctl stop elasticsearch
      • sudo systemctl start elasticsearch
  5. Type in the following command to view the interset.log in the Elasticsearch log (/var/log/elasticsearch) directory:
    • sudo less /var/log/elasticsearch/interset.log
  6. In the interset.log hit the follow key combination jump to the end of the log:
    • Shift + G
  7. When a node starts up properly, the following will be outputted in the interset.log file:
    • [20XX-XX-XXT09:39:49,674][INFO ][o.e.h.HttpServer ] [<SEARCH_NODE_FQDN>] publish_address {<SEARCH_NODE_FQDN>:9200}, bound_addresses {<SEARCH_NODE_FQDN>:9200}
    • [20XX-XX-XXT09:39:49,674][INFO ][o.e.n.Node ] [<SEARCH_NODE_FQDN>] started
      • NOTE: The Elasticsearch node that is chosen as the master for the Elasticsearch cluster will have the following outputted in the interset.log: 
        • [20XX-XX-XXT09:39:49,080][INFO ][o.e.c.s.ClusterService ] [<SEARCH_NODE_FQDN>] master {new {<SEARCH_NODE_FQDN>}{XzM9TSt3Ql62Pp6Y9WHW1A}{uEFlJWWNQDu8b5F-B84eiA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}}, removed {{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{6szmJfWCSAyQqp8nS-SYbw}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300},}, added {{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{yA2X3jZJTECJrrJMC6PSDA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300},}, reason: zen-disco-elected-as-master ([2] nodes joined)[{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{yA2X3jZJTECJrrJMC6PSDA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}, {<SEARCH_NODE_FQDN>}{RYYfDVzETQCcx-cBAsDInw}{77ZlAVlNTfuHYjBgLXToLw}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}]
        • [20XX-XX-XXT09:40:18,357][INFO ][o.e.c.r.a.AllocationService] [<SEARCH_NODE_FQDN>] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[<index_name>][3]] ...]).
  8. After all Elasticsearch nodes are up and running, type in the following command to verify if the cluster is up and running:
    • curl -ks -X GET http<s>://<SEARCH_NODE_FQDN>:9200/_cluster/health?pretty
  9. The output will be similar to the following:
    • {
    •   "cluster_name" : "interset",
    •   "status" : "green",
    •   "timed_out" : false,
    •   "number_of_nodes" : 4,
    •   "number_of_data_nodes" : 3,
    •   "active_primary_shards" : 152,
    •   "active_shards" : 304,
    •   "relocating_shards" : 0,
    •   "initializing_shards" : 0,
    •   "unassigned_shards" : 0,
    •   "delayed_unassigned_shards" : 0,
    •   "number_of_pending_tasks" : 0,
    •   "number_of_in_flight_fetch" : 0,
    •   "task_max_waiting_in_queue_millis" : 0,
    •   "active_shards_percent_as_number" : 100.0
    • }

Applies To

  • Interset 5.4.x or higher
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments