Follow

Error outputted in Flume log "_Connection refused_ _SEARCH_NODE_FQDN_XXX.XXX.XXX_9300_"

Issue

The following error can be outputted in Flume log(s):

  • Connection refused: <SEARCH_NODE_FQDN>/XXX.XXX.XXX:9300

The error can be outputted in the following log(s):

  • flume-interset_<ds>_events_<did>_<tid>_es.log

Cause

This error is caused by the Elasticsearch nodes not being reachable when Flume is attempting to write data to a new interset rawdata index.

Resolution Steps

NOTE: This information is only useful for CSV data ingest using Flume 

Validate port accessibility

  1. SSH to the STREAM NODE(s) as the Interset User
  2. Type in the following command to telnet to each Elasticsearch node and attempt to connect to port 9300:
    • sudo telnet <SEARCH_NODE_FQDN> <PORT>
      • EXAMPLE: sudo telnet search.acme.com 9300
  3. If telnet connection succeeds or fails, please continue to the next step.

Validate firewall status

Check if iptables/ip6tables/firewalld is running on each STREAM and SEARCH NODE.

  1. SSH on to STREAM, and SEARCH NODE(s) as the Interset User
  2. Type in the following command to check if iptables/ip6tables/firewalld daemon is running:
    • For EL6:
      • sudo service iptables status
      • sudo service ip6tables status
    • For EL7:
      • sudo systemctl status firewalld
      • sudo systemctl status iptables
  3. If iptables/ip6tables/firewalld is running, type in the following command to stop the daemon(s).
    1. NOTE: Please verify with your network security team if this is appropriate
      • For EL6:
        1. sudo service iptables stop
        2. sudo service ip6tables stop
      • For EL7:
        • sudo systemctl stop firewalld
        • sudo systemctl stop iptables
  4. If iptables/ip6tables/firewalld is NOT running, please validate if there is a firewall in between the STREAM, and SEARCH NODE(s).
  5. If there is a firewall, please allow a bi-directional connection between the STREAM, and SEARCH NODE(s) on port 9300.

Below are two options that validates/resolves the SEARCH NODE FQDN and port issues:

  • Validate SEARCH NODE FQDN(s) in Flume configuration
  • Regenerate Flume configuration

NOTE: If “Validate Elasticsearch cluster name in Flume configuration” option becomes too difficult, it is recommended to regenerate the Flume config to minimize mistakes.

Validate SEARCH NODE FQDN(s) in Flume configuration

  1. Open up a web browser and navigate to the Ambari UI URL:
  2. Log in to the Ambari UI as the Ambari admin. The default credentials for the Ambari admin user are as follow:
    • Username: admin
    • Password: admin
  3. Once logged in, click on Flume (from component list)
  4. In Flume, click on the Configs tab
  5. Click the Groups dropdown and select Ingest
  6. Under the flume.conf section, copy the Flume configuration to a text editor.
  7. In the text editor, look for the following parameters:
    • sinks.esSink.hostNames
    • sinks.esSink.serializer.hostNames
    • sinks.esSink.hostNames
    • sinks.esSink.serializer.hostNames
  8. Modify/validate the value specified for each parameter set to the SEARCH NODE FQDN along with the proper port.
    • EXAMPLE: search.acme.com:9300
      • NOTE: If there are multiple SEARCH NODES, the value for each parameter will resemble the following:
        • acme.com:9300,search2.acme.com:9300
  9. After the SEARCH NODE FQDN(s) value is/are modified/validated, copy the flume configuration from the text editor and replace the flume configuration in the flume.conf section in the Ambari UI.
  10. Once the configuration has been replaced, click Save to continue, and then OK.
  11. Click the Restart button at top of the webpage and select Restart All Affected.
  12. Click Confirm Restart All to restart Flume and use the new configuration
  13. Click OK once Flume restarts successfully

Regenerate Flume configuration

  1. To regenerate the Flume ingest configuration, please see the Configure a New Data Source section in the Interset <version> Installation and Configuration guide.

Validate Elasticsearch cluster

  1. SSH to the SEARCH NODE as the Interset User.
  2. Type in the following command to validate if Elasticsearch responds name as it will be needed in the following step:
    • curl -ks -X GET http<s>://<SEARCH_NODE_FQDN>:9200/_cluster/health?pretty
  3. The curl command will return one of the following:

    Cluster is up

    Cluster is down

      "cluster_name" : "interset",

      "status" : "green",

      "timed_out" : false,

      "number_of_nodes" : 3,

      "number_of_data_nodes" : 2,

      "active_primary_shards" : 96,

      "active_shards" : 192,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0,

      "delayed_unassigned_shards" : 0,

      "number_of_pending_tasks" : 0,

      "number_of_in_flight_fetch" : 0,

      "task_max_waiting_in_queue_millis" : 0,

      "active_shards_percent_as_number" : 100.0

      "error" : {

        "root_cause" : [

          {

            "type" : "master_not_discovered_exception",

            "reason" : null

          }

        ],

        "type" : "master_not_discovered_exception",

        "reason" : null

      },

      "status" : 503

  4. If the cluster is up, please see the Check Flume logs section.
  5. If the cluster is down, this indicates that one or more Elasticsearch node is either NOT running or is in an unresponsive state. Please continue to the next step.

Validate Elasticsearch node(s)

Validate each Elasticsearch node is up and running. Please follow the steps below:

  1. SSH to the SEARCH NODE(s) as the Interset User.
  2. Type in the following command to validate the status of Elasticsearch:
    • sudo systemctl status elasticsearch
  3. If Elasticsearch is not running, please type in the following command to start Elasticsearch:
    • sudo systemctl start elasticsearch
  4. Type in the following command to view the interset.log in the Elasticsearch log (/var/log/elasticsearch) directory:
    • sudo less /var/log/elasticsearch/interset.log
  5. In the interset.log hit the follow key combination jump to the end of the log:
    • Shift + G
  6. When a node starts up properly, the following will be outputted in the interset.log file:
    • [20XX-XX-XXT09:39:49,674][INFO ][o.e.h.HttpServer ] [<SEARCH_NODE_FQDN>] publish_address {<SEARCH_NODE_IP>:9200}, bound_addresses {<SEARCH_NODE_IP>:9200}
    • [20XX-XX-XXT09:39:49,674][INFO ][o.e.n.Node ] [<SEARCH_NODE_FQDN>] started
      • NOTE: The Elasticsearch node that is chosen as the master for the Elasticsearch cluster will have the following outputted in the interset.log:
        • [20XX-XX-XXT09:39:49,080][INFO ][o.e.c.s.ClusterService ] [<SEARCH_NODE_FQDN>] master {new {<SEARCH_NODE_FQDN>}{XzM9TSt3Ql62Pp6Y9WHW1A}{uEFlJWWNQDu8b5F-B84eiA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}}, removed {{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{6szmJfWCSAyQqp8nS-SYbw}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300},}, added {{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{yA2X3jZJTECJrrJMC6PSDA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300},}, reason: zen-disco-elected-as-master ([2] nodes joined)[{<SEARCH_NODE_FQDN>}{bS8-wLAzTma9gAeDFRAQwg}{yA2X3jZJTECJrrJMC6PSDA}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}, {<SEARCH_NODE_FQDN>}{RYYfDVzETQCcx-cBAsDInw}{77ZlAVlNTfuHYjBgLXToLw}{<SEARCH_NODE_FQDN>}{<SEARCH_NODE_IP>:9300}]
        • [20XX-XX-XXT09:40:18,357][INFO ][o.e.c.r.a.AllocationService] [<SEARCH_NODE_FQDN>] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[entity_stats_0_20XX-XX-XX_00:51:13][3]] ...]).
  7. After all Elasticsearch nodes are up and running, type in the following command to verify if the cluster is up and running:
    • curl -ks -X GET http<s>://<SEARCH_NODE_FQDN>:9200/_cluster/health?pretty
  8. The output will be similar to the following:
    • "cluster_name" : "interset",
    • "status" : "<green/yellow/red>",
    • "timed_out" : false,
    • "number_of_nodes" : 3,
    • "number_of_data_nodes" : 2,
    • "active_primary_shards" : 96,
    • "active_shards" : 192,
    • "relocating_shards" : 0,
    • "initializing_shards" : 0,
    • "unassigned_shards" : 0,
    • "delayed_unassigned_shards" : 0,
    • "number_of_pending_tasks" : 0,
    • "number_of_in_flight_fetch" : 0,
    • "task_max_waiting_in_queue_millis" : 0,
    • "active_shards_percent_as_number" : 100.0

Check Flume logs

  1. SSH to the STREAM NODE(s) as the Interset User.
  2. Type in the following command to navigate to the /var/log/flume directory:
    • cd /var/log/flume
  3. Type in the following to view the Flume log file:
    • sudo less flume-interset_<ds>_events_<did>_<tid>_es.log
  4. In the log file, hit the follow key combination jump to the end of the log:
    • Shift + G
  5. Look for the “EventPutSuccessCount”. This value should keep incrementing until data ingested completed.

Applies To

  • Interset 5.4.x or higher 
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk