Analytics failed and following error is outputted in the analytics.log "org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list_ /:16020"

Issue

Analytics failed, and the following error is outputted in the analytics.log:

• XX/XX/XX XX:XX:XX INFO Client:
  • client token: N/A
  • diagnostics: User class threw exception: java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
• XXX Feb XX XX:XX:XX EST 20XX, RpcRetryingCaller{globalStartTime=<epoch_time>, pause=100, retries=35}, java.net.ConnectException: Connection refused
• XX Feb XX XX:XX:XX EST 20XX, RpcRetryingCaller{globalStartTime=<epoch_time>, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: <COMPUTE_NODE_FQDN>/<COMPUTE_NODE_IP>:16020

Cause

The error is outputted when YARN is attempting to connect to the HBase RegionServer(s) and it receives a connection refused.

Resolution Steps

Following steps are taken:

Check RegionServer(s) in Ambari

1. Open up a web browser and navigate to the Ambari UI URL:
   • EXAMPLE: http://ambari.acme.com:8080
2. Log in to the Ambari UI as the Ambari admin. The default credentials for the Ambari admin user are as follow:
   • Username: admin
   • Password: admin
3. Once logged in, on the left side, click on HBase
4. In HBase, click the RegionServers link in the Summary section
5. A list of nodes are displayed which has the RegionServer component installed.
6. For each node in the list, please perform the following:
   • Click a NODE FQDN link under the Name column
   • Look for RegionServer / HBase, click the drop down and select Start
   • Click OK on the Confirmation pop up.
   • Click OK once Start RegionServer succeeds. 

Validate RegionServer is up

1. SSH to the COMPUTE NODE as the Interset User
2. Type in the following command to verify that the HBase RegionServer is started:
   • sudo ps -ef | grep regionserver
3. Locate the following and make note of the Process ID that is assigned:
   • /usr/java/jdk1.8.0_131/bin/java -Dproc_regionserver
4. Type in the following command to look for the Process ID that is binding to port 16020:
   • sudo netstat -talpn | grep 16020 | awk '{print $7}' NR==1
5. Ensure the output from both step 3 and 4 are the same Process ID

Applies To

• Interset 5.4.x or higher