Issue
Analytics failed, and the following error is outputted in the analytics.log:
- XX/XX/XX XX:XX:XX INFO Client:
- client token: N/A
- diagnostics: User class threw exception: java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
- XXX Feb XX XX:XX:XX EST 20XX, RpcRetryingCaller{globalStartTime=<epoch_time>, pause=100, retries=35}, java.net.ConnectException: Connection refused
- XX Feb XX XX:XX:XX EST 20XX, RpcRetryingCaller{globalStartTime=<epoch_time>, pause=100, retries=35}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: <COMPUTE_NODE_FQDN>/<COMPUTE_NODE_IP>:16020
Cause
The error is outputted when YARN is attempting to connect to the HBase RegionServer(s) and it receives a connection refused.
Resolution Steps
Following steps are taken:
Check RegionServer(s) in Ambari
- Open up a web browser and navigate to the Ambari UI URL:
- EXAMPLE: http://ambari.acme.com:8080
- Log in to the Ambari UI as the Ambari admin. The default credentials for the Ambari admin user are as follow:
- Username: admin
- Password: admin
- Once logged in, on the left side, click on HBase
- In HBase, click the RegionServers link in the Summary section
- A list of nodes are displayed which has the RegionServer component installed.
- For each node in the list, please perform the following:
- Click a NODE FQDN link under the Name column
- Look for RegionServer / HBase, click the drop down and select Start
- Click OK on the Confirmation pop up.
- Click OK once Start RegionServer succeeds.
Validate RegionServer is up
- SSH to the COMPUTE NODE as the Interset User
- Type in the following command to verify that the HBase RegionServer is started:
- sudo ps -ef | grep regionserver
- Locate the following and make note of the Process ID that is assigned:
- /usr/java/jdk1.8.0_131/bin/java -Dproc_regionserver
- Type in the following command to look for the Process ID that is binding to port 16020:
- sudo netstat -talpn | grep 16020 | awk '{print $7}' NR==1
- Ensure the output from both step 3 and 4 are the same Process ID
Applies To
- Interset 5.4.x or higher
Comments