How do I know if Analytics has succeeded or failed?
This document will provide a method to verify if Analytics has succeeded. Moreover, if Analytics has failed, this KB article will outline how to identify where the failure occurred and extract the relevant error so that it can searched on Interset Knowledgebase for resolution steps, or alternatively, send the to Interset Support (firstname.lastname@example.org)
The following nodes will need to be accessed (via web or SSH to the respective nodes):
- AMBARI (web)
- MASTER (ANALYTICS) (SSH)
- SSH to the MASTER (ANALYTICS) NODE as the Interset User
- Type in the following to navigate to the Interset log (/opt/interset/log) directory:
- cd /opt/interset/log
- In the log directory, there is an analytics.log which contains the Analytics spark jobs that are submitted to spark to execute. To verify whether or not analytics has completed successfully, type in the following command:
- cat analytics.log | grep -B 3 -A 3 finished
- If the command above returns a value, then it means that Analytics has completed successfully.
- If the command above does not return a value,, type in the following command to view the analytics.log:
- less analytics.log
- Look through the log and identify where the failure occurred. Normally, the error outputted will be a stack trace error. Please examine the error outputted and search the stack trace error with in the Interset Knowledgebase.
- If the stack trace error is not available in the Interset Knowledgebase, please continue to next section to investigate the YARN ResourceManagerUI.
For more information on the Analytics jobs, please see the section below.
Check YARN ResourceManagerUI
- Open up a web browser and navigate to the Ambari UI URL:
- EXAMPLE: http://ambari.acme.com:8080
- Log in to the Ambari UI as the Ambari admin. The default credentials for the Ambari admin user are as follow:
- Username: admin
- Password: admin
- Once logged in, click on YARN (from components list)
- In YARN, click on Quick Links > ResourceManager UI
- The ResourceManager UI lists all Analytics spark jobs that have been submitted to the cluster in chronological order as well as their current state and final status.
- Below is a list of jobs that constitute a complete analytics run (in order and by category):
- Total: 29 Jobs
If a spark job failed, please continue to the next section to investigate the cause of the failure.
Check for Analytics failure
- If a specific job fails, the ResourceManager UI will mark the State and FinalState as FAILED.
- In this instance, in the ResourceManager UI, click on FAILED, under the Cluster pane. This will filter only on FAILED jobs.
- Note the FAILED job names under the Names column, as each job in the latest Analytics run should be investigated accordingly. Each job name are appended with batch/windowed and tenant ID.
- EXAMPLE: com.interset.analytics.scoring.EntityVulnerabilityJob-batch-<myTID>
- To explore the logs for the FAILED job, In the ResourceManager UI that is filtered by FAILED jobs, click on the link in the ID column that corresponds to the FAILED job.
- Scroll down to the bottom of the page, and a list of job attempts (Attempt ID column), along with node association are listed. Each attempt will have logs associated under the Logs column.
- To investigate further on why the jobs failed, one must click on the Logs link under the Logs column.
- NOTE: If more than one attempt is listed, please click the Logs link for the latest attempt.
- The Logs page aggregates different log types. With a failing job, the best place to start is looking at the stderr log.
- In the Logs page, scroll down and look for the following:
- Log Type: stderr
- Three lines below, it will display Showing XXXX bytes of XXXX total. Click here for the full log. Click on the here link.
- Scroll down in the stderr log until a stack trace error is thrown. Please examine the error outputted and search the stack trace error with in the Interset Knowledgebase.
- If no KBs are found, please contact Interset Support (email@example.com).
- Interset 5.4.x or higher