Follow

Post-Ingest Sanity Checking

Once it is believed that data has been ingested into the system, there are two quick sanity checks that can be performed that do not require a full run of Analytics.

 

  1. Go to Kibana (http://<REPORTING_NODE>/search), select (or create) the index associated with the relevant data type (e.g. interset_a*d_rawdata_0 for authentication data), and confirm that data exists for the time period in question.
  2. Launch /opt/interset/analytics/bin/sql.sh, and run the following query -

    SELECT MAX(DATEMINUTE) FROM OBSERVED_ENTITY_RELATION_MINUTELY_COUNTS; 

    This will confirm the latest timestamp in the underlying data set that ha been ingested. Similarly, this can be run with MIN in place of MAX to return the earliest timestamp.

 

If data is not present in these locations, the administrator will need to look through the Flume logs to see confirm where issues have occurred. In the examples below, we will assume that we are using a CSV source of repository data, and the data has not made it into Elasticsearch (Kibana) or HBase (sql.sh).

 

  1. Confirm if the file(s) in the ingest directory (e.g. /tmp/ingest) have been renamed to have a .COMPLETED extension. If they have not the /var/log/flume/flume-interset_repo_events_<DID>_<TID>_csv.log file should be reviewed for exceptions.
  2. If the file has been renamed, the /var/log/flume/flume-interset_repo_events_<DID>_<TID>_es.log and /var/log/flume/flume-interset_repo_events_<DID>_<TID>_hbase.log files should be reviewed for exceptions, however it is possible that a third issue has occurred.
  3. If the file is renamed to .COMPLETED and neither the ES or HBase logs show exceptions, the CSV log (e.g. /var/log/flume/flume-interset_repo_events_<DID>_<TID>_csv.log) should still be reviewed, as it is possible that records are being dropped due to missing key values, or a configuration issue around the available fields and/or data types available in the data.

Lastly, it is important to note that a user can also stop Flume, via Ambari, to force the log files noted above to print out all of their run time metrics as one of the final messages in the log. This information can be used to review record counts in terms of in/out through the Flume to further identify where problems may be occurring.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk