Follow

How to Interpret Flume Logs

Note that all metrics are only stored for the lifetime of the agent. When it is stopped the metrics are reset.

Run-time Metrics

While Flume is running, it will update its .log file(s) with various metrics written out at 10s intervals. The values below are useful for diagnosing common issues. Other metrics do exist, but are not documented here as they are not generally useful.

  • BatchEmptyCount - This value represents the number of attempts made to pull data from Kafka (or whatever Source is configured) that were empty (i.e. no data was retrieved)
  • EventReceivedCount - Similar to the above, this will note how many events have been received.
  • ChannelCapacity - This is a configurable parameter, however the default size we ship with is 20,000. This is the number of records Flume will store in memory in its "channel", which is a holding area between the source (Kafka) and sink (ES or HBase).
  • ChannelFillPercentage - Building off of the value above, this notes the percentage of the channel that is actively in use. If this percentage is high, it suggests that the sinks are either too slow (in which case you may see accompanying warnings/errors), or the sink is misconfigured/not available.
  • EventTakeSuccessCount - This notes how many events the Sink configuration has been able to pull off of the channel.
  • EventPutAttemptCount - This notes the total number of attempts that have been made to push a record to the Sink target (e.g. HBase).
  • EventPutSuccessCount - This notes the total number of successful attempts that have been made to push a record to the Sink target (e.g. HBase).

Shutdown Metrics

When an Interset Flume agent is stopped it will write out various metrics with relation to its interceptors. These can be vital pieces of information as this may note where events are being dropped or provide insight into malformed records, etc...

The information below is not exhaustive, but is provided to be able to better understand what is being presented. This will focus on the metrics written on the shutdown of an agent writing to HBase as this tends to be the most frequent problem in the field:

 

12 Aug 2019 12:21:37,902 INFO [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle supervisor 13
12 Aug 2019 12:21:37,905 INFO [agent-shutdown-hook] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop:83) - Configuration provider stopping
12 Aug 2019 12:21:37,905 INFO [agent-shutdown-hook] (com.interset.flume.metric.SimpleMetric.printWithLogger:71) - AsyncMetricsSerializer: No timestamps.
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (com.interset.flume.metric.AbstractMetric.printWithLogger:87) - AsyncMetricsSerializer: Count of 'Events Created 0 Relations': 0
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (com.interset.flume.metric.AbstractMetric.printWithLogger:87) - AsyncMetricsSerializer: Count of 'Events Created At Least 1 Relation': 0
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (com.interset.flume.metric.AbstractMetric.printWithLogger:87) - AsyncMetricsSerializer: Count of 'Events In': 0
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (com.interset.flume.metric.AbstractMetric.printWithLogger:87) - AsyncMetricsSerializer: Count of 'Relations Out': 0
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (com.interset.relations.logging.AbstractMetric.printWithLogger:77) - RelationGenerator - Simple: No counts.
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (com.interset.relations.logging.AbstractMetric.printWithLogger:77) - RelationGenerator - Detailed: No counts.
12 Aug 2019 12:21:37,912 INFO [agent-shutdown-hook] (org.apache.flume.sink.hbase.AsyncHBaseSink.shutdownHBaseClient:553) - Shutting down HBase Client

To break this down into specific items:

  • Events Created 0 Relations - This notes the number of events that were available to the HBase Sink but were effectively dropped due to missing information. This could be missing fields, malformed timestamps, etc...
  • Events Created At Least 1 Relation - This is the number of events actually ingested into HBase
  • Events In - Total number of events received by the Sink.
  • Relations Out - Total number of events written to HBase by the Sink.
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments