Issue
The following error can be outputted in Flume log(s):
- XX Jan 20XX XX:XX:XX,175 WARN [<kafka_consumer_group>_<STREAM_NODE_FQDN>-1516733223919-769f069b-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - Fetching topic metadata with correlation id <#> for topics [Set(<Kafka_datasource_event_topic>)] from broker [BrokerEndPoint(1001,<STREAM_NODE_FQDN>,6667)] failed
- java.nio.channels.ClosedChannelException
- at kafka.network.BlockingChannel.send(BlockingChannel.scala:122)
- at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
- at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
- at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
- at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
- at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
- at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
- at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
The error above are outputted in the following log(s):
- flume-interset_<ds>_events_<did>_<tid>_csv_transform.log
- flume-interset_<ds>_events_<did>_<tid>_es.log
- flume-interset_<ds>_events_<did>_<tid>_hbase.log
- flume-interset_<ds>_raw_<did>_<tid>_csv_multiline_extract.log
Cause
The cause is due to the Kafka Broker not running within Ambari.
Resolution Steps
NOTE: This information is only useful for CSV data ingest using Flume
There are two steps to complete in order to resolve this issue. They are:
- Validate Kafka component is running
- Validate Flume logs
Validate Kafka component is running
- Open up a web browser and navigate to the Ambari UI URL:
- EXAMPLE: http://ambari.acme.com:8080
- Log in to the Ambari UI as the Ambari admin. The default credentials for the Ambari admin user are as follow:
- Username: admin
- Password: admin
- Once logged in, click on Kafka (from component list)
- Ensure the Kafka Broker is Started under the Summary section
- If it is NOT running, please click Service Actions > Start > Confirm Start
- Once the Kafka Broker is started, please continue to Validate Flume Logs section
Verify Flume logs
- SSH to the STREAM NODE as the Interset User.
- Type in the following to navigate to the flume log (/var/log/flume) directory:
- cd /var/log/flume
- The following Flume logs will need to be validated in order to validate the issue has been resolved, and data is being ingested successfully:
- flume-interset_<ds>_events_<did>_<tid>_csv_transform.log
- flume-interset_<ds>_events_<did>_<tid>_es.log
- flume-interset_<ds>_events_<did>_<tid>_hbase.log
- flume-interset_<ds>_raw_<did>_<tid>_csv_multiline_extract.log
- NOTE:
- <ds> - denotes the data source type ingested (i.e. auth, repo. webproxy)
- <did> - denotes the data source instance ID the data is ingested to
- <tid> - denotes the tenant ID the data is ingested for
- NOTE:
- To view the respective log information, see the example below to view the respective Flume log:
- sudo less flume_interset_<log_name>.log
- In the log file, hit the follow key combination jump to the end of the log:
- Shift + G
- In the respective Flume log, look for “EventPutSuccessCount”. This value will keep incrementing until data ingested completed.
- NOTE: This is not applicable if it is streaming/real time data
Applies To
- Interset 5.4.x or higher
Comments