Follow

Enable Spark History auto cleanup

 

Question

How do I enable Spark History auto cleanup?

Summary

This "How to" article provides the steps on how to enable Spark History auto cleaning.

Steps

  1. Open up a web browser, and navigate to the AMBARI UI log in page. The URL default is:
    • http://<AMBARI_NODE_FQDN>:8080
      • NOTE: If HTTPS is enabled to AMBARI, then the URL will differ
  2. Once logged in, on the left side, click on SPARK
  3. Click the Configs tab, scroll down and expand the Custom spark-defaults section.
  4. Click Add Property.
  5. Beside the Type field, ensure to Bulk property add mode is selected
  6. Enter the following values into the Properties box, one value per line:
    • spark.history.fs.cleaner.enabled=true
    • spark.history.fs.cleaner.interval=1d
    • spark.history.fs.cleaner.maxAge=45d
      • NOTE:
        • spark.history.fs.cleaner.interval = how often it checks spark history
        • spark.history.fs.cleaner.maxAge = how much spark history is kept
  7. Once the attributes and values are entered, click Add
  8. Click Save.
  9. Type in a note for the configuration change, and click Save
    • NOTE: A popup warning may appear. If it does, click "Proceed Anyways"
  10. Click OK once the configuration changes are saved
  11. A Restart Required banner will appear right below the Configs tab.
  12. Click the Restart dropdown menu and select Restart All Affected
  13. In the Confirmation popup click Confirm Restart All
  14. A new popup will appear displaying Restart all components with Stale Configs for Spark restarting. Wait for it to complete.
  15. Once it completes click OK.
Applies To
  • Interset 5.4.x or higher
  • HDP 2.5.3 or higher
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments