Follow

Configuring Rollover for Elasticsearch Indices

What does Interset use Elasticsearch for?

Elasticsearch is an open source, broadly-distributable and easily-scalable enterprise-grade search engine. Elasticsearch houses all of the Interset Analytics raw events, and provides all of the data that drives the Investigator user interface.

The purpose of this document is to provide administrators with a way to customize their rollover policy, resulting in a better optimized system.

 

What indices do we write to and read from?

Interset currently reads raw data from indices with the following aliases:

 

“interset_ad_rawdata_<tid>”

“interset_auditd_rawdata_<tid>”

“interset_netflow_rawdata_<tid>”

“interset_printer_rawdata_<tid>”

“interset_sensor_rawdata_<tid>”

“interset_wdc_rawdata_<tid>”

“interset_webproxy_rawdata_<tid>”

“interset_repo_rawdata_<tid>”

“interset_violations_<tid>”

 

The graphic below provides an example of a raw data index being actively written to. It also shows a rolled over and shrunk readonly index.

What is the expected size and growth per day of the indices?

The expected size and growth per day of your indices is based on how much data you have flowing into them. Large amounts of data will see a higher size and growth rate of indices.

 

Why and when do we rollover?

Why do a rollover? A rollover helps consolidate data and prevents a strain on resources. Too many shards of data will slow down searches and strain your system. When to do a rollover depends on your needs. If indices are experiencing large flows of data you may want to roll over based on when X amount of documents are created. If your indice documents are not filling up quickly, you may choose to roll them over based on time intervals to prevent them from getting old.

 

How to configure a rollover policy?

To configure a rollover policy on your Interset system, see Changing Rollover Values below.

 

Changing Rollover Values

The following steps will allow you to change and configure the rollover policy for your raw data. This will allow you to edit the increments at which your documents rollover. As a result, your data will be condensed and set to similar sizes. Changing this configuration to be based on your needs will ensure improved index search and write performance.

 

Note: Data is placed in an index based on when it arrived.

 

Steps

To set your rollover over policy:

 

  1.       On the Analytics node, navigate to the /opt/interset/etc/elasticsearch/config directory.
  2.    Edit the rollover-policies.yml file. Time values should be set as minutes “m”, hours “h”, or days “d”. The file will look similar to this:

- indexWriteAlias: "active_interset_ad_rawdata"

 policy:

conditions:

  maxAge: "7m"

  maxDocs: 5

- indexWriteAlias: "active_interset_wdc_rawdata"

 policy:

conditions:

  maxAge: "10m"

  maxDocs: 8

Important: Only change the values under conditions

  1.       Edit the condition values to match your desired rollover values.

Note: If both conditions are set, the first condition that is triggered will be followed.

Save changes.

  1.       Activate your rollover policy using the following command:

/opt/interset/bin/elasticsearch/rollover.sh rollover --action update --esHost <search_fqdn>

The following should appear:

Note: The cluster name is “interset” by default. If the ES cluster name has been changed, the flag --esClusterName must be used to specify it.

 

If at any time you would like to set your conditions back to the default settings, run the following command:

/opt/interset/bin/elasticsearch/rollover.sh rollover --action reset --esHost <search_fqdn>

If at any time you would like to trigger a manually rollover, run the following command:

/opt/interset/bin/elasticsearch/rollover.sh rollover --action rollover --esHost <search_fqdn>

 

For more options regarding rollover change set up, run the following command:

/opt/interset/bin/elasticsearch/rollover.sh --help

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments