Follow

How To: Configure Redundancy & High Availability in an Interset 5.3 or higher Environment

This article will discuss how to configure redundancy and high availability across the Interset cluster - this includes all services where it is available in HDP/Ambari, as well as Elasticsearch, and Cassandra/Interset Endpoint.

HDP Services (e.g. HBase, HDFS, Storm, YARN, ZooKeeper):

Note that doing this requires a minimum of two Master nodes in your environment, however we do recommend 3 nodes - one of which has lower resource requirements as it will only run Zookeeper and an HDFS Journal Node.

To configure additional master nodes:

  1. Run bash <(curl -ks http://<interset_mirror_fqdn>/interset-<release>/deploy.sh) on each of the new nodes you wish to add to the cluster.
  2. Open the installer's etc/config file in an editor (e.g. vi).
  3. Modify the MASTER="hostname1" line to read as MASTER="hostname1 hostname2 hostname3" where each hostname refers to a different server in your environment. This can be any number of hostnames, and must be space delimited.
  4. Launch the installer by running ./install.sh in the interset_installer directory.
  5. Select option 1, which will baseline the new nodes and ensure the required configuration is placed on them.

The new hosts must then be added to Ambari:

  1. Open the Ambari web interface (e.g. http://AMBARI_NODE:8080).
  2. Log in.
  3. Click on Hosts.
  4. Under Actions select Add New Hosts.
  5. In Target Hosts add the hostname(s) of the new nodes being added.
  6. Under Host Registration Information copy the SSH private key of the interset user from the Ambari node (e.g. /home/interset/.ssh/id_rsa on the Ambari node), and change the SSH User Account from root to interset.
  7. Click Register and Confirm.
  8. Wait for the tests to complete on the Confirm Hosts page, and click Next.

    Note - Warnings about Snappy packages can be ignored as they are expected.

  9. On the Assign Slaves and Clients page select only Client, and click Next.

    Note - If any of these nodes will be used as Compute nodes, select Data Node, Node Manager, and Region Server in addition to Client.

  10. On the Configurations page, click Next.
  11. Review the information on the Review page, and click Deploy.

Enable HDFS High Availability (HA):

The Ambari wizard, which will guide you through the process of enabling HDFS HA, will require you to SSH into the Master nodes to execute commands manually. All commands are provided in the wizard, and indic- ate the machines on which the commands are to be executed.

  1. Open Ambari, and then select the HDFS service.
  2. On the HDFS settings tab, click Service Actions, and then select Enable NameNode HA.
  3. Follow the instructions in the HDFS HA setup wizard, ensuring that you shut down HBase before proceeding.
  4. When selecting hosts for the additional HA components, spread the three JournalNodes over each of the three Master nodes.

    Note - if you are only using two Master nodes, you may need another node to provide double-duty on this. The journal node is relatively light weight, so a Compute node is a reasonable option in that scenario.

  5. Select an alternate Master node for the additional NameNode.
  6. When the wizard is complete, restart HBase and exit maintenance mode.
    Tip: When finalizing the HA setup, the last step of the wizard, Start All Services, may fail. Should this occur, wait 60 seconds and then click Retry. The services should start after one or two attempts.

Enable YARN High Availability:

  1. Open Ambari, and then select the YARN service.
  2. On the YARN settings tab, click Service Actions, and then select Enable HA.
  3. Select one of the Master nodes for the additional Resource Manager, and then click Next.
  4. Click Next until the wizard is finished.

    Tip - When finalizing the HA setup, the last step of the wizard, Start All Services, may fail. Should this occur, wait 60 seconds and then click Retry. The services should start after one or two attempts.

Enable HBase High Availability:

To enable HBase HA, you must be running multiple HBase Master services.

  1. Open Ambari, and then select the HBase service.
  2. On the HBase settings tab, click Service Actions, and then select Add HBase Master. A dialog box opens, providing a list of host available to run the new HBase Master.
  3. Choose an available host to run the new HBase Master, and then click Confirm Add. Ambari will install the new service on the node selected.
  4. When the installation is complete, start the service:
    1. Select the node on which the new HBase Master was installed.
    2. Click the HBase Master drop-down list, and select Start Service.

Enable Storm High Availability:

To enable Storm HA, a minimum of two Nimbus servers must be installed and running.

  1. Open Ambari, and then select the Storm service.
  2. On the Storm settings tab, click Service Actions, and then select Add Nimbus.
    A dialog box opens, providing a list of host available to run the new Nimbus service.
  3. Choose an available host to run the new Nimbus service, and then click Confirm Add. Ambari will install the new service on the node selected.
  4. When the installation is complete, start the service:
    1. On the Storm settings tab, click Restart, and then click Restart All Affected. The Storm services restart.
    2. Select the node on which the new Nimbus service was installed, and then click the drop-down list and select Start.

Enable ZooKeeper High Availability:

ZooKeeper operates in a quorum to determine the validity of information. With this in mind, an odd number of ZooKeeper services is always recommended in a cluster (e.g. 1,3,5). For Interset's purposes, 3 is sufficient.

  1. Open Ambari, and then select the ZooKeeper service.
  2. Under Service Actions select Add ZooKeeper Server.
  3. Select the desired target host on the Confirmation page, and click Confirm Add.
  4. Repeat steps 1-3 for each additional server you wish to add ZooKeeper on.

    Note - The new ZooKeeper servers will not start automatically, and may require the restart of other services. A full restart of the HDP services, when plausible, is recommended.

Elasticsearch:

Adding additional Elasticsearch nodes can be done via the Interset Installer:

  1. Run bash <(curl -ks http://<interset_mirror_fqdn>/interset-5.3/deploy.sh) on each of the new nodes you wish to add to the cluster.
  2. Open the installer's etc/config file in an editor (e.g. vi).
  3. Modify the SEARCH="hostname1" line to read as SEARCH="hostname1 hostname2 hostname3" where each hostname refers to a different server in your environment. This can be any number of hostnames, and must be space delimited.
  4. Launch the installer by running ./install.sh in the interset_installer directory.
  5. Select option 1, which will baseline the new nodes and ensure the required configuration is placed on them.
  6. Select option 6, which will configure the new search nodes, as well as update the configuration on the existing nodes.

Note that it may take a considerable amount of time for Elasticsearch to rebalance the cluster with the addition of new nodes, and a performance hit should be expected while this is taking place.

 

Cassandra/Interset Endpoint:

We recommend reaching out to Interset Support (support@interset.com) for assistance with a multi-node Endpoint installation.

 

If you have any further questions around the content presented here, or require additional assistance in configuring your services, please reach out to support@interset.com.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk