Imagine your Apache HTTP Server goes down mid-night and the personnel responsible for this is not in office until morning. It is totally a nightmare!

Manual intervention is not a good idea when you require an immediate attention on such service failure as well as quicker remediation to avoid service disruption.

Adopt continuous service monitoring & auto-healing solution built in integration with Flint & Hashicorp Consul. While Consul keeps monitoring the health of the service, upon failure Flint will mend the disrupted service and restore its health.

Let's see how we can build such an integration with Flint IT Automation Platform & Consul.

Prerequisites

For our purpose, we have configured two machines as server and client respectively.

Server, in this case, will maintain service information, key/value storage pools, and other details for the client machine. And the Consul client will be installed on the machine with Flint and Apache2 service (Apache HTTP Server as a subject of monitoring).

Bootstrap Consul Server - 192.168.2.33 - server.example.com Consul Client - 192.168.2.11 - client.example.com

Consul documentation recommends that only one server must be in the bootstrap mode. We have used Linux Ubuntu 14.04LTS for our use-case. However, any latest Linux machine should work equally well. Also, Consul [ v0.5.2 ] and Flint [ v0.28.1.0 ] versions were used here.

Kick-start with Consul

To begin with, we need to - Start the Consul agents in server and client modes, Create the Consul key-value store on the server and Register services to be monitored on the client. We skip Consul installation assume it is done as described in the installation guide at http://www.consul.io/intro/getting-started/install.html

A. Starting the Bootstrap Server

On server.example.com machine start the Consul agent in server mode. To do so execute the following command on the terminal. Also, the bootstrap option will cause the server to be self-elected ( as there will be only one server in the entire cluster with this option ).

consul agent -server -bootstrap-expect 1 -data-dir=/tmp/consul

With this, we have our server up and running

B. Joining the Cluster as a Client

On client.example.com machine start the Consul agent in client mode. To do so execute the following command on the terminal.

consul agent -data-dir=/tmp/consul -join server.example.com

By now, we have the Consul client up and running and server (in bootstrap mode) within the cluster. You can add additional servers and client to cluster if required by repeating the above steps.

C. Create Consul key-value store to save Flint application configuration parameters

Here, we need to store Flint hostname, username, password, Flintbit name and Flintbit request parameters in Consul key-value store.

To store flint hostname -

curl -X PUT -d 'http://192.168.2.11:3501' 'http://192.168.2.33:8500/v1/kv/flint/hostname'

To store flint username -

curl -X PUT -d 'admin' http://localhost:8500/v1/kv/flint/username'

To store flint password -

curl -X PUT -d 'admin123' 'http://localhost:8500/v1/kv/flint/password'

To store flintbit name -

curl -X PUT -d 'example:apache2_service.rb' 'http://localhost:8500/v1/kv/flint/services/apache2'

To store flintbit request parameters -

curl -X PUT -H "Content-Type: application/json" -d '{ "username": "username", "password": "password" }' 'http://localhost:8500/v1/kv/flint/services/apache2/flintbit/parameters

You can verify all the keys and their values by executing `curl -v http://localhost:8500/v1/kv/?recurse`. After running this command, you observe that the Value field is base64 encoded by Consul. This is done to allow non-UTF8 characters.

D. Adding Services and Checks to Consul

Create a configuration directory to store service definitions

mkdir ~/services

Inside the services directory, create a JSON file named "apache2.json” that will describe the apache2 service.

We have used the following service definition:

{
   "service": {
        "name": "apache2",
        "address": "192.168.2.11",
        "port": 80,
        "check": {
        "script": "service apache2 status",
        "interval": "10s }
   }
}

The name, address and port of the service belong to the service definition whereas script and interval belong to the check definition.

Consul will look out for a service at 192.168.2.11 and keep checking port#80 for the same.

We did add the check definition for Consul so to keep checking the health status of the service specified. The check must be of the script, HTTP, or TTL type. Our check will run with a 10 seconds of an interval, invoking service Apache2 status.

If the command exits with a non-zero exit code, then the agent node will be flagged unhealthy. This is the contract for any script-based health check.

E. Restart the Consul Client

While restarting, point the consul configuration directory to services having apache2.JSON service definition we just created.

consul agent -data-dir=/tmp/consul -config-dir=services -join server.example.com

Flint set-up/configuration

Using Flint's SSH connector, we will remote login into the machine on which apache2 service is installed i.e. client.example.com.

We already have a ready Flintbit i.e. “apache2_service.rb” for you, which will start the Apache2 service using the flint SSH connector. This Flintbit will be triggered by Consul watch handler when the Apache2 service health status becomes critical.

Flintbox i.e. “auto-remediation-example” contains the Flintbit. So to add and configure this Flintbox you need to know the git URL i.e. https://github.com/infiverve/auto-remediation-example.git

F. Configure SSH Connector

Login to Flint -> Go to Connectors from left side navigation and click on Add Connector button

Configure SSH Connector

G. Provide configuration parameters for the connector

  • Assign a Name to SSH Connector
  • Select flint-ssh-connector as connector type from drop down
  • Add Description for the connector
  • Provide number of instances of (may be '10' for now) SSH connector
  • Select Auto-HA as High-Availability options from radio buttons
  • Add Config as empty JSON for the connector
  • Click on Add & Enable button to add and enable or activate this on Flint

SSH Connector Parameters

No Config ( JSON ) is required here, as parameters required by the flint ssh connector used in Flintbit will be provided at run time by Consul from its key-value store which we have already created in the previous steps under Kick-start with Consul section.

H. Add Flintbox

Login to Flint -> Go to Flintbox from left side navigation and click on Add Flintbox button

Add Flintbox

I. Configure Flintbox Enter the git URL given above and then click on the Add button. Username and password are not required for this Flintbox.

Configure Flintbox

We will now have “auto-remediation-example” Flintbox added to flint, which can be viewed by visiting the INACTIVE FLINTBOX tab.

J. Enable Flintbox

To enable, visit the INACTIVE FLINTBOX tab and click on Enable button against the configured Flintbox i.e. “auto-remediation-example”.

Enable Flintbox

Watching the Service

K. Establish the watch

Will now monitor for the change in service health status by using the watch feature of Consul. consul watch -service=apache2 -type=service

Our watch type is service and Apache2 is the name of the service to be watched ( as provided in the service definition file i.e. ~/services/apache2.json ).

L. Providing a Handler

A Handler will be invoked when service health status changes. A Handler could be any executable. We have created a shell script named service_handler.sh which will trigger flint only when Apache2 service health is critical.

We can get service_handler.sh from this link

So, now after creating the handler lets re-phrase the command and execute it.

consul watch -service=apache2 -type=service sh service_handler.sh

Every time your service goes down, Flint will mend the service with the help of Consul. Thus, you get self-healing, undisruptive services where Consul spots the cause and Flint will be the aid to the service.

To know more about this and similar solutions post us at info@getflint.io or visit Flint website

Next Post Previous Post