Auto monitoring of aws instance using python boto

Over here, we can discuss about " monitoring automation ", a generic way: which will work for any one who is using "AWS" "Nagios" for monitoring.

Features:
[1]: automatically add the new host into monitoring, when ever we add a new system into the aws system.
[2]: automatically will remove the system from monitoring if we terminate the system from aws system.
[3]: read group information from custom tags.

NOTE: As we are going auto monitoring, few rule we have to maintain, else the monitoring will fail.

Rule1: We can have only two tags to any of our aws instance. [1. default: Name, 2. groups ] NOTE, these are case sensitive, so please maintain the same.

Rule2: As of now we have the following key words that can be part of the groups custom tags: [Note: if  you need new, you have to let me know before putting the value. This is also case sensitive ] [ you can update the nagios hostgroup config file to add new hostgroup, before adding them into groups custom tags.]

        hostgroup_name      hadoop
        hostgroup_name      db
        hostgroup_name      http


Following is the python boto script:

#!/usr/bin/env python
import boto.ec2
import subprocess
#import os, subprocess

conn=boto.ec2.connect_to_region('us-east-1')

reservations = conn.get_all_instances()
for res in reservations:
    for inst in res.instances:
            print ("define host{")
            print "%s \t %s" % ("use","generic-host") # \t for tab
            print "%s  %s" % ("host_name", inst.tags['Name'])
            if inst.tags['Name'] == 'qa1':
                print "%s \t%s" % ("check_command", "check_ssh")
                # different check for qa1 as it is fedora system.
            print "%s \t %s: %s" % ("alias", inst.tags['Name'], inst.public_dns_name)          
            print "%s  %s" % ("address", inst.private_ip_address)
            # Swapped the alias and address value, because of cost effective)
            ## Following few code block will check for a custom tags knonw as groups
            ## if its find the groups, then that host will be part of those hosts.
            alltags = (inst.tags) # Will get all the other tags.
            alltagsC = str(alltags) # changing the variable type to string.
            isgroup = (alltagsC.find('groups'))
            if isgroup > 0:
                sp = isgroup+11 #found the groups index value and picking the other groups
                #global otherGroups
                otherGroups = alltagsC[sp:-2]
                #print  "%s  %s  %s" % ("hostgroups", inst.instance_type, otherGroups)
                print "%s  %s" % ("hostgroups", otherGroups)
            #else:
                #print "%s %s" % ("hostgroups", inst.instance_type)
            print ("}\n")

NOTE: As of now I don't know how to get the custom tags value so did some hacks.
NOTE: Removing instance type as part of group, because the monitor will fail, if we have define any group with a instance type and no host is part of that group.


And put the following script into a file and put the file under root crontab:

#!/bin/bash
sudo /path/to/getInstanceDetails.py > /path/to/all_hosts.cfg
sleep 2
sudo service nagios3 restart

##Added this above script in cron as root user: sudo crontab -e
## */15 * * * * sudo /path/to/aboveScrptName.sh


## Now where I will update, what to check where ##

define service{
        hostgroup_name                  db ;<-NOTE: over here you just have to put hostgroup.
        service_description             MYSQL
        check_command                   check_nrpe_1arg!check_mysql
        use                             generic-service-after-15 ; Name of service template to use
        notification_interval           0 ; set > 0 if you want to be renotified
}

NOTE: you can create generic-service-xxx names with its own properties and add them over here.

nagios monitoring ang amazon [ aws ] internal external dns name vs ipaddress

nagios monitoring ang amazon [ aws ] internal external dns name vs ipaddress

NOTE:

How do you monitor an aws ec2 host if it is a spot instance and the internal ip keep changing and you are on ec2 classic network [ not the vpn ]

so, when you are monitoring the remote host, the remote host nrpe cfg file need to permit your nagios server host. But when you have to monitor the a server, you can use the internal ip to monitor and it will work fine, but when in the above case, even if your security group allow using the external IP or public dns or using the elastic ip, the monitoring will fail. In that case you need to use the public dns name.

Example:

NOTE: The following [ using public dns name ] will work.
Lets say your nagios server belongs to "security-group-x" and its allow for your nagios communication port. [ default 5666 ]and your nagios server address is updated at your nrpe.cfg's allow host list.

define host{
        use                     generic-host
        host_name          spot-ec2-in-classic-network
        alias                   spot-ec2-in-classic-network
        address                 ec2-23-10-100-200.compute-1.amazonaws.com
        }


NOTE: In the following example it will not work.

Why?: In the above example I have given the address as the public-dns name provided by AWS and AWS can get the further information, like from which security group it is comming from and migh be in the following example, its doing the reverse dig and getting a different dns name [ if you have set ] or not doing that even?


define host{
        use                     generic-host
        host_name          spot-ec2-in-classic-network
        alias                   spot-ec2-in-classic-network
        address                 23.10.100.200
        }

NOTE: So, if you have above type requirement, then will suggest to use the Amazon [ AWS ] DNS names. Some time I believe, you should use the amazon dns name for all the communication, even if it for internal ip :)

Nagios Active and Passive checks with nsca and nrpe

Here is the few notes on the Nagios Active and Passive checks with nsca and nrpe:

Ref link: http://nsclient.org/nscp/wiki/doc/usage/nagios/nsca

$ dpkg --listfiles nsca-client
/.
/etc
/etc/send_nsca.cfg
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nsca-client
/usr/share/doc/nsca-client/copyright
/usr/share/doc/nsca-client/NEWS.Debian.gz
/usr/share/doc/nsca-client/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/send_nsca.1.gz
/usr/sbin
/usr/sbin/send_nsca


$ send_nsca --help

NSCA Client 2.7.2
Copyright (c) 2000-2007 Ethan Galstad (www.nagios.org)
Last Modified: 07-03-2007
License: GPL v2
Encryption Routines: AVAILABLE

Usage: send_nsca -H <host_address> [-p port] [-to to_sec] [-d delim] [-c config_file]

Options:
 <host_address> = The IP address of the host running the NSCA daemon
 [port]         = The port on which the daemon is running - default is 5667
 [to_sec]       = Number of seconds before connection attempt times out.
                  (default timeout is 10 seconds)
 [delim]        = Delimiter to use when parsing input (defaults to a tab)
 [config_file]  = Name of config file to use

Note:
This utility is used to send passive check results to the NSCA daemon.  Host and
Service check data that is to be sent to the NSCA daemon is read from standard
input. Input should be provided in the following format (tab-delimited unless
overriden with -d command line argument, one entry per line):

Service Checks:
<host_name>[tab]<svc_description>[tab]<return_code>[tab]<plugin_output>[newline]

Host Checks:
<host_name>[tab]<return_code>[tab]<plugin_output>[newline]



$ dpkg --listfiles nsca
/.
/etc
/etc/nsca.cfg
/etc/init.d
/etc/init.d/nsca
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nsca
/usr/share/doc/nsca/examples
/usr/share/doc/nsca/examples/nsca.xinetd
/usr/share/doc/nsca/README.gz
/usr/share/doc/nsca/copyright
/usr/share/doc/nsca/README.Debian
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/nsca
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/nsca.1.gz
/usr/sbin
/usr/sbin/nsca
/usr/share/doc/nsca/NEWS.Debian.gz
/usr/share/doc/nsca/changelog.Debian.gz

Artifactory Repository Notes

Url link on how to install Artifactory:

1. Downloaded the binary file from sourceforge:
 $ wget http://sourceforge.net/projects/artifactory/files/latest/download

2. Rename the file [ It will be named as download ]
$ mv download artifactory-3.1.1.zip

3. For my use I moved the same file to /opt dir.
$ sudo mv artifactory-3.1.1.zip /opt

4. Extracted the zip file
$ unzip artifactory-3.1.1.zip

Documentation I am following as of now:

http://www.jfrog.com/confluence/display/RTF/Installing+on+Linux+Solaris+or+Mac+OS

http://www.jfrog.com/confluence/display/RTF/Installing+on+Linux+Solaris+or+Mac+OS#InstallingonLinuxSolarisorMacOS-ManualInstallation










*) What you need: 

NOTE: You need to hava java 1.7. So if you don't have the java 1.7, download the jre tar file from oracle site [ We prefer to have oracle java then openjava. ]. Need not to have JDK. JRE should be enough.

*) lets say you don't have java1.7: on that case:

5a. Download the jar1.7 from oracle site.
5b. I have extracted the same at /opt/ [ you can do at your wish. ]


6a. Go to your the artifactory extracted path [ example /opt/artifactory-3.1.1 for me] then "bin" dir and run "sudo sh installService.sh" ... This will install the artifactory as a service... so that you can use... sudo service artifactory {start|stop}

6b. After the above command: [ do the following: NOTE: if you don't have JAVA1.7 only. ]
sudo vi /etc/opt/jfrog/artifactory/deault

and update the JAVA_HOME [ For me as per the example: export JAVA_HOME=/opt/jre1.7.0 ]

Then you can do: sudo service artifactory start.

Do a netstat: and you can do a quick check on what port your artifactory is running. 

For me it started on port 8081 [ Make sure you don't have any firewall blocking that port.]

http://localhost:8081/artifactory   <--- here you have your artifactory running.



log in to artifactory at the above url: default login [ admin / password ]


Default log location: [ tail -f $ARTIFACTORY_HOME/logs/artifactory.log ]



*) How do I configure rest at artifactory:

URL: http://www.jfrog.com/confluence/display/RTF/Administering+Artifactory

7a. Once you login to the artifactory UI page, you can select "ADMIN" tab to configure rest of the conigurations.