Showing posts with label aws. Show all posts
Showing posts with label aws. Show all posts
Thursday, 27 March 2014
pulling aws instance details using python boto1
########### Example #########
import boto.ec2
# NOTE: if you don't put the key at conn then it will search for
# your profile file, if not /etc/boto.cfg if not your varible 'BOTO_CONFIG=/file/path'
conn=boto.ec2.connect_to_region('us-east-1')
#conn=boto.ec2.connect_to_region('us-east-1',aws_access_key_id='access_key' , aws_secret_access_key='secret_key')
# Following is the prod key: [ Full access ]
#AWS_ACCESS_KEY_ID = 'access_key'
#AWS_SECRET_ACCESS_KEY = 'secret_key'
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print "%s " % (inst.tags['Name'])
print "%s %s " % ("publicDnsName:", inst.public_dns_name)
print "%s %s " % ("internalDnsName:", inst.private_dns_name)
print "%s %s " % ("publicIP:", inst.ip_address)
print "%s %s " % ("internalIP:", inst.private_ip_address)
print "%s %s " % ("architecture:", inst.architecture)
print "%s %s " % ("image_id:", inst.image_id)
print "%s %s " % ("instance_type", inst.instance_type)
print ""
###################################################
import boto.ec2
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print "%s " % (inst.tags['Name'])
print "%s %s " % ("publicDnsName:", inst.public_dns_name)
print "%s %s " % ("internalDnsName:", inst.private_dns_name)
print "%s %s " % ("publicIP:", inst.ip_address)
print "%s %s " % ("internalIP:", inst.private_ip_address)
print "%s %s " % ("architecture:", inst.architecture)
print "%s %s " % ("image_id:", inst.image_id)
print "%s %s " % ("instance_type", inst.instance_type)
print ""
##### To find all the details #####
import boto.ec2
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print(inst.__dict__)
break # remove this to list all instances
### Output: ###
{'_in_monitoring_element': False,
'ami_launch_index': u'0',
'architecture': u'x86_64',
'block_device_mapping': {},
'connection': EC2Connection:ec2.amazonaws.com,
'dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'id': u'i-xxxxxxxx',
'image_id': u'ami-xxxxxxxx',
'instanceState': u'\n ',
'instance_class': None,
'instance_type': u'm1.large',
'ip_address': u'xxx.xxx.xxx.xxx',
'item': u'\n ',
'kernel': None,
'key_name': u'FARM-xxxx',
'launch_time': u'2009-10-27T17:10:22.000Z',
'monitored': False,
'monitoring': u'\n ',
'persistent': False,
'placement': u'us-east-1d',
'previous_state': None,
'private_dns_name': u'ip-10-xxx-xxx-xxx.ec2.internal',
'private_ip_address': u'10.xxx.xxx.xxx',
'product_codes': [],
'public_dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'ramdisk': None,
'reason': '',
'region': RegionInfo:us-east-1,
'requester_id': None,
'rootDeviceType': u'instance-store',
'root_device_name': None,
'shutdown_state': None,
'spot_instance_request_id': None,
'state': u'running',
'state_code': 16,
'subnet_id': None,
'vpc_id': None}
NOTE: You can use above any of this sub value to get only those information.
Wednesday, 26 March 2014
Auto monitoring of aws instance using python boto
Over here, we can discuss about " monitoring automation ", a generic
way: which will work for any one who is using "AWS" "Nagios" for
monitoring.
Features:
[1]: automatically add the new host into monitoring, when ever we add a new system into the aws system.
[2]: automatically will remove the system from monitoring if we terminate the system from aws system.
[3]: read group information from custom tags.
NOTE: As we are going auto monitoring, few rule we have to maintain, else the monitoring will fail.
Rule1: We can have only two tags to any of our aws instance. [1. default: Name, 2. groups ] NOTE, these are case sensitive, so please maintain the same.
Rule2: As of now we have the following key words that can be part of the groups custom tags: [Note: if you need new, you have to let me know before putting the value. This is also case sensitive ] [ you can update the nagios hostgroup config file to add new hostgroup, before adding them into groups custom tags.]
hostgroup_name hadoop
hostgroup_name db
hostgroup_name http
Following is the python boto script:
#!/usr/bin/env python
import boto.ec2
import subprocess
#import os, subprocess
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print ("define host{")
print "%s \t %s" % ("use","generic-host") # \t for tab
print "%s %s" % ("host_name", inst.tags['Name'])
if inst.tags['Name'] == 'qa1':
print "%s \t%s" % ("check_command", "check_ssh")
# different check for qa1 as it is fedora system.
print "%s \t %s: %s" % ("alias", inst.tags['Name'], inst.public_dns_name)
print "%s %s" % ("address", inst.private_ip_address)
# Swapped the alias and address value, because of cost effective)
## Following few code block will check for a custom tags knonw as groups
## if its find the groups, then that host will be part of those hosts.
alltags = (inst.tags) # Will get all the other tags.
alltagsC = str(alltags) # changing the variable type to string.
isgroup = (alltagsC.find('groups'))
if isgroup > 0:
sp = isgroup+11 #found the groups index value and picking the other groups
#global otherGroups
otherGroups = alltagsC[sp:-2]
#print "%s %s %s" % ("hostgroups", inst.instance_type, otherGroups)
print "%s %s" % ("hostgroups", otherGroups)
#else:
#print "%s %s" % ("hostgroups", inst.instance_type)
print ("}\n")
NOTE: As of now I don't know how to get the custom tags value so did some hacks.
NOTE: Removing instance type as part of group, because the monitor will fail, if we have define any group with a instance type and no host is part of that group.
And put the following script into a file and put the file under root crontab:
#!/bin/bash
sudo /path/to/getInstanceDetails.py > /path/to/all_hosts.cfg
sleep 2
sudo service nagios3 restart
##Added this above script in cron as root user: sudo crontab -e
## */15 * * * * sudo /path/to/aboveScrptName.sh
## Now where I will update, what to check where ##
define service{
hostgroup_name db ;<-NOTE: over here you just have to put hostgroup.
service_description MYSQL
check_command check_nrpe_1arg!check_mysql
use generic-service-after-15 ; Name of service template to use
notification_interval 0 ; set > 0 if you want to be renotified
}
NOTE: you can create generic-service-xxx names with its own properties and add them over here.
Features:
[1]: automatically add the new host into monitoring, when ever we add a new system into the aws system.
[2]: automatically will remove the system from monitoring if we terminate the system from aws system.
[3]: read group information from custom tags.
NOTE: As we are going auto monitoring, few rule we have to maintain, else the monitoring will fail.
Rule1: We can have only two tags to any of our aws instance. [1. default: Name, 2. groups ] NOTE, these are case sensitive, so please maintain the same.
Rule2: As of now we have the following key words that can be part of the groups custom tags: [Note: if you need new, you have to let me know before putting the value. This is also case sensitive ] [ you can update the nagios hostgroup config file to add new hostgroup, before adding them into groups custom tags.]
hostgroup_name hadoop
hostgroup_name db
hostgroup_name http
Following is the python boto script:
#!/usr/bin/env python
import boto.ec2
import subprocess
#import os, subprocess
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print ("define host{")
print "%s \t %s" % ("use","generic-host") # \t for tab
print "%s %s" % ("host_name", inst.tags['Name'])
if inst.tags['Name'] == 'qa1':
print "%s \t%s" % ("check_command", "check_ssh")
# different check for qa1 as it is fedora system.
print "%s \t %s: %s" % ("alias", inst.tags['Name'], inst.public_dns_name)
print "%s %s" % ("address", inst.private_ip_address)
# Swapped the alias and address value, because of cost effective)
## Following few code block will check for a custom tags knonw as groups
## if its find the groups, then that host will be part of those hosts.
alltags = (inst.tags) # Will get all the other tags.
alltagsC = str(alltags) # changing the variable type to string.
isgroup = (alltagsC.find('groups'))
if isgroup > 0:
sp = isgroup+11 #found the groups index value and picking the other groups
#global otherGroups
otherGroups = alltagsC[sp:-2]
#print "%s %s %s" % ("hostgroups", inst.instance_type, otherGroups)
print "%s %s" % ("hostgroups", otherGroups)
#else:
#print "%s %s" % ("hostgroups", inst.instance_type)
print ("}\n")
NOTE: As of now I don't know how to get the custom tags value so did some hacks.
NOTE: Removing instance type as part of group, because the monitor will fail, if we have define any group with a instance type and no host is part of that group.
And put the following script into a file and put the file under root crontab:
#!/bin/bash
sudo /path/to/getInstanceDetails.py > /path/to/all_hosts.cfg
sleep 2
sudo service nagios3 restart
##Added this above script in cron as root user: sudo crontab -e
## */15 * * * * sudo /path/to/aboveScrptName.sh
## Now where I will update, what to check where ##
define service{
hostgroup_name db ;<-NOTE: over here you just have to put hostgroup.
service_description MYSQL
check_command check_nrpe_1arg!check_mysql
use generic-service-after-15 ; Name of service template to use
notification_interval 0 ; set > 0 if you want to be renotified
}
NOTE: you can create generic-service-xxx names with its own properties and add them over here.
Tuesday, 25 March 2014
aws hosts information for auto monitoring using python boto 1
import boto.ec2
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print ("define host{")
print ("use\tgeneric-host") # \t for tab
print "%s %s " % ("host_name", inst.tags['Name'])
#print "%s "% (inst.tags['Name'])
print "%s \t %s " % ("alias", inst.tags['Name'])
print "%s %s " % ("address", inst.public_dns_name)
print(inst.instance_type)
print ("}\n")
#print "%s (%s) [%s] [%s]" % (inst.tags['Name'], inst.id, inst.state, inst.public_dns_name)
else:
print "%s [%s]" % (inst.id, inst.state)
######NOTE#########
I believe in the following script output you can see the output name " example: instance_type " so you can use the same for getting that information:
example: for my above script:
inst.public_dns_name will give public_dns_name
inst.region will give the region details and so..on
NOTE: To get the details of the instance that is part of a custom tags then:
reservations = conn.get_all_instances(filters={'tag-key': 'groups'})
############ external script ##############
External Links:
http://www.saltycrane.com/blog/2010/03/how-list-attributes-ec2-instance-python-and-boto/
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print ("define host{")
print ("use\tgeneric-host") # \t for tab
print "%s %s " % ("host_name", inst.tags['Name'])
#print "%s "% (inst.tags['Name'])
print "%s \t %s " % ("alias", inst.tags['Name'])
print "%s %s " % ("address", inst.public_dns_name)
print(inst.instance_type)
print ("}\n")
#print "%s (%s) [%s] [%s]" % (inst.tags['Name'], inst.id, inst.state, inst.public_dns_name)
else:
print "%s [%s]" % (inst.id, inst.state)
######NOTE#########
I believe in the following script output you can see the output name " example: instance_type " so you can use the same for getting that information:
example: for my above script:
inst.public_dns_name will give public_dns_name
inst.region will give the region details and so..on
NOTE: To get the details of the instance that is part of a custom tags then:
reservations = conn.get_all_instances(filters={'tag-key': 'groups'})
############ external script ##############
from pprint import pprint from boto import ec2 AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXX' AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' ec2conn = ec2.connection.EC2Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) reservations = ec2conn.get_all_instances() instances = [i for r in reservations for i in r.instances] for i in instances: pprint(i.__dict__) break # remove this to list all instances
{'_in_monitoring_element': False,
'ami_launch_index': u'0',
'architecture': u'x86_64',
'block_device_mapping': {},
'connection': EC2Connection:ec2.amazonaws.com,
'dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'id': u'i-xxxxxxxx',
'image_id': u'ami-xxxxxxxx',
'instanceState': u'\n ',
'instance_class': None,
'instance_type': u'm1.large',
'ip_address': u'xxx.xxx.xxx.xxx',
'item': u'\n ',
'kernel': None,
'key_name': u'FARM-xxxx',
'launch_time': u'2009-10-27T17:10:22.000Z',
'monitored': False,
'monitoring': u'\n ',
'persistent': False,
'placement': u'us-east-1d',
'previous_state': None,
'private_dns_name': u'ip-10-xxx-xxx-xxx.ec2.internal',
'private_ip_address': u'10.xxx.xxx.xxx',
'product_codes': [],
'public_dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'ramdisk': None,
'reason': '',
'region': RegionInfo:us-east-1,
'requester_id': None,
'rootDeviceType': u'instance-store',
'root_device_name': None,
'shutdown_state': None,
'spot_instance_request_id': None,
'state': u'running',
'state_code': 16,
'subnet_id': None,
'vpc_id': None}
External Links:
http://www.saltycrane.com/blog/2010/03/how-list-attributes-ec2-instance-python-and-boto/
Tuesday, 25 February 2014
aws python api with boto note1
* How to install boto:
1. Need python.
2. download boto [ https://github.com/boto/boto/downloads/ ]
3. Install boto:
2a. tar xfz boto-2.1.tar.gz
2b. cd boto-2.1
2c. sudo python setup.py install
other way:
1. sudo apt-get install python-setuptools
2. sudo easy_install boto
NOTE: Some time few of the boto function don't work, e:g: boto 1.9 don't have instance.tags support, in that case you have to update the boto.
How to update boto: [ I am using easy_install: part of "python-setuptools: "
easy_install -U boto
To see your current boto version:
$python
>>> import boto
>>> boto.Version
'2.27.0'
example of setting the /etc/boto.cfg file:
[Credentials]
aws_access_key_id = ABCDEFGHIJK
aws_secret_access_key = ABCDEFASDFAS123ASF123
########### Example #########
import boto.ec2
# NOTE: if you don't put the key at conn then it will search for
# your profile file, if not /etc/boto.cfg if not your varible 'BOTO_CONFIG=/file/path'
conn=boto.ec2.connect_to_region('us-east-1')
#conn=boto.ec2.connect_to_region('us-east-1',aws_access_key_id='access_key' , aws_secret_access_key='secret_key')
# Following is the prod key: [ Full access ]
#AWS_ACCESS_KEY_ID = 'access_key'
#AWS_SECRET_ACCESS_KEY = 'secret_key'
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print "%s " % (inst.tags['Name'])
print "%s %s " % ("publicDnsName:", inst.public_dns_name)
print "%s %s " % ("internalDnsName:", inst.private_dns_name)
print "%s %s " % ("publicIP:", inst.ip_address)
print "%s %s " % ("internalIP:", inst.private_ip_address)
print "%s %s " % ("architecture:", inst.architecture)
print "%s %s " % ("image_id:", inst.image_id)
print "%s %s " % ("instance_type", inst.instance_type)
print ""
########## ###################### ################
$ to see help with in boto:
>>>help(boto)
>>> dir(boto)
['BUCKET_NAME_RE', 'BotoConfigLocations', 'BucketStorageUri', 'Config', 'ENDPOINTS_PATH', 'FileStorageUri', 'GENERATION_RE', 'InvalidUriError', 'NullHandler', 'TOO_LONG_DNS_NAME_COMP', 'UserAgent', 'VERSION_RE', 'Version', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'auth', 'auth_handler', 'boto', 'cacerts', 'compat', 'config', 'connect_autoscale', 'connect_beanstalk', 'connect_cloudformation', 'connect_cloudfront', 'connect_cloudsearch', 'connect_cloudtrail', 'connect_cloudwatch', 'connect_directconnect', 'connect_dynamodb', 'connect_ec2', 'connect_ec2_endpoint', 'connect_elastictranscoder', 'connect_elb', 'connect_emr', 'connect_euca', 'connect_fps', 'connect_glacier', 'connect_gs', 'connect_ia', 'connect_iam', 'connect_kinesis', 'connect_mturk', 'connect_opsworks', 'connect_rds', 'connect_rds2', 'connect_redshift', 'connect_route53', 'connect_s3', 'connect_sdb', 'connect_ses', 'connect_sns', 'connect_sqs', 'connect_sts', 'connect_support', 'connect_swf', 'connect_vpc', 'connect_walrus', 'connection', 'datetime', 'ec2', 'exception', 'gs', 'handler', 'https_connection', 'init_logging', 'log', 'logging', 'os', 'perflog', 'platform', 'plugin', 'provider', 'pyami', 're', 'regioninfo', 'resultset', 's3', 'set_file_logger', 'set_stream_logger', 'storage_uri', 'storage_uri_for_key', 'sys', 'urlparse', 'utils']
>>> dir(boto.connect_ec2)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
>>> dir(boto.connect_ec2.__get__)
['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__name__', '__new__', '__objclass__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> dir(boto.ec2)
['EC2Connection', 'RegionData', 'RegionInfo', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'address', 'attributes', 'blockdevicemapping', 'bundleinstance', 'connect_to_region', 'connection', 'ec2object', 'get_region', 'get_regions', 'group', 'image', 'instance', 'instanceinfo', 'instancestatus', 'instancetype', 'keypair', 'launchspecification', 'load_regions', 'networkinterface', 'placementgroup', 'regioninfo', 'regions', 'reservedinstance', 'securitygroup', 'snapshot', 'spotdatafeedsubscription', 'spotinstancerequest', 'spotpricehistory', 'tag', 'volume', 'volumestatus', 'zone']
>>> dir(boto.ec2.connect_to_region)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
* Installing easy_install to install rest of the python package.
1. sudo apt-get install python-setuptools
2. sudo easy_install paramiko [ ssh2 like tool. ]
3. sudo apt-get install -y euca2ools [ same as aws command line tools. ]
* aws credentials for boto:
1. at the code itself: [example: ]
>>> import boto
>>> ec2 = boto.connect_ec2(aws_access_key_id='my_access_key',
aws_secret_access_key='my_secret_key')
2. at the system variable: as:
~/.bashrc:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
3. Boto configuration files:
3a. ~/.boto
3b. /etc/boto.cfg
3c. set the varible 'BOTO_CONFIG=/file/path'
with following format:
[Credentials]
aws_access_key_id = your_access_key
aws_secret_access_key = your_secret_key
##### to test the above settings: ######
##### the /etc/boto.cfg file should have the read access for the Credentials ]
>>> import boto
>>> ec2 = boto.connect_ec2()
>>> ec2.get_all_zones()
[Zone:us-east-1a, Zone:us-east-1b, Zone:us-east-1c, Zone:us-east-1d, Zone:us-east-1e]
import boto.ec2
conn=boto.ec2.connect_to_region('us-east-1', aws_access_key_id='<access_key>', aws_secret_access_key='<secret key>')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print "%s (%s) [%s]" % (inst.tags['Name'], inst.id, inst.state)
else:
print "%s [%s]" % (inst.id, inst.state)
[2] In following example, the script is accessing the variable details from the profile file to get the access and the secret key:
import boto.ec2
conn=boto.ec2.connect_to_region("us-east-1")
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print "%s (%s) [%s]" % (inst.tags['Name'], inst.id, inst.state)
else:
print "%s [%s]" % (inst.id, inst.state)
1. Need python.
2. download boto [ https://github.com/boto/boto/downloads/ ]
3. Install boto:
2a. tar xfz boto-2.1.tar.gz
2b. cd boto-2.1
2c. sudo python setup.py install
other way:
1. sudo apt-get install python-setuptools
2. sudo easy_install boto
NOTE: Some time few of the boto function don't work, e:g: boto 1.9 don't have instance.tags support, in that case you have to update the boto.
How to update boto: [ I am using easy_install: part of "python-setuptools: "
easy_install -U boto
To see your current boto version:
$python
>>> import boto
>>> boto.Version
'2.27.0'
example of setting the /etc/boto.cfg file:
[Credentials]
aws_access_key_id = ABCDEFGHIJK
aws_secret_access_key = ABCDEFASDFAS123ASF123
########### Example #########
import boto.ec2
# NOTE: if you don't put the key at conn then it will search for
# your profile file, if not /etc/boto.cfg if not your varible 'BOTO_CONFIG=/file/path'
conn=boto.ec2.connect_to_region('us-east-1')
#conn=boto.ec2.connect_to_region('us-east-1',aws_access_key_id='access_key' , aws_secret_access_key='secret_key')
# Following is the prod key: [ Full access ]
#AWS_ACCESS_KEY_ID = 'access_key'
#AWS_SECRET_ACCESS_KEY = 'secret_key'
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print "%s " % (inst.tags['Name'])
print "%s %s " % ("publicDnsName:", inst.public_dns_name)
print "%s %s " % ("internalDnsName:", inst.private_dns_name)
print "%s %s " % ("publicIP:", inst.ip_address)
print "%s %s " % ("internalIP:", inst.private_ip_address)
print "%s %s " % ("architecture:", inst.architecture)
print "%s %s " % ("image_id:", inst.image_id)
print "%s %s " % ("instance_type", inst.instance_type)
print ""
########## ###################### ################
$ to see help with in boto:
>>>help(boto)
>>> dir(boto)
['BUCKET_NAME_RE', 'BotoConfigLocations', 'BucketStorageUri', 'Config', 'ENDPOINTS_PATH', 'FileStorageUri', 'GENERATION_RE', 'InvalidUriError', 'NullHandler', 'TOO_LONG_DNS_NAME_COMP', 'UserAgent', 'VERSION_RE', 'Version', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'auth', 'auth_handler', 'boto', 'cacerts', 'compat', 'config', 'connect_autoscale', 'connect_beanstalk', 'connect_cloudformation', 'connect_cloudfront', 'connect_cloudsearch', 'connect_cloudtrail', 'connect_cloudwatch', 'connect_directconnect', 'connect_dynamodb', 'connect_ec2', 'connect_ec2_endpoint', 'connect_elastictranscoder', 'connect_elb', 'connect_emr', 'connect_euca', 'connect_fps', 'connect_glacier', 'connect_gs', 'connect_ia', 'connect_iam', 'connect_kinesis', 'connect_mturk', 'connect_opsworks', 'connect_rds', 'connect_rds2', 'connect_redshift', 'connect_route53', 'connect_s3', 'connect_sdb', 'connect_ses', 'connect_sns', 'connect_sqs', 'connect_sts', 'connect_support', 'connect_swf', 'connect_vpc', 'connect_walrus', 'connection', 'datetime', 'ec2', 'exception', 'gs', 'handler', 'https_connection', 'init_logging', 'log', 'logging', 'os', 'perflog', 'platform', 'plugin', 'provider', 'pyami', 're', 'regioninfo', 'resultset', 's3', 'set_file_logger', 'set_stream_logger', 'storage_uri', 'storage_uri_for_key', 'sys', 'urlparse', 'utils']
>>> dir(boto.connect_ec2)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
>>> dir(boto.connect_ec2.__get__)
['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__name__', '__new__', '__objclass__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> dir(boto.ec2)
['EC2Connection', 'RegionData', 'RegionInfo', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'address', 'attributes', 'blockdevicemapping', 'bundleinstance', 'connect_to_region', 'connection', 'ec2object', 'get_region', 'get_regions', 'group', 'image', 'instance', 'instanceinfo', 'instancestatus', 'instancetype', 'keypair', 'launchspecification', 'load_regions', 'networkinterface', 'placementgroup', 'regioninfo', 'regions', 'reservedinstance', 'securitygroup', 'snapshot', 'spotdatafeedsubscription', 'spotinstancerequest', 'spotpricehistory', 'tag', 'volume', 'volumestatus', 'zone']
>>> dir(boto.ec2.connect_to_region)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
* Installing easy_install to install rest of the python package.
1. sudo apt-get install python-setuptools
2. sudo easy_install paramiko [ ssh2 like tool. ]
3. sudo apt-get install -y euca2ools [ same as aws command line tools. ]
* aws credentials for boto:
1. at the code itself: [example: ]
>>> import boto
>>> ec2 = boto.connect_ec2(aws_access_key_id='my_access_key',
aws_secret_access_key='my_secret_key')
2. at the system variable: as:
~/.bashrc:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
3. Boto configuration files:
3a. ~/.boto
3b. /etc/boto.cfg
3c. set the varible 'BOTO_CONFIG=/file/path'
with following format:
[Credentials]
aws_access_key_id = your_access_key
aws_secret_access_key = your_secret_key
##### to test the above settings: ######
##### the /etc/boto.cfg file should have the read access for the Credentials ]
>>> import boto
>>> ec2 = boto.connect_ec2()
>>> ec2.get_all_zones()
[Zone:us-east-1a, Zone:us-east-1b, Zone:us-east-1c, Zone:us-east-1d, Zone:us-east-1e]
>>> ec3 = boto.connect_ec2()
>>> ec3.get_all_zones()
[Zone:us-east-1a, Zone:us-east-1b, Zone:us-east-1c, Zone:us-east-1d, Zone:us-east-1e]
>>> boto.ec2.regions()
[RegionInfo:eu-west-1, RegionInfo:sa-east-1, RegionInfo:us-east-1, RegionInfo:ap-northeast-1, RegionInfo:us-west-2, RegionInfo:us-west-1, RegionInfo:ap-southeast-1, RegionInfo:ap-southeast-2]
default region for boto is "us-east-1"
If you want to connect to a different regions, then:
import boto.ec2
ec_conn = boto.ec2.connect_to_region('eu-west-1')
>>> ec_conn.get_all_zones()
[Zone:eu-west-1a, Zone:eu-west-1b, Zone:eu-west-1c]
NOTE: If you want by default to connect to some other region then you can update your boto config file as:
[Boto]
ec2_region_name = eu-west-1
Q) If I have multiple aws account, then ?:
I can think of set the variable BOTO_CONFIG = /path/to/file/account1 [ on boto scripts. ]
############
[1] Following is one of the python code to get few of the param of the instances:import boto.ec2
conn=boto.ec2.connect_to_region('us-east-1', aws_access_key_id='<access_key>', aws_secret_access_key='<secret key>')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print "%s (%s) [%s]" % (inst.tags['Name'], inst.id, inst.state)
else:
print "%s [%s]" % (inst.id, inst.state)
[2] In following example, the script is accessing the variable details from the profile file to get the access and the secret key:
import boto.ec2
conn=boto.ec2.connect_to_region("us-east-1")
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if 'Name' in inst.tags:
print "%s (%s) [%s]" % (inst.tags['Name'], inst.id, inst.state)
else:
print "%s [%s]" % (inst.id, inst.state)
Tuesday, 31 December 2013
AWS Auto Scaling
### System environment settings ###
In Your bashrc:
export AWS_AUTO_SCALING_HOME=/data/aws-keys/as
export AWS_CREDENTIAL_FILE=/data/aws-keys/as/credential-file-path.template
I believe you already have JAVA_HOME :)
And for AutoScaling cmd API location:
http://ec2-downloads.s3.amazonaws.com/AutoScaling-2011-01-01.zip
You can also go to the following link to get the above command location:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/UsingTheCommandLineTools.html#setting-up-your-tools
Go to above link, and select Auto Scaling Command Line Tools To get the following location.
http://aws.amazon.com/developertools/2535?_encoding=UTF8&jiveRedirect=1
#### Starting Auto scalling sample ####
amit@amitAsus:~$ as-create-launch-config MyCmdASLaunchConfig --image-id <Your ami> --instance-type m1.small
OK-Created launch config
amit@amitAsus:~$ as-describe-launch-configs
LAUNCH-CONFIG MyCmdASLaunchConfig <Your ami> m1.small
NOTE: Do not forget to provide at-least the security group and the key information.
amit@amitAsus:~$ as-create-auto-scaling-group MyCmdASGroup --launch-configuration MyCmdASLaunchConfig --availability-zones us-east-1e --min-size 1 --max-size 2 --desired-capacity 1
OK-Created AutoScalingGroup
amit@amitAsus:~$ as-describe-auto-scaling-groups
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 1 2 1 Default
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
amit@amitAsus:~$ as-describe-auto-scaling-groups --headers
AUTO-SCALING-GROUP GROUP-NAME LAUNCH-CONFIG AVAILABILITY-ZONES MIN-SIZE MAX-SIZE DESIRED-CAPACITY TERMINATION-POLICIES
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 1 2 1 Default
INSTANCE INSTANCE-ID AVAILABILITY-ZONE STATE STATUS LAUNCH-CONFIG
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
amit@amitAsus:~$ as-describe-auto-scaling-instances
INSTANCE i-b94ac699 MyCmdASGroup us-east-1e InService HEALTHY MyCmdASLaunchConfig
######## For deleting #######
If you have not updated the Auto-scaling-group to have min 0 then you will get the following error:
amit@amitAsus:~$ as-delete-auto-scaling-group MyCmdASGroup
Are you sure you want to delete this AutoScalingGroup? [Ny]y
as-delete-auto-scaling-group: Malformed input-You cannot delete an AutoScalingGroup while there are instances
or pending Spot instance request(s) still in the group.
Usage:
as-delete-auto-scaling-group
AutoScalingGroupName [--force-delete ] [General Options]
For more information and a full list of options, run "as-delete-auto-scaling-group --help"
IMP: NOTE: If the user wants to terminate all the instances, first update the Auto Scaling group with the following command: [ To have min-size '0' ]
amit@amitAsus:~$ as-update-auto-scaling-group MyCmdASGroup --min-size 0
OK-Updated AutoScalingGroup
amit@amitAsus:~$ as-describe-auto-scaling-groups
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 0 2 1 Default
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
amit@amitAsus:~$ as-describe-auto-scaling-groups --headers
AUTO-SCALING-GROUP GROUP-NAME LAUNCH-CONFIG AVAILABILITY-ZONES MIN-SIZE MAX-SIZE DESIRED-CAPACITY TERMINATION-POLICIES
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 0 2 1 Default
INSTANCE INSTANCE-ID AVAILABILITY-ZONE STATE STATUS LAUNCH-CONFIG
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
# Now terminate the instance in auto-scaling-group:
INPUT EXAMPLES
Terminates instance 'i-1' and decrements group size.
$PROMPT> as-terminate-instance-in-auto-scaling-group i-1 --decrement-desired-capacity
Terminates instance 'i-2' but does not decrement the group size.
$PROMPT> as-terminate-instance-in-auto-scaling-group i-2 --no-decrement-desired-capacity
amit@amitAsus:~$ as-terminate-instance-in-auto-scaling-group i-b94ac699 --decrement-desired-capacity
Are you sure you want to terminate this instance? [Ny]y
INSTANCE 2d7fc3b8-e725-4b4e-b433-9364f5e6fabe InProgress At 2013-12-31T18:48:53Z instance i-b94ac699 was taken out of service in response to a user request, shrinking the capacity from 1 to 0.
# Now you can delete the auto-scaling group:
amit@amitAsus:~$ as-delete-auto-scaling-group MyCmdASGroup
Are you sure you want to delete this AutoScalingGroup? [Ny]y
OK-Deleted AutoScalingGroup
amit@amitAsus:~$ as-describe-auto-scaling-groups
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 0 0 0 Default
amit@amitAsus:~$ as-describe-auto-scaling-instances
No instances found
amit@amitAsus:~$ as-describe-auto-scaling-groups
No AutoScalingGroups found
amit@amitAsus:~$ as-describe-launch-configs
LAUNCH-CONFIG MyCmdASLaunchConfig ami-f5381c9c m1.small
amit@amitAsus:~$ as-delete-launch-config MyCmdASLaunchConfig
Are you sure you want to delete this launch configuration? [Ny]y
OK-Deleted launch configuration
amit@amitAsus:~$ as-describe-launch-configs
No launch configurations found
NOTE: On Scripting, you might need to wait for some time after issuing few commands because, it takes some time, to give you rest of the information of do a recheck of the expected command out-out. Few Ref Link:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/WhatIsAutoScaling.html
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/US_BasicSetup.html
http://www.newvem.com/how-to-create-update-and-delete-an-aws-auto-scaling-group/
In Your bashrc:
export AWS_AUTO_SCALING_HOME=/data/aws-keys/as
export AWS_CREDENTIAL_FILE=/data/aws-keys/as/credential-file-path.template
I believe you already have JAVA_HOME :)
And for AutoScaling cmd API location:
http://ec2-downloads.s3.amazonaws.com/AutoScaling-2011-01-01.zip
You can also go to the following link to get the above command location:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/UsingTheCommandLineTools.html#setting-up-your-tools
Go to above link, and select Auto Scaling Command Line Tools To get the following location.
http://aws.amazon.com/developertools/2535?_encoding=UTF8&jiveRedirect=1
#### Starting Auto scalling sample ####
amit@amitAsus:~$ as-create-launch-config MyCmdASLaunchConfig --image-id <Your ami> --instance-type m1.small
OK-Created launch config
amit@amitAsus:~$ as-describe-launch-configs
LAUNCH-CONFIG MyCmdASLaunchConfig <Your ami> m1.small
NOTE: Do not forget to provide at-least the security group and the key information.
amit@amitAsus:~$ as-create-auto-scaling-group MyCmdASGroup --launch-configuration MyCmdASLaunchConfig --availability-zones us-east-1e --min-size 1 --max-size 2 --desired-capacity 1
OK-Created AutoScalingGroup
amit@amitAsus:~$ as-describe-auto-scaling-groups
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 1 2 1 Default
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
amit@amitAsus:~$ as-describe-auto-scaling-groups --headers
AUTO-SCALING-GROUP GROUP-NAME LAUNCH-CONFIG AVAILABILITY-ZONES MIN-SIZE MAX-SIZE DESIRED-CAPACITY TERMINATION-POLICIES
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 1 2 1 Default
INSTANCE INSTANCE-ID AVAILABILITY-ZONE STATE STATUS LAUNCH-CONFIG
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
amit@amitAsus:~$ as-describe-auto-scaling-instances
INSTANCE i-b94ac699 MyCmdASGroup us-east-1e InService HEALTHY MyCmdASLaunchConfig
######## For deleting #######
If you have not updated the Auto-scaling-group to have min 0 then you will get the following error:
amit@amitAsus:~$ as-delete-auto-scaling-group MyCmdASGroup
Are you sure you want to delete this AutoScalingGroup? [Ny]y
as-delete-auto-scaling-group: Malformed input-You cannot delete an AutoScalingGroup while there are instances
or pending Spot instance request(s) still in the group.
Usage:
as-delete-auto-scaling-group
AutoScalingGroupName [--force-delete ] [General Options]
For more information and a full list of options, run "as-delete-auto-scaling-group --help"
IMP: NOTE: If the user wants to terminate all the instances, first update the Auto Scaling group with the following command: [ To have min-size '0' ]
amit@amitAsus:~$ as-update-auto-scaling-group MyCmdASGroup --min-size 0
OK-Updated AutoScalingGroup
amit@amitAsus:~$ as-describe-auto-scaling-groups
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 0 2 1 Default
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
amit@amitAsus:~$ as-describe-auto-scaling-groups --headers
AUTO-SCALING-GROUP GROUP-NAME LAUNCH-CONFIG AVAILABILITY-ZONES MIN-SIZE MAX-SIZE DESIRED-CAPACITY TERMINATION-POLICIES
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 0 2 1 Default
INSTANCE INSTANCE-ID AVAILABILITY-ZONE STATE STATUS LAUNCH-CONFIG
INSTANCE i-b94ac699 us-east-1e InService Healthy MyCmdASLaunchConfig
# Now terminate the instance in auto-scaling-group:
INPUT EXAMPLES
Terminates instance 'i-1' and decrements group size.
$PROMPT> as-terminate-instance-in-auto-scaling-group i-1 --decrement-desired-capacity
Terminates instance 'i-2' but does not decrement the group size.
$PROMPT> as-terminate-instance-in-auto-scaling-group i-2 --no-decrement-desired-capacity
amit@amitAsus:~$ as-terminate-instance-in-auto-scaling-group i-b94ac699 --decrement-desired-capacity
Are you sure you want to terminate this instance? [Ny]y
INSTANCE 2d7fc3b8-e725-4b4e-b433-9364f5e6fabe InProgress At 2013-12-31T18:48:53Z instance i-b94ac699 was taken out of service in response to a user request, shrinking the capacity from 1 to 0.
# Now you can delete the auto-scaling group:
amit@amitAsus:~$ as-delete-auto-scaling-group MyCmdASGroup
Are you sure you want to delete this AutoScalingGroup? [Ny]y
OK-Deleted AutoScalingGroup
amit@amitAsus:~$ as-describe-auto-scaling-groups
AUTO-SCALING-GROUP MyCmdASGroup MyCmdASLaunchConfig us-east-1e 0 0 0 Default
amit@amitAsus:~$ as-describe-auto-scaling-instances
No instances found
amit@amitAsus:~$ as-describe-auto-scaling-groups
No AutoScalingGroups found
amit@amitAsus:~$ as-describe-launch-configs
LAUNCH-CONFIG MyCmdASLaunchConfig ami-f5381c9c m1.small
amit@amitAsus:~$ as-delete-launch-config MyCmdASLaunchConfig
Are you sure you want to delete this launch configuration? [Ny]y
OK-Deleted launch configuration
amit@amitAsus:~$ as-describe-launch-configs
No launch configurations found
NOTE: On Scripting, you might need to wait for some time after issuing few commands because, it takes some time, to give you rest of the information of do a recheck of the expected command out-out. Few Ref Link:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/WhatIsAutoScaling.html
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/US_BasicSetup.html
http://www.newvem.com/how-to-create-update-and-delete-an-aws-auto-scaling-group/
Thursday, 21 November 2013
s3cmd : A command line tools to work with the amazon s3
Some time we need to work with our "aws s3" bucket and need a command line tool to access amazon's s3 restful api. " s3cmd " one of the command that will help you working on this.
-> To install s3cmd at your ubuntu system:
sudo apt-get install s3cmd
Once you install your s3cmd you need to configure your system to have your access key and secrete key to work with your s3 bucket.
To configure your s3cmd having the config, run the following command:
s3cmd --configure
You will be asked for the two keys, access key and secrete key
NOTE: You use this key, whey you use the ec2-upload-bundle command.
-> To install s3cmd at your ubuntu system:
sudo apt-get install s3cmd
Once you install your s3cmd you need to configure your system to have your access key and secrete key to work with your s3 bucket.
To configure your s3cmd having the config, run the following command:
s3cmd --configure
You will be asked for the two keys, access key and secrete key
NOTE: You use this key, whey you use the ec2-upload-bundle command.
Friday, 14 June 2013
How to setup ssh environment with a password bashed key pairs
1. Background:
The purpose of this document is to overcome the issue that we face at our AWS environment. Its true that amazon not keeping our private key, when we have create a new instance. But we are not very strict about those aws keys, most of the team inside the company use the key and keep the same in lots of the place, and this can cause of a security hole.
Example:
Let say foo.com is an online company and the qa environment people is not keeping track of where the keys are keeping. They are not very sure about the keys. Let say if any of the qa team person kept the key at document root and later some how that key got into hand of some cracker, then he/she can log-in to the QA environment and that's compromise our environment.
2. What we can do?
Its true that without the key no one can able to log-in. But we don't even want to share those aws keys to everyone.
So, create your own ssh-key pairs, with a pass-phase.
1. ssh-keygen [enter]
2. select your preferred key type [rsa / dsa] or can go for the default one. [ enter ]
3. in pass-phase enter a password.
After that you can update the public key of that key pairs to qa server's authorized_keys and share the QA team. Now onwards when ever they want to login to the qa servers, they can use the same key from a blessed host. [ You can create a secure host, from where every-one login to company servers]. And you can protect the blessed hosts in the same way.
3. Question(s)?
How do I perform the automation:
You can do the following for that:
1. Log in to bless host [ with your personal keys ].
2. use "ssh-agent bash" [ I am using bash, you can use any of your shell ]
3. ssh-add [ at this time it will ask for the password, provide the password. ]
Later you can log-in to the qa server from the bless hosts without typing the password again and again.
You might also like to go through "screen" command.
4. Further reading:
Please go through further documentation on the following command for more details:
ssh-keygen
ssh-agent
ssh-add
screen
Wednesday, 12 June 2013
AWS_Amazon_Spot_Instance
AWS: Amazon spot instance:
The purpose of using an Amazon Spot instance is for:
1. To save money (it’s cheaper to run servers on Amazon as Spot instances)
2. To make recovering a failed instance very fast and easy.
Q1. How this is cheaper?
Few example:
http://www.youtube.com/embed/WD9N73F3Fao?rel=0&hd=1
http://www.youtube.com/embed/BD1X5ItelOk?rel=0&hd=1
Because, even if we have bid for the higher price, its charges for the current spot price.
Example:
1. A linux c1xl server's current bid price is $0.070 [ 7 cents only per hour ]
2. The On demand price is around $0.50 [ 50 cents per hour]
and out bid price is $0.75 [ 75 cents per hour ] which is higher even the on demand price, but how its can cost less? Because its charge on the current spot price [ which is 7 cents now]. Then why we are requesting for that high bid price [ around 75 cents ] even higher then spot price?
Note that, the spot instance goes away if some one bid in higher price, so when we have our bid price is more then the on-demand price, then most likely we have a higher chances that out hosts will not go down, and the spot price stay much lower then on demand price for a longer time. Hence we save an overall money and a higher chances of getting the spot instance for longer time.
For more details, please follow the above youtube links.
A Spot Instance on Amazon is how customers can ‘bid’ for unused capacity on Amazon’s infrastructure. The cost to run a Spot Instance is always fluctuating. As long as that cost is below the maximum ‘bid’ price that we bid, then the server continues to run. However, when Amazon has system issues, network issues, etc, then the current Spot Instance price will go up, exceeding our maximum bid price. When this happens, the server will be terminated instantly without any warning. Therefore, Spot Instances are useless for things like a Database Server. And they are only useful for areas of the product that are designed to take advantage of them.
AWS wiki link: http://aws.amazon.com/ec2/spot-instances/ [ Further details ]
The purpose of using an Amazon Spot instance is for:
1. To save money (it’s cheaper to run servers on Amazon as Spot instances)
2. To make recovering a failed instance very fast and easy.
Q1. How this is cheaper?
Few example:
http://www.youtube.com/embed/WD9N73F3Fao?rel=0&hd=1
http://www.youtube.com/embed/BD1X5ItelOk?rel=0&hd=1
Because, even if we have bid for the higher price, its charges for the current spot price.
Example:
1. A linux c1xl server's current bid price is $0.070 [ 7 cents only per hour ]
2. The On demand price is around $0.50 [ 50 cents per hour]
and out bid price is $0.75 [ 75 cents per hour ] which is higher even the on demand price, but how its can cost less? Because its charge on the current spot price [ which is 7 cents now]. Then why we are requesting for that high bid price [ around 75 cents ] even higher then spot price?
Note that, the spot instance goes away if some one bid in higher price, so when we have our bid price is more then the on-demand price, then most likely we have a higher chances that out hosts will not go down, and the spot price stay much lower then on demand price for a longer time. Hence we save an overall money and a higher chances of getting the spot instance for longer time.
For more details, please follow the above youtube links.
A Spot Instance on Amazon is how customers can ‘bid’ for unused capacity on Amazon’s infrastructure. The cost to run a Spot Instance is always fluctuating. As long as that cost is below the maximum ‘bid’ price that we bid, then the server continues to run. However, when Amazon has system issues, network issues, etc, then the current Spot Instance price will go up, exceeding our maximum bid price. When this happens, the server will be terminated instantly without any warning. Therefore, Spot Instances are useless for things like a Database Server. And they are only useful for areas of the product that are designed to take advantage of them.
AWS wiki link: http://aws.amazon.com/ec2/spot-instances/ [ Further details ]
Spot Instance Issues
A Spot Instance has to be set up in a special way in order for it to work correctly.
When
our maximum ‘bid’ price is exceeded, the server will be terminated.
Then, when the current price drops back below our maximum bid price, the
server will be re-launched. When it is re-launched, it’s like it was
launched for the first time: it gets a new InstanceId, all data is lost,
etc. Therefore, any data that must be retained has to be on a separate EBS volume.
Also,
the IP addresses that the spot instance uses will also change when it
is re-launched. This causes issues for both monitoring and for
database access, since the server’s IP address is used for both.
This means that the Spot Instance must have boot-up scripts that:
- mount an /ebs volume where the data is stored,
-
all data that we change on a regular basis (Apache Document Root, etc)
must be links from the root disk to locations under /ebs
- we must assign an Elastic IP address when the instance boots up, so it gets the same public IP address and Amazon DNS name each time it is re-launched. The Amazon DNS name will not change, but the Private IP Address that it resolves to *does* change.
Saturday, 9 February 2013
SettingUpAWSapi
Setting Up AWS api: [ Amazon command line tools ] :
I am using ubuntu 12.10 and for the long time I was getting error while setting up AWS api in my system:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
export EC2_HOME=~/ec2
export EC2_PRIVATE_KEY=<Private_key_path>
export EC2_CERT=<Cert_path>
For other aws api:
export AWS_ELB_HOME=~/elbexport AWS_CLOUDWATCH_HOME=~/cw
PATH=$PATH:$JAVA_HOME/bin:$EC2_HOME/bin
Note:
You can create different scripts, where you specify different EC2_CERT and EC2_PRIVATE_KEY to access your different aws account, example [dev, qa or prod]
For other api call:
http://aws.amazon.com/developertools/For AMI CLI:
http://awsiammedia.s3.amazonaws.com/public/tools/cli/latest/IAMCli.zip
For ElasticBeanstalk CLI:
https://s3.amazonaws.com/elasticbeanstalk/cli/AWS-ElasticBeanstalk-CLI-2.6.0.zip
For ELB CLI:
http://ec2-downloads.s3.amazonaws.com/ElasticLoadBalancing.zip
For AutoScalling CLI:
http://ec2-downloads.s3.amazonaws.com/AutoScaling-2011-01-01.zip
For S3 CLI:
http://code.google.com/p/lits3/downloads/detail?name=LitS3-Commander-0.8.4.zip
Even after above setting I was getting java related error... and I stared trying solving the same issue, checked some on-line links and found lots of people have similar issue. [ I was my mistake that I was keeping all the aws command in a same directory. ]
I removed all the jre and java related package from my ubuntu system and reinstall those pkg again but the similar issue.
The error was something like:
ec2-describe-regions
Error: Could not find or load main class com.amazon.aes.webservices.client.cmd.DescribeRegions
Then I started looking around the JAVA CLASSPATH... again spent some more time. But the good part that I understand that in hard way... after working for long hours. As a SRE/DEVOPS person, when started from Linux System Administrator, understanding java error took some time :)
Started looking the aws command code:
amitmund@amitmundlaptop:~/ec2/bin$ cat ../ec2-api-tools-1.6.0.0/bin/ec2-describe-regions
#!/usr/bin/env bash
# Copyright 2006-2010 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the
# Amazon Software License (the "License"). You may not use this file except in compliance with the License. A copy of the
# License is located at http://aws.amazon.com/asl or in the "license" file accompanying this file. This file is distributed on an "AS
# IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
__ZIP_PREFIX__EC2_HOME="${EC2_HOME:?EC2_HOME is not set}"
__RPM_PREFIX__EC2_HOME=/usr/local/aes/cmdline
"${EC2_HOME}"/bin/ec2-cmd DescribeRegions "$@"
And then followed the ec2-cmd:
amitmund@amitmundlaptop:~/ec2/bin$ cat ec2-cmd
#!/usr/bin/env bash
# Copyright 2006-2009 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the
# Amazon Software License (the "License"). You may not use this file except in compliance with the License. A copy of the
# License is located at http://aws.amazon.com/asl or in the "license" file accompanying this file. This file is distributed on an "AS
# IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
# This script "concentrates" all of our Java invocations into a single location
# for maintainability.
# 'Globals'
__ZIP_PREFIX__EC2_HOME="${EC2_HOME:-EC2_HOME is not set}"
__RPM_PREFIX__EC2_HOME=/usr/local/aes/cmdline
LIBDIR="${EC2_HOME}/lib"
# Check our Java env
JAVA_HOME=${JAVA_HOME:?JAVA_HOME is not set}
# If a classpath exists preserve it
CP="${CLASSPATH}"
# Check for cygwin bash so we use the correct path separator
case "`uname`" in
CYGWIN*) cygwin=true;;
esac
# ---- Start of Cygwin test ----
cygprop=""
# And add our own libraries too
if [ "${cygwin}" == "true" ] ; then
cygprop="-Dec2.cygwin=true"
# Make sure that when using Cygwin we use Unix
# Semantics for EC2_HOME
if [ -n "${EC2_HOME}" ]
then
if echo "${EC2_HOME}"|egrep -q '[[:alpha:]]:\\'
then
echo
echo " *INFO* Your EC2_HOME variable needs to specified as a Unix path under Cygwin"
echo
fi
fi
# ---- End of Cygwin Tests ----
for jar in "${LIBDIR}"/*.jar ; do
cygjar=$(cygpath -w -a "${jar}")
CP="${CP};${cygjar}"
done
else
for jar in "${LIBDIR}"/*.jar ; do
CP="${CP}:${jar}"
done
fi
CMD=$1
shift
"${JAVA_HOME}/bin/java" ${EC2_JVM_ARGS} ${cygprop} -classpath "${CP}" "com.amazon.aes.webservices.client.cmd.${CMD}" $EC2_DEFAULT_ARGS "$@"
then I found it is looking for classpath and the jar files from the same directory's lib folder from were I have copied the aws commands.
Later I created a "lib" directory in the same location where I created the "bin" directory for all the aws command line tools and started coping the "jar" files from all the original directory to this "lib" directory and its started working. :)
so.... where I was doing mistake?
I have downloaded all the aip from Amazon site... and it around few different zip files from windows and linux environment...
e.g:
AutoScaling-2010-08-01.zip
CloudWatch-2010-08-01.zip
ec2-api-tools.zip RDSCli.zip
AWSCloudFormation-cli.zip
ec2-ami-tools.zip
IAMCli.zip
But what I did, to set the a single EC2_HOME directory, I extracted all these directory and copied all the aws command from those directories "bin" directory and copied them to ~/ec2/bin and not the related "lib" directory where all the related jar files were there. Because of that I was getting the following error:
Error: Could not find or load main class com.amazon.aes.webservices.client.cmd.DescribeRegions
Subscribe to:
Posts (Atom)