Over here, we can discuss about " monitoring automation ", a generic
way: which will work for any one who is using "AWS" "Nagios" for
monitoring.
Features:
[1]: automatically add the new host into monitoring, when ever we add a new system into the aws system.
[2]: automatically will remove the system from monitoring if we terminate the system from aws system.
[3]: read group information from custom tags.
NOTE: As we are going auto monitoring, few rule we have to maintain, else the monitoring will fail.
Rule1: We can have only two tags to any of our aws instance. [1. default: Name, 2. groups ] NOTE, these are case sensitive, so please maintain the same.
Rule2:
As of now we have the following key words that can be part of the
groups custom tags: [Note: if you need new, you have to let me know
before putting the value. This is also case sensitive ] [ you can update
the nagios hostgroup config file to add new hostgroup, before adding
them into groups custom tags.]
hostgroup_name hadoop
hostgroup_name db
hostgroup_name http
Following is the python boto script:
#!/usr/bin/env python
import boto.ec2
import subprocess
#import os, subprocess
conn=boto.ec2.connect_to_region('us-east-1')
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
print ("define host{")
print "%s \t %s" % ("use","generic-host") # \t for tab
print "%s %s" % ("host_name", inst.tags['Name'])
if inst.tags['Name'] == 'qa1':
print "%s \t%s" % ("check_command", "check_ssh")
# different check for qa1 as it is fedora system.
print "%s \t %s: %s" % ("alias", inst.tags['Name'], inst.public_dns_name)
print "%s %s" % ("address", inst.private_ip_address)
# Swapped the alias and address value, because of cost effective)
## Following few code block will check for a custom tags knonw as groups
## if its find the groups, then that host will be part of those hosts.
alltags = (inst.tags) # Will get all the other tags.
alltagsC = str(alltags) # changing the variable type to string.
isgroup = (alltagsC.find('groups'))
if isgroup > 0:
sp = isgroup+11 #found the groups index value and picking the other groups
#global otherGroups
otherGroups = alltagsC[sp:-2]
#print "%s %s %s" % ("hostgroups", inst.instance_type, otherGroups)
print "%s %s" % ("hostgroups", otherGroups)
#else:
#print "%s %s" % ("hostgroups", inst.instance_type)
print ("}\n")
NOTE: As of now I don't know how to get the custom tags value so did some hacks.
NOTE: Removing instance type as part of group, because the monitor will fail, if we have define any group with a instance type and no host is part of that group.
And put the following script into a file and put the file under root crontab:
#!/bin/bash
sudo /path/to/getInstanceDetails.py > /path/to/all_hosts.cfg
sleep 2
sudo service nagios3 restart
##Added this above script in cron as root user: sudo crontab -e
## */15 * * * * sudo /path/to/aboveScrptName.sh
## Now where I will update, what to check where ##
define service{
hostgroup_name db ;<-NOTE: over here you just have to put hostgroup.
service_description MYSQL
check_command check_nrpe_1arg!check_mysql
use generic-service-after-15 ; Name of service template to use
notification_interval 0 ; set > 0 if you want to be renotified
}
NOTE: you can create generic-service-xxx names with its own properties and add them over here.
Disclaimer: Its a collection from lots of other site(s) and few of my notes. I would also like to declare that I am not owning lots of its content. Please feel free to contact me directly if you want me to remove any of your content, that you don't want to share to other through this blog.
Showing posts with label Monitoring. Show all posts
Showing posts with label Monitoring. Show all posts
Wednesday, 26 March 2014
Wednesday, 5 March 2014
nagios monitoring ang amazon [ aws ] internal external dns name vs ipaddress
nagios monitoring ang amazon [ aws ] internal external dns name vs ipaddress
NOTE:
How do you monitor an aws ec2 host if it is a spot instance and the internal ip keep changing and you are on ec2 classic network [ not the vpn ]
so, when you are monitoring the remote host, the remote host nrpe cfg file need to permit your nagios server host. But when you have to monitor the a server, you can use the internal ip to monitor and it will work fine, but when in the above case, even if your security group allow using the external IP or public dns or using the elastic ip, the monitoring will fail. In that case you need to use the public dns name.
Example:
NOTE: The following [ using public dns name ] will work.
Lets say your nagios server belongs to "security-group-x" and its allow for your nagios communication port. [ default 5666 ]and your nagios server address is updated at your nrpe.cfg's allow host list.
define host{
use generic-host
host_name spot-ec2-in-classic-network
alias spot-ec2-in-classic-network
address ec2-23-10-100-200.compute-1.amazonaws.com
}
NOTE: In the following example it will not work.
Why?: In the above example I have given the address as the public-dns name provided by AWS and AWS can get the further information, like from which security group it is comming from and migh be in the following example, its doing the reverse dig and getting a different dns name [ if you have set ] or not doing that even?
define host{
use generic-host
host_name spot-ec2-in-classic-network
alias spot-ec2-in-classic-network
address 23.10.100.200
}
NOTE: So, if you have above type requirement, then will suggest to use the Amazon [ AWS ] DNS names. Some time I believe, you should use the amazon dns name for all the communication, even if it for internal ip :)
NOTE:
How do you monitor an aws ec2 host if it is a spot instance and the internal ip keep changing and you are on ec2 classic network [ not the vpn ]
so, when you are monitoring the remote host, the remote host nrpe cfg file need to permit your nagios server host. But when you have to monitor the a server, you can use the internal ip to monitor and it will work fine, but when in the above case, even if your security group allow using the external IP or public dns or using the elastic ip, the monitoring will fail. In that case you need to use the public dns name.
Example:
NOTE: The following [ using public dns name ] will work.
Lets say your nagios server belongs to "security-group-x" and its allow for your nagios communication port. [ default 5666 ]and your nagios server address is updated at your nrpe.cfg's allow host list.
define host{
use generic-host
host_name spot-ec2-in-classic-network
alias spot-ec2-in-classic-network
address ec2-23-10-100-200.compute-1.amazonaws.com
}
NOTE: In the following example it will not work.
Why?: In the above example I have given the address as the public-dns name provided by AWS and AWS can get the further information, like from which security group it is comming from and migh be in the following example, its doing the reverse dig and getting a different dns name [ if you have set ] or not doing that even?
define host{
use generic-host
host_name spot-ec2-in-classic-network
alias spot-ec2-in-classic-network
address 23.10.100.200
}
NOTE: So, if you have above type requirement, then will suggest to use the Amazon [ AWS ] DNS names. Some time I believe, you should use the amazon dns name for all the communication, even if it for internal ip :)
Nagios Active and Passive checks with nsca and nrpe
Here is the few notes on the Nagios Active and Passive checks with nsca and nrpe:
Ref link: http://nsclient.org/nscp/wiki/doc/usage/nagios/nsca
$ dpkg --listfiles nsca-client
/.
/etc
/etc/send_nsca.cfg
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nsca-client
/usr/share/doc/nsca-client/copyright
/usr/share/doc/nsca-client/NEWS.Debian.gz
/usr/share/doc/nsca-client/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/send_nsca.1.gz
/usr/sbin
/usr/sbin/send_nsca
$ send_nsca --help
NSCA Client 2.7.2
Copyright (c) 2000-2007 Ethan Galstad (www.nagios.org)
Last Modified: 07-03-2007
License: GPL v2
Encryption Routines: AVAILABLE
Usage: send_nsca -H <host_address> [-p port] [-to to_sec] [-d delim] [-c config_file]
Options:
<host_address> = The IP address of the host running the NSCA daemon
[port] = The port on which the daemon is running - default is 5667
[to_sec] = Number of seconds before connection attempt times out.
(default timeout is 10 seconds)
[delim] = Delimiter to use when parsing input (defaults to a tab)
[config_file] = Name of config file to use
Note:
This utility is used to send passive check results to the NSCA daemon. Host and
Service check data that is to be sent to the NSCA daemon is read from standard
input. Input should be provided in the following format (tab-delimited unless
overriden with -d command line argument, one entry per line):
Service Checks:
<host_name>[tab]<svc_description>[tab]<return_code>[tab]<plugin_output>[newline]
Host Checks:
<host_name>[tab]<return_code>[tab]<plugin_output>[newline]
$ dpkg --listfiles nsca
/.
/etc
/etc/nsca.cfg
/etc/init.d
/etc/init.d/nsca
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nsca
/usr/share/doc/nsca/examples
/usr/share/doc/nsca/examples/nsca.xinetd
/usr/share/doc/nsca/README.gz
/usr/share/doc/nsca/copyright
/usr/share/doc/nsca/README.Debian
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/nsca
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/nsca.1.gz
/usr/sbin
/usr/sbin/nsca
/usr/share/doc/nsca/NEWS.Debian.gz
/usr/share/doc/nsca/changelog.Debian.gz
Ref link: http://nsclient.org/nscp/wiki/doc/usage/nagios/nsca
$ dpkg --listfiles nsca-client
/.
/etc
/etc/send_nsca.cfg
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nsca-client
/usr/share/doc/nsca-client/copyright
/usr/share/doc/nsca-client/NEWS.Debian.gz
/usr/share/doc/nsca-client/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/send_nsca.1.gz
/usr/sbin
/usr/sbin/send_nsca
$ send_nsca --help
NSCA Client 2.7.2
Copyright (c) 2000-2007 Ethan Galstad (www.nagios.org)
Last Modified: 07-03-2007
License: GPL v2
Encryption Routines: AVAILABLE
Usage: send_nsca -H <host_address> [-p port] [-to to_sec] [-d delim] [-c config_file]
Options:
<host_address> = The IP address of the host running the NSCA daemon
[port] = The port on which the daemon is running - default is 5667
[to_sec] = Number of seconds before connection attempt times out.
(default timeout is 10 seconds)
[delim] = Delimiter to use when parsing input (defaults to a tab)
[config_file] = Name of config file to use
Note:
This utility is used to send passive check results to the NSCA daemon. Host and
Service check data that is to be sent to the NSCA daemon is read from standard
input. Input should be provided in the following format (tab-delimited unless
overriden with -d command line argument, one entry per line):
Service Checks:
<host_name>[tab]<svc_description>[tab]<return_code>[tab]<plugin_output>[newline]
Host Checks:
<host_name>[tab]<return_code>[tab]<plugin_output>[newline]
$ dpkg --listfiles nsca
/.
/etc
/etc/nsca.cfg
/etc/init.d
/etc/init.d/nsca
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nsca
/usr/share/doc/nsca/examples
/usr/share/doc/nsca/examples/nsca.xinetd
/usr/share/doc/nsca/README.gz
/usr/share/doc/nsca/copyright
/usr/share/doc/nsca/README.Debian
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/nsca
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/nsca.1.gz
/usr/sbin
/usr/sbin/nsca
/usr/share/doc/nsca/NEWS.Debian.gz
/usr/share/doc/nsca/changelog.Debian.gz
Thursday, 20 February 2014
monitoring
Some monitoring software: [ Need to update the following topics: ]
Sensu
Graphite
Flapjack
Statsd
Logstash
cacti
ganglia
zabbix
munin
Zenoss
Sensu
Graphite
Flapjack
Statsd
Logstash
cacti
ganglia
zabbix
munin
Zenoss
Friday, 3 January 2014
Apache Per Second Request Count And Plotting That Data With GnuPlot
Over here I will explain how can get your Apache's http server's per second request count and plot the same data using the gnuplot.
While I was trying to get per second request count, parsing the apache httpd access log to plot with the standard apache log format, I found some difficulty, as the log was getting written to the log file in async format, So though of adding "epoch" time on the log and do the next things. Although you can see this status with server-status module [ http://amitmund.blogspot.in/2013/12/apachehttpservers-server-status.html ]
What I did to add the epoch time:
I am using the following LogFormat [ enabled at my /etc/apache2/sites-enabled/000-default ] at my site:
CustomLog "|/usr/sbin/rotatelogs /mnt/httpd/logs/access_log.%Y-%m-%d-%H_%M_%S 100M" combined
While I was trying to get per second request count, parsing the apache httpd access log to plot with the standard apache log format, I found some difficulty, as the log was getting written to the log file in async format, So though of adding "epoch" time on the log and do the next things. Although you can see this status with server-status module [ http://amitmund.blogspot.in/2013/12/apachehttpservers-server-status.html ]
What I did to add the epoch time:
I am using the following LogFormat [ enabled at my /etc/apache2/sites-enabled/000-default ] at my site:
CustomLog "|/usr/sbin/rotatelogs /mnt/httpd/logs/access_log.%Y-%m-%d-%H_%M_%S 100M" combined
So, I have to update the same "combined" format at my apache2.conf file, and I have added the following format at my LogFormat, so that it will record the epoch time.
NOTE: Following line %{s}t will add epoch time at the log
LogFormat "%{%s}t %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
Before that I had a log as:
10.210.193.192 - - [26/Dec/2013:10:16:25 +0000] "GET / HTTP/1.1" 200 517 "-" "check_http/v1.4.14 (nagios-plugins 1.4.14)"
Now I can see a epoch time added before the above log format:
1388744705 10.210.193.192 - - [26/Dec/2013:10:16:25 +0000] "GET / HTTP/1.1" 200 517 "-" "check_http/v1.4.14 (nagios-plugins 1.4.14)"
Further details about the log format:
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html
Few of the useful options:
%{%s}t : epoch time
%{format}t : The time, in the form given by format, which should be in
strftime(3) format. (potentially localized)
%D : The time taken to serve the request, in microseconds.
%f : Filename
% : The request method
%s : Status. For requests that got internally redirected, this is
the status of the *original* request --- %>s
for the last.
%T : The time taken to serve the request, in seconds.
After updating the logFormat, I did a apache restart and cross checked to see the update format and found its working fine.
Getting Data:
After the log format got updated, I did a grep of the epoch time and redirected to a file.
$ awk '{print $1}' access-file > epochTime.dat
The above command will take the epoch time from the access-file and put the same at epochTime.dat file. [ Over here I have added epoch time at the 1st field, so I have used the $1 over here.
This is useful if you are doing a benchmark of your web server and want to know and plot how many apache request it is serving every second.
Using above command, you will find the data like following:
$ head epochTime.dat
1388745807
1388745807
1388745807
1388745807
1388745807
1388745807
1388745807
1388745807
Now you have to get the count of the same epoch time: so,
$ cat epochTime.dat | sort | uniq -c
By above command you will see the output as:
2 1388745779
400 1388745780
1016 1388745781
1048 1388745782
1057 1388745783
1063 1388745784
440 1388745785
Now I did:
awk '{print $2,$1}' epochTime.dat > epochTime1.dat
After the above command you can use the above data to plot graph easyly:
Now you get the data something like following:
$ head epochTime1.dat
1388745779 2
1388745780 400
1388745781 1016
1388745782 1048
1388745783 1057
1388745784 1063
1388745785 440
Now I have the data How I will plot the same:
I am using the gnuplot to plot the above data. [ other gnuplot example: http://amitmund.blogspot.in/2014/01/gnuplot.html ]
$ vi PerSecondReq.gplot: # the gnu plot code to plot the data
set xdata time
set timefmt "%s"
set xlabel "TIME"
set ylabel "Req"
plot "epochTime1.dat" using 1:2 title "Req" with lines
Now You have your plotted data.[ Following is an example of the same. ]
Friday, 20 December 2013
Monitoring java process
Few notes on how to monitor the java application:
Following is one of the command, using this we can get some information on java status.
$ jstat --help
invalid argument count
Usage: jstat -help|-options
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Definitions:
<option> An option reported by the -options option
<vmid> Virtual Machine Identifier. A vmid takes the following form:
<lvmid>[@<hostname>[:<port>]]
Where <lvmid> is the local vm identifier for the target
Java virtual machine, typically a process id; <hostname> is
the name of the host running the target Java virtual machine;
and <port> is the port number for the rmiregistry on the
target host. See the jvmstat documentation for a more complete
description of the Virtual Machine Identifier.
<lines> Number of samples between header lines.
<interval> Sampling interval. The following forms are allowed:
<n>["ms"|"s"]
Where <n> is an integer and the suffix specifies the units as
milliseconds("ms") or seconds("s"). The default units are "ms".
<count> Number of samples to take before terminating.
-J<flag> Pass <flag> directly to the runtime system.
Following is the example: jstat -<option> <pid-of-java-process> <mili-sec-interval> <NumberOfTime>
NOTE: How you will file the pid of some java process. [ e.g: ps -ef | grep java ] of [ jps ]
Once you get the pid of the java process, you can use the following command. E.g: 21253 is one PID of one of the java service.
$ jps
21253 jar
3089 Jps
$ jstat -gc 21253 250 7
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
$ jstat -gc `jps | grep -i jar | awk '{print $1}'` 250 7
[ NOTE: This above command is picking the pid of java jar as an param of the command ]
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
$ jstat -options
-class
-compiler
-gc
-gccapacity
-gccause
-gcnew
-gcnewcapacity
-gcold
-gcoldcapacity
-gcpermcapacity
-gcutil
-printcompilation
$ jstat -gccapacity `jps | grep -i jar | awk '{print $1}'` 250 7
NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC PGCMN PGCMX PGC PC YGC FGC
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
jmap
Usage:
jmap [option] <pid>
(to connect to running process)
jmap [option] <executable <core>
(to connect to a core file)
jmap [option] [server_id@]<remote server IP or hostname>
(to connect to remote debug server)
where <option> is one of:
<none> to print same info as Solaris pmap
-heap to print java heap summary
-histo[:live] to print histogram of java object heap; if the "live"
suboption is specified, only count live objects
-permstat to print permanent generation statistics
-finalizerinfo to print information on objects awaiting finalization
-dump:<dump-options> to dump java heap in hprof binary format
dump-options:
live dump only live objects; if not specified,
all objects in the heap are dumped.
format=b binary format
file=<file> dump heap to <file>
Example: jmap -dump:live,format=b,file=heap.bin <pid>
-F force. Use with -dump:<dump-options> <pid> or -histo
to force a heap dump or histogram when <pid> does not
respond. The "live" suboption is not supported
in this mode.
-h | -help to print this help message
-J<flag> to pass <flag> directly to the runtime system
jps
4919 SecondaryNameNode
4375 NameNode
4668 DataNode
5237 Jps
sudo jmap -heap 4668
Attaching to process ID 4668, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.7-b01
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1048576000 (1000.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 174063616 (166.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 32374784 (30.875MB)
used = 21415272 (20.423194885253906MB)
free = 10959512 (10.451805114746094MB)
66.14799962835274% used
From Space:
capacity = 5373952 (5.125MB)
used = 5365648 (5.1170806884765625MB)
free = 8304 (0.0079193115234375MB)
99.84547684832317% used
To Space:
capacity = 5373952 (5.125MB)
used = 0 (0.0MB)
free = 5373952 (5.125MB)
0.0% used
PS Old Generation
capacity = 86245376 (82.25MB)
used = 363240 (0.34641265869140625MB)
free = 85882136 (81.9035873413086MB)
0.4211704057038374% used
PS Perm Generation
capacity = 21757952 (20.75MB)
used = 16151640 (15.403404235839844MB)
free = 5606312 (5.346595764160156MB)
74.23327342573418% used
6521 interned Strings occupying 519584 bytes.
sudo jmap -dump:live,format=b,file=heap.bin 4668
4668: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
-------
if you are using Tomcat. Update CATALINA_OPTS environment variable
export CATALINA_OPTS=-Xms16m -Xmx256m;
Few other notes:
http://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jmx/details
http://blog.lesc.se/2012/02/how-to-take-java-heap-dump-from-command.html
http://visualvm.java.net/heapdump.html
http://www.rallydev.com/community/engineering/outofmemoryerror-fun-heap-dump-analysis
http://java.dzone.com/articles/java-heap-dump-are-you-task
http://architects.dzone.com/articles/how-analyze-java-thread-dumps
http://www.javaworld.com/article/2072864/heap-dump-and-analysis-with-visualvm.html
https://blog.codecentric.de/en/2011/08/create-and-understand-java-heapdumps-act-4/
Following is one of the command, using this we can get some information on java status.
$ jstat --help
invalid argument count
Usage: jstat -help|-options
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Definitions:
<option> An option reported by the -options option
<vmid> Virtual Machine Identifier. A vmid takes the following form:
<lvmid>[@<hostname>[:<port>]]
Where <lvmid> is the local vm identifier for the target
Java virtual machine, typically a process id; <hostname> is
the name of the host running the target Java virtual machine;
and <port> is the port number for the rmiregistry on the
target host. See the jvmstat documentation for a more complete
description of the Virtual Machine Identifier.
<lines> Number of samples between header lines.
<interval> Sampling interval. The following forms are allowed:
<n>["ms"|"s"]
Where <n> is an integer and the suffix specifies the units as
milliseconds("ms") or seconds("s"). The default units are "ms".
<count> Number of samples to take before terminating.
-J<flag> Pass <flag> directly to the runtime system.
Following is the example: jstat -<option> <pid-of-java-process> <mili-sec-interval> <NumberOfTime>
NOTE: How you will file the pid of some java process. [ e.g: ps -ef | grep java ] of [ jps ]
Once you get the pid of the java process, you can use the following command. E.g: 21253 is one PID of one of the java service.
$ jps
21253 jar
3089 Jps
$ jstat -gc 21253 250 7
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
$ jstat -gc `jps | grep -i jar | awk '{print $1}'` 250 7
[ NOTE: This above command is picking the pid of java jar as an param of the command ]
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
$ jstat -options
-class
-compiler
-gc
-gccapacity
-gccause
-gcnew
-gcnewcapacity
-gcold
-gcoldcapacity
-gcpermcapacity
-gcutil
-printcompilation
NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC PGCMN PGCMX PGC PC YGC FGC
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
jmap
Usage:
jmap [option] <pid>
(to connect to running process)
jmap [option] <executable <core>
(to connect to a core file)
jmap [option] [server_id@]<remote server IP or hostname>
(to connect to remote debug server)
where <option> is one of:
<none> to print same info as Solaris pmap
-heap to print java heap summary
-histo[:live] to print histogram of java object heap; if the "live"
suboption is specified, only count live objects
-permstat to print permanent generation statistics
-finalizerinfo to print information on objects awaiting finalization
-dump:<dump-options> to dump java heap in hprof binary format
dump-options:
live dump only live objects; if not specified,
all objects in the heap are dumped.
format=b binary format
file=<file> dump heap to <file>
Example: jmap -dump:live,format=b,file=heap.bin <pid>
-F force. Use with -dump:<dump-options> <pid> or -histo
to force a heap dump or histogram when <pid> does not
respond. The "live" suboption is not supported
in this mode.
-h | -help to print this help message
-J<flag> to pass <flag> directly to the runtime system
jps
4919 SecondaryNameNode
4375 NameNode
4668 DataNode
5237 Jps
sudo jmap -heap 4668
Attaching to process ID 4668, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.7-b01
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1048576000 (1000.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 174063616 (166.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 32374784 (30.875MB)
used = 21415272 (20.423194885253906MB)
free = 10959512 (10.451805114746094MB)
66.14799962835274% used
From Space:
capacity = 5373952 (5.125MB)
used = 5365648 (5.1170806884765625MB)
free = 8304 (0.0079193115234375MB)
99.84547684832317% used
To Space:
capacity = 5373952 (5.125MB)
used = 0 (0.0MB)
free = 5373952 (5.125MB)
0.0% used
PS Old Generation
capacity = 86245376 (82.25MB)
used = 363240 (0.34641265869140625MB)
free = 85882136 (81.9035873413086MB)
0.4211704057038374% used
PS Perm Generation
capacity = 21757952 (20.75MB)
used = 16151640 (15.403404235839844MB)
free = 5606312 (5.346595764160156MB)
74.23327342573418% used
6521 interned Strings occupying 519584 bytes.
sudo jmap -dump:live,format=b,file=heap.bin 4668
4668: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
sudo jmap -F -dump:live,format=b,file=heap.bin 4668
Attaching to process ID 4668, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.7-b01
Dumping heap to heap.bin ...
none
null_check
null_assert
range_check
class_check
array_check
intrinsic
bimorphic
unloaded
uninitialized
unreached
unhandled
constraint
div0_check
age
predicate
loop_limit_check
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Heap dump file created
-rw-r--r-- 1 root root 11785562 Dec 23 15:30 heap.bin
amit@amitAsus:~$ du -ksh heap.bin
12M heap.bin
NOTE: I found VisualVM [ GUI tool ] that can be used to read the heap dump.
sudo apt-get install visualvm
sudo apt-get install visualvm
-------
You should use jmap -heap:format=b <process-id> without any paths. So it creates a *.bin file which you can open with jvisualvm.exe (same path as jmap). It's a great tool to open such dump files.
------
If you use Eclipse as your IDE I would recommend the excellent eclipse plugin memory analyzer
------
Another option is to use JVisualVM, it can read (and create) heap dumps as well, and is shipped with every JDK. You can find it in the bin directory of your JDK.
You can load in visualvm
-----
You can use jhat (Java Heap Analysis Tool) to read the generated file:
======
jhat [ options ] <heap-dump-file>
The jhat command parses a java heap dump file and launches a webserver. jhat enables you to browse heap dumps using your favorite webbrowser.
Note that you should have a hprof binary format output to be able to parse it with jhat. You can use format=b option to generate the dump in this format.
-dump:format=b,file=<filename>
jhat
ERROR: No arguments supplied
Usage: jhat [-stack <bool>] [-refs <bool>] [-port <port>] [-baseline <file>] [-debug <int>] [-version] [-h|-help] <file>
-J<flag> Pass <flag> directly to the runtime system. For
example, -J-mx512m to use a maximum heap size of 512MB
-stack false: Turn off tracking object allocation call stack.
-refs false: Turn off tracking of references to objects
-port <port>: Set the port for the HTTP server. Defaults to 7000
-exclude <file>: Specify a file that lists data members that should
be excluded from the reachableFrom query.
-baseline <file>: Specify a baseline object dump. Objects in
both heap dumps with the same ID and same class will
be marked as not being "new".
-debug <int>: Set debug level.
0: No debug output
1: Debug hprof file parsing
2: Debug hprof file parsing, no server
-version Report version number
-h|-help Print this help and exit
<file> The file to read
For a dump file that contains multiple heap dumps,
you may specify which dump in the file
by appending "#<number>" to the file name, i.e. "foo.hprof#3".
All boolean options default to "true"
------------
$ jinfo
Usage:
jinfo [option] <pid>
(to connect to running process)
jinfo [option] <executable <core>
(to connect to a core file)
jinfo [option] [server_id@]<remote server IP or hostname>
(to connect to remote debug server)
where <option> is one of:
-flag <name> to print the value of the named VM flag
-flag [+|-]<name> to enable or disable the named VM flag
-flag <name>=<value> to set the named VM flag to the given value
-flags to print VM flags
-sysprops to print Java system properties
<no option> to print both of the above
-h | -help to print this help message
-----
MAT, jprofiler,jhat are possible options. since jhat comes with jdk, you can easily launch it to do some basic analysis. check this out
-----
sudo apt-get install visualvm
[sudo] password for amit:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
kde-l10n-engb
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
antlr3 aspectj javahelp2 junit4 libantlr-java libaspectj-java libfelix-framework-java libfelix-main-java
libgeronimo-jpa-2.0-spec-java libgeronimo-osgi-support-java libhamcrest-java libjna-java libnb-org-openide-modules-java
libnb-org-openide-util-java libnb-org-openide-util-lookup-java libnb-platform13-java libosgi-compendium-java libosgi-core-java
libosgi-foundation-ee-java libservlet2.5-java libstringtemplate-java libswing-layout-java
Suggested packages:
javahelp2-doc libfelix-framework-java-doc libfelix-main-java-doc libgeronimo-jpa-2.0-spec-java-doc
libgeronimo-osgi-support-java-doc libjna-java-doc libnb-platform13-java-doc libosgi-compendium-java-doc libosgi-core-java-doc
libosgi-foundation-ee-java-doc libswing-layout-java-doc
The following NEW packages will be installed:
antlr3 aspectj javahelp2 junit4 libantlr-java libaspectj-java libfelix-framework-java libfelix-main-java
libgeronimo-jpa-2.0-spec-java libgeronimo-osgi-support-java libhamcrest-java libjna-java libnb-org-openide-modules-java
libnb-org-openide-util-java libnb-org-openide-util-lookup-java libnb-platform13-java libosgi-compendium-java libosgi-core-java
libosgi-foundation-ee-java libservlet2.5-java libstringtemplate-java libswing-layout-java visualvm
0 upgraded, 23 newly installed, 0 to remove and 1 not upgraded.
Need to get 26.7 MB of archives.
After this operation, 49.7 MB of additional disk space will be used.
Java memory sizes:
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
sudo apt-get install visualvm
[sudo] password for amit:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
kde-l10n-engb
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
antlr3 aspectj javahelp2 junit4 libantlr-java libaspectj-java libfelix-framework-java libfelix-main-java
libgeronimo-jpa-2.0-spec-java libgeronimo-osgi-support-java libhamcrest-java libjna-java libnb-org-openide-modules-java
libnb-org-openide-util-java libnb-org-openide-util-lookup-java libnb-platform13-java libosgi-compendium-java libosgi-core-java
libosgi-foundation-ee-java libservlet2.5-java libstringtemplate-java libswing-layout-java
Suggested packages:
javahelp2-doc libfelix-framework-java-doc libfelix-main-java-doc libgeronimo-jpa-2.0-spec-java-doc
libgeronimo-osgi-support-java-doc libjna-java-doc libnb-platform13-java-doc libosgi-compendium-java-doc libosgi-core-java-doc
libosgi-foundation-ee-java-doc libswing-layout-java-doc
The following NEW packages will be installed:
antlr3 aspectj javahelp2 junit4 libantlr-java libaspectj-java libfelix-framework-java libfelix-main-java
libgeronimo-jpa-2.0-spec-java libgeronimo-osgi-support-java libhamcrest-java libjna-java libnb-org-openide-modules-java
libnb-org-openide-util-java libnb-org-openide-util-lookup-java libnb-platform13-java libosgi-compendium-java libosgi-core-java
libosgi-foundation-ee-java libservlet2.5-java libstringtemplate-java libswing-layout-java visualvm
0 upgraded, 23 newly installed, 0 to remove and 1 not upgraded.
Need to get 26.7 MB of archives.
After this operation, 49.7 MB of additional disk space will be used.
Java memory sizes:
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
export CATALINA_OPTS=-Xms16m -Xmx256m;
Few other notes:
Concept of object is language specific: can use JMX for Java
External Links:
http://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jmx/details
http://blog.lesc.se/2012/02/how-to-take-java-heap-dump-from-command.html
http://visualvm.java.net/heapdump.html
http://www.rallydev.com/community/engineering/outofmemoryerror-fun-heap-dump-analysis
http://java.dzone.com/articles/java-heap-dump-are-you-task
http://architects.dzone.com/articles/how-analyze-java-thread-dumps
http://www.javaworld.com/article/2072864/heap-dump-and-analysis-with-visualvm.html
https://blog.codecentric.de/en/2011/08/create-and-understand-java-heapdumps-act-4/
Subscribe to:
Posts (Atom)
