Amit Mund: Monitoring java process

Few notes on how to monitor the java application:

Following is one of the command, using this we can get some information on java status.

$ jstat --help
invalid argument count
Usage: jstat -help|-options
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]

Definitions:
<option> An option reported by the -options option
<vmid> Virtual Machine Identifier. A vmid takes the following form:
<lvmid>[@<hostname>[:<port>]]
Where <lvmid> is the local vm identifier for the target
Java virtual machine, typically a process id; <hostname> is
the name of the host running the target Java virtual machine;
and <port> is the port number for the rmiregistry on the
target host. See the jvmstat documentation for a more complete
description of the Virtual Machine Identifier.
<lines> Number of samples between header lines.
<interval> Sampling interval. The following forms are allowed:
<n>["ms"|"s"]
Where <n> is an integer and the suffix specifies the units as
milliseconds("ms") or seconds("s"). The default units are "ms".
<count> Number of samples to take before terminating.
-J<flag> Pass <flag> directly to the runtime system.

Following is the example: jstat -<option> <pid-of-java-process> <mili-sec-interval> <NumberOfTime>

NOTE: How you will file the pid of some java process. [ e.g: ps -ef | grep java ] of [ jps ]
Once you get the pid of the java process, you can use the following command. E.g: 21253 is one PID of one of the java service.

$ jps
21253 jar
3089 Jps

$ jstat -gc 21253 250 7
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 5706.0 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926

$ jstat -gc `jps | grep -i jar | awk '{print $1}'` 250 7

[ NOTE: This above command is picking the pid of java jar as an param of the command ]

S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926
3264.0 3264.0 0.0 0.0 26240.0 6363.5 735232.0 10732.7 23068.0 13769.4 21 0.099 192 15.827 15.926

$ jstat -options
-class
-compiler
-gc
-gccapacity
-gccause
-gcnew
-gcnewcapacity
-gcold
-gcoldcapacity
-gcpermcapacity
-gcutil

-printcompilation

$ jstat -gccapacity `jps | grep -i jar | awk '{print $1}'` 250 7
NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC PGCMN PGCMX PGC PC YGC FGC
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192
32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192

32768.0 32768.0 32768.0 3264.0 3264.0 26240.0 735232.0 735232.0 735232.0 735232.0 12288.0 65536.0 23068.0 23068.0 21 192

jmap
Usage:
jmap [option] <pid>
(to connect to running process)
jmap [option] <executable <core>
(to connect to a core file)
jmap [option] [server_id@]<remote server IP or hostname>
(to connect to remote debug server)

where <option> is one of:
<none> to print same info as Solaris pmap
-heap to print java heap summary
-histo[:live] to print histogram of java object heap; if the "live"
suboption is specified, only count live objects
-permstat to print permanent generation statistics
-finalizerinfo to print information on objects awaiting finalization
-dump:<dump-options> to dump java heap in hprof binary format
dump-options:
live dump only live objects; if not specified,
all objects in the heap are dumped.
format=b binary format
file=<file> dump heap to <file>
Example: jmap -dump:live,format=b,file=heap.bin <pid>
-F force. Use with -dump:<dump-options> <pid> or -histo
to force a heap dump or histogram when <pid> does not
respond. The "live" suboption is not supported
in this mode.
-h | -help to print this help message

-J<flag> to pass <flag> directly to the runtime system

jps
4919 SecondaryNameNode
4375 NameNode
4668 DataNode
5237 Jps

sudo jmap -heap 4668
Attaching to process ID 4668, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.7-b01

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1048576000 (1000.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 174063616 (166.0MB)
G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
capacity = 32374784 (30.875MB)
used = 21415272 (20.423194885253906MB)
free = 10959512 (10.451805114746094MB)
66.14799962835274% used
From Space:
capacity = 5373952 (5.125MB)
used = 5365648 (5.1170806884765625MB)
free = 8304 (0.0079193115234375MB)
99.84547684832317% used
To Space:
capacity = 5373952 (5.125MB)
used = 0 (0.0MB)
free = 5373952 (5.125MB)
0.0% used
PS Old Generation
capacity = 86245376 (82.25MB)
used = 363240 (0.34641265869140625MB)
free = 85882136 (81.9035873413086MB)
0.4211704057038374% used
PS Perm Generation
capacity = 21757952 (20.75MB)
used = 16151640 (15.403404235839844MB)
free = 5606312 (5.346595764160156MB)
74.23327342573418% used

6521 interned Strings occupying 519584 bytes.

sudo jmap -dump:live,format=b,file=heap.bin 4668
4668: Unable to open socket file: target process not responding or HotSpot VM not loaded

The -F option can be used when the target process is not responding

sudo jmap -F -dump:live,format=b,file=heap.bin 4668

Attaching to process ID 4668, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 23.7-b01

Dumping heap to heap.bin ...

none

null_check

null_assert

range_check

class_check

array_check

intrinsic

bimorphic

unloaded

uninitialized

unreached

unhandled

constraint

div0_check

age

predicate

loop_limit_check

Finding object size using Printezis bits and skipping over...

Heap dump file created

-rw-r--r-- 1 root root 11785562 Dec 23 15:30 heap.bin

amit@amitAsus:~$ du -ksh heap.bin

12M heap.bin

NOTE: I found VisualVM [ GUI tool ] that can be used to read the heap dump.
sudo apt-get install visualvm

-------

You should use jmap -heap:format=b <process-id> without any paths. So it creates a *.bin file which you can open with jvisualvm.exe (same path as jmap). It's a great tool to open such dump files.

------

If you use Eclipse as your IDE I would recommend the excellent eclipse plugin memory analyzer

------

Another option is to use JVisualVM, it can read (and create) heap dumps as well, and is shipped with every JDK. You can find it in the bin directory of your JDK.

You can load in visualvm

-----

You can use jhat (Java Heap Analysis Tool) to read the generated file:

======

jhat [ options ] <heap-dump-file>

The jhat command parses a java heap dump file and launches a webserver. jhat enables you to browse heap dumps using your favorite webbrowser.

Note that you should have a hprof binary format output to be able to parse it with jhat. You can use format=b option to generate the dump in this format.

-dump:format=b,file=<filename>

jhat

ERROR: No arguments supplied

Usage: jhat [-stack <bool>] [-refs <bool>] [-port <port>] [-baseline <file>] [-debug <int>] [-version] [-h|-help] <file>

-J<flag> Pass <flag> directly to the runtime system. For

example, -J-mx512m to use a maximum heap size of 512MB

-stack false: Turn off tracking object allocation call stack.

-refs false: Turn off tracking of references to objects

-port <port>: Set the port for the HTTP server. Defaults to 7000

-exclude <file>: Specify a file that lists data members that should

be excluded from the reachableFrom query.

-baseline <file>: Specify a baseline object dump. Objects in

both heap dumps with the same ID and same class will

be marked as not being "new".

-debug <int>: Set debug level.

0: No debug output

1: Debug hprof file parsing

2: Debug hprof file parsing, no server

-version Report version number

-h|-help Print this help and exit

<file> The file to read

For a dump file that contains multiple heap dumps,

you may specify which dump in the file

by appending "#<number>" to the file name, i.e. "foo.hprof#3".

All boolean options default to "true"

------------

$ jinfo

Usage:

jinfo [option] <pid>

(to connect to running process)

jinfo [option] <executable <core>

(to connect to a core file)

jinfo [option] [server_id@]<remote server IP or hostname>

(to connect to remote debug server)

where <option> is one of:

-flag <name> to print the value of the named VM flag

-flag [+|-]<name> to enable or disable the named VM flag

-flag <name>=<value> to set the named VM flag to the given value

-flags to print VM flags

-sysprops to print Java system properties

<no option> to print both of the above

-h | -help to print this help message

-----

MAT, jprofiler,jhat are possible options. since jhat comes with jdk, you can easily launch it to do some basic analysis. check this out

-----
sudo apt-get install visualvm
[sudo] password for amit:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
kde-l10n-engb
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
antlr3 aspectj javahelp2 junit4 libantlr-java libaspectj-java libfelix-framework-java libfelix-main-java
libgeronimo-jpa-2.0-spec-java libgeronimo-osgi-support-java libhamcrest-java libjna-java libnb-org-openide-modules-java
libnb-org-openide-util-java libnb-org-openide-util-lookup-java libnb-platform13-java libosgi-compendium-java libosgi-core-java
libosgi-foundation-ee-java libservlet2.5-java libstringtemplate-java libswing-layout-java
Suggested packages:
javahelp2-doc libfelix-framework-java-doc libfelix-main-java-doc libgeronimo-jpa-2.0-spec-java-doc
libgeronimo-osgi-support-java-doc libjna-java-doc libnb-platform13-java-doc libosgi-compendium-java-doc libosgi-core-java-doc
libosgi-foundation-ee-java-doc libswing-layout-java-doc
The following NEW packages will be installed:
antlr3 aspectj javahelp2 junit4 libantlr-java libaspectj-java libfelix-framework-java libfelix-main-java
libgeronimo-jpa-2.0-spec-java libgeronimo-osgi-support-java libhamcrest-java libjna-java libnb-org-openide-modules-java
libnb-org-openide-util-java libnb-org-openide-util-lookup-java libnb-platform13-java libosgi-compendium-java libosgi-core-java
libosgi-foundation-ee-java libservlet2.5-java libstringtemplate-java libswing-layout-java visualvm
0 upgraded, 23 newly installed, 0 to remove and 1 not upgraded.
Need to get 26.7 MB of archives.

After this operation, 49.7 MB of additional disk space will be used.

Java memory sizes:

-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size

-Xss<size> set java thread stack size

if you are using Tomcat. Update CATALINA_OPTS environment variable
export CATALINA_OPTS=-Xms16m -Xmx256m;

Few other notes:

Concept of object is language specific: can use JMX for Java

External Links:

http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html

http://www.cubrid.org/blog/dev-platform/how-to-monitor-java-garbage-collection/

http://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jmx/details

http://blog.lesc.se/2012/02/how-to-take-java-heap-dump-from-command.html

http://visualvm.java.net/heapdump.html

http://www.rallydev.com/community/engineering/outofmemoryerror-fun-heap-dump-analysis

http://java.dzone.com/articles/java-heap-dump-are-you-task

http://architects.dzone.com/articles/how-analyze-java-thread-dumps

http://www.javaworld.com/article/2072864/heap-dump-and-analysis-with-visualvm.html

https://blog.codecentric.de/en/2011/08/create-and-understand-java-heapdumps-act-4/

Amit Mund

Pages

Friday, 20 December 2013

Monitoring java process

No comments:

Post a Comment