Apache Per Second Request Count And Plotting That Data With GnuPlot

Over here I will explain how can get your Apache's http server's per second request count and plot the same data using the gnuplot.

While I was trying to get per second request count, parsing the apache httpd access log to plot with the standard apache log format, I found some difficulty, as the log was getting written to the log file in async format, So though of adding "epoch" time on the log and do the next things. Although you can see this status with server-status module [ http://amitmund.blogspot.in/2013/12/apachehttpservers-server-status.html ]

What I did to add the epoch time:

I am using the following LogFormat [ enabled at my /etc/apache2/sites-enabled/000-default ] at my site:
CustomLog "|/usr/sbin/rotatelogs /mnt/httpd/logs/access_log.%Y-%m-%d-%H_%M_%S 100M" combined

So, I have to update the same "combined" format at my apache2.conf file, and I have added the following format at my LogFormat, so that it will record the epoch time.

NOTE: Following line %{s}t will add epoch time at the log

LogFormat "%{%s}t %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

Before that I had a log as:

10.210.193.192 - - [26/Dec/2013:10:16:25 +0000] "GET / HTTP/1.1" 200 517 "-" "check_http/v1.4.14 (nagios-plugins 1.4.14)"

Now I can see a epoch time added before the above log format:

1388744705 10.210.193.192 - - [26/Dec/2013:10:16:25 +0000] "GET / HTTP/1.1" 200 517 "-" "check_http/v1.4.14 (nagios-plugins 1.4.14)"

Further details about the log format:

http://httpd.apache.org/docs/2.2/mod/mod_log_config.html

Few of the useful options:

%{%s}t : epoch time

%{format}t : The time, in the form given by format, which should be in

        strftime(3) format. (potentially localized)

%D : The time taken to serve the request, in microseconds.

%f : Filename

% : The request method

%s : Status. For requests that got internally redirected, this is

        the status of the *original* request --- %>s
        for the last.

%T : The time taken to serve the request, in seconds.

After updating the logFormat, I did a apache restart and cross checked to see the update format and found its working fine.

Getting Data:

After the log format got updated, I did a grep of the epoch time and redirected to a file.

$ awk '{print $1}' access-file > epochTime.dat

The above command will take the epoch time from the access-file and put the same at epochTime.dat file. [ Over here I have added epoch time at the 1st field, so I have used the $1 over here.

This is useful if you are doing a benchmark of your web server and want to know and plot how many apache request it is serving every second.

Using above command, you will find the data like following:

$ head epochTime.dat

1388745807
1388745807
1388745807
1388745807
1388745807
1388745807
1388745807
1388745807

Now you have to get the count of the same epoch time: so,

$ cat epochTime.dat | sort | uniq -c 

By above command you will see the output as: 

    2 1388745779

    400 1388745780

   1016 1388745781

   1048 1388745782

   1057 1388745783

   1063 1388745784

    440 1388745785

Now I did:

awk '{print $2,$1}' epochTime.dat > epochTime1.dat

After the above command you can use the above data to plot graph easyly:

Now you get the data something like following:

$ head epochTime1.dat

1388745779 2

1388745780 400

1388745781 1016

1388745782 1048

1388745783 1057

1388745784 1063

1388745785 440

Now I have the data How I will plot the same:

I am using the gnuplot to plot the above data. [ other gnuplot example: http://amitmund.blogspot.in/2014/01/gnuplot.html ]

$ vi PerSecondReq.gplot: # the gnu plot code to plot the data

set xdata time
set timefmt "%s"
set xlabel "TIME"
set ylabel "Req"
plot "epochTime1.dat" using 1:2 title "Req" with lines

Now You have your plotted data.[ Following is an example of the same. ]