Visualize your requests

In the last year, customers often complained about our bad performance. We had just fixed a small memory leak (which crashed our publishing instances about every hour or so), so we were quite interested in getting reliable data to confirm or deny their anger. That time I thought that we need to have a possibility to get a quick overview of the performance of our CQ instances. One look to see “Ok, it must be the network, our systems perform great!”

So I dug out my perl knowhow and wrote a little script which parses through a request.log and prints out data which which is understood by gnuplot. And gnuplot draws then some nice graphs of it. It displays the number of requests per minute and also the average request duration for these requests.

request-graph-all(Click on the image for a larger version.)

These images proved themselves as pretty useful, because you show them to your manager (“Look, the average response went down from 800 miliseconds to 600 although the number of requests went up by 30%.”) and they help you in daily bussiness, because you can spot problems quite well. When at a certain time the response times go up, you better had a look at the system and find the reason for it.

request-graph-html-ukBecause this scripts is quite fast (it parses 300 megabytes of request.log in about 15 seconds on a fast Opteron-based machine), we usually render these images online and integrate the resulting images in a small web application (no CQ but a small hacked up PHP script). For some more interactivity I added the possibility to display only the requests which matches a certain string  (click on the image to view a larger version). So it’s very easy to answer questions such “Is the performance of my landing page that bad as customer report?”

You can download this little perl script here. Run it with “–help” first and it will display a little help screen. Give a number of request.log files as parameter to it, pipe the output directly into gnuplot (I tested with version 4.0, but will probably also work with newer versions) and it will output a png file. Adjust the scripts to your needs and contribute back, I released it under GPL version 2.

(For the hackers: Some things can probably be performed better and I also have some new functionality already prepared in it, but not active. Patches are welcome :-))

Advertisements

3 thoughts on “Visualize your requests

  1. Bastian

    Hi there, i tried to get the script running on my windows box but i’ve not been very lucky until now :-/

    You pipe the scriptoutput right to gnuplot right? like: perl graph-request-log.pl | wgnuplot.exe

    1. jhoh228 Post author

      Hi Bastian

      Yes, I pipe it directly into gnuplot. If my DOS knowledge is right, it should also work on Windows. In case it doesn’t, you can redirect the output into a file and then process the file with gnuplot.

  2. Bastian

    Hmm i dug deeper into the script now it gives me following output


    #!/usr/bin/gnuplot -persist

    set timefmt "%d.%m.%Y %H:%M"
    set format x "%H:%M"
    set xdata time
    set title "(no title given) "
    set xlabel "Time"
    set ylabel "answered requests [1/min]"
    set y2label "delivery time per request [ms]"
    set term png small size 770,400
    set y2tics
    set logscale y2
    set grid

    #set logscale y2 2
    #set y2range [:5000]

    plot "-" using 1:4 axes x1y2 title "average delivery time per request " , \
    "-" using 1:3 title "number of delivered requests"
    e
    e

    when i uncomment line 116 it prints me the duration (wich is right) but it wont appear in the sniplet wich will be piped to gnuplot. Any idea?

Comments are closed.