Little's Law on Performance Test

Little's Law, as a part of Queuing theory, which was introduced to me by Wilson Mar about 2 years ago. We can make use of it to help performance test planning and modeling.

I also would like to usually calculate the arrival rate of system :

L = λW
λ = L/W

Where,
W = Average response time + Think time
L = Virtual Users we simulate in load generation tool

After calculating the arrival rate λ , I will compare arrival rate λ with Throughput X to see if the system can catch up the incoming load, if not, then start tuning it!

One example:

W =(0.352 + 0); -- which means no think time for test
L = 8;

then,
λ = L/W = 8/(0.352 + 0) = 22.73

Meanwhile, get Throughput measured by load generation tool
Throughput = 22
Not that bad currently :)

Some Update after several years I met with such an elegant equation, following results are what i have worked with my perf team to prove the Little's law:

Little's Law for Finding Response Time: 

MeanResponseTime(AVG) = MeanNumberInSystem(VUs) / MeanThroughput(TPS)

Assumptions:

• Stable system, in the long terms
• No Obvious System Bottlenecks
• Similar Actions in the system

Test And Measurement -- Get XXX Pages: 

VUs	AVG(ms)	90%(ms)	TPS	Calculated VUs using Little's Law
1	58	72	16.98	0.99
12	74	108	161.63	11.96
25	83	121	297.99	24.73
50	138	209	360.67	49.77
75	197	349	379.59	74.78
100	261	492	381.21	99.50
125	326	636	381.80	124.71

Predicting and Modeling: 

• Find relations between VU and Avg Response Time By Curve fitting:

y = 0.0067x2 + 1.3698x + 53.613

• Response Time Validation: 

Vus	Predict Result	Test Result
35	109.7	101
65	170.9	172
90	231.1	238
110	285.3	290

• Find relations between VU and Throughput By Curve fitting:

y = 82.39ln(x) + 7.8124

• Throughput Validation: 

Vus	Predict Result	Test Result
35	300.7	342.61
65	351.7	376.35
90	378.5	374.49
110	395	377.74

Simulate Moving your mouse by AutoIT and WebDriver

You can use AutoIT within WebDriver to move your mouse freely, Create your own APIs to simulate your mouse move and click very easily, enjoy it! :)

 autoitx ai = autoitx.INSTANCE;  
 WebDriver driver = new FirefoxDriver();  
 driver.get("http://www.autoitscript.com/autoit3/downloads.shtml");  
 FirefoxWebElement downloadpic = (FirefoxWebElement) driver.findElement(By.cssSelector("img[alt=\"Download AutoIt\"]"));  
 int X = (int) downloadpic.getLocationOnScreenOnceScrolledIntoView().getX();  
 int Y = (int) downloadpic.getLocationOnScreenOnceScrolledIntoView().getY();  
 int offsetX = (int) downloadpic.getSize().getWidth();  
 int offsetY = (int) downloadpic.getSize().getHeight();
 int posx = X + offsetX/2;  
 int posy = Y + offsetY/2;  
 ai.AU3_MouseMove(posx, posy, 20);  
 ai.AU3_MouseClick("left", posx, posy, 1, 10);  

Reference for my previous post on WebDriver and AutoIT: http://joychester.blogspot.com/2011/01/deal-with-file-download-in-webdriver-by.html

Set up Browser caching with Caution to speed up your website

"If you use tech improperly , things will be messed up!"

There are good start tips for Browser caching strategy posted by HttpWatch long time ago: http://blog.httpwatch.com/2007/12/10/two-simple-rules-for-http-caching/
It works great in most of situation.

However, if you have your website more complex, such as using ajax, the post above may not be sufficient.

I found interesting post from caching on IE for caching Ajax requests:

This means that any Ajax derived content in IE is never updated before its expiration date – even if you use a forced refresh (Ctrl+F5)—which means even you add “Pragma: no-cache” header with your request.
The only way to ensure you get an update is to manually remove the content from the cache.

Detail info goes to Fact#2 of this post: http://blog.httpwatch.com/2009/08/07/ajax-caching-two-important-facts/

Another conclusion for browser caching:

“In order to ensure consistent caching behaviour with IE and Firefox you should:
Always specify an Expires header. It will normally be set to -1 for immediate expiration of HTML pages or a date well into the future for other resources such as images, CSS and Javascript;
If you want to force a page to be reloaded, even with the Back button, then use Cache-Control: no-cache, no-store”

Detail info goes to this post: http://blog.httpwatch.com/2008/10/15/two-important-differences-between-firefox-and-ie-caching/

How to set up Cache-control and Expire header on Apache web Server based on your derision and deep understanding of your application:
http://www.askapache.com/htaccess/speed-up-your-site-with-caching-and-cache-control.html

Deal with File download in WebDriver by AutoIT

"This is annoying , but you have to face it sometimes"

Solution:
1> Install AutoIT V3.2, not the latest version, like V3.3.6, BTW, you had better install AutoIT to its default folder:"C:\Program Files\AutoIt3"
2> Download jWinAuto
3> Add all .jars to your libs, including jna.jar and jAutoIt.jar within jWinAuto
4> Write your testing code
I put the Watir scripts in comments for each steps, FYI:

 import org.openqa.selenium.By;  
 import org.openqa.selenium.WebDriver;  
 import org.openqa.selenium.firefox.FirefoxDriver;  
 import com.compdev.jautoit.autoitx.autoitx;  
 public class AutoIT {  

      public static void main(String[] args) throws InterruptedException {  

           //ai = WIN32OLE.new("AutoItX3.Control")  
           autoitx ai = autoitx.INSTANCE;  
            //ie=FireWatir::Firefox.new  
           WebDriver driver = new FirefoxDriver();  
            //ie.goto("http://www.autoitscript.com/autoit3/downloads.shtml")  
           driver.get("http://www.autoitscript.com/autoit3/downloads.shtml");  
            //ie.image(:alt,'Download AutoIt').click  
            driver.findElement(By.cssSelector("img[alt=\"Download AutoIt\"]")).click();  
            //res=ai.WinWait("Opening autoit-v3-setup.exe","",10)  
            int res = ai.AU3_WinWait("Opening autoit-v3-setup.exe","",10);  
            // if res = 1, then it shows you find the window with proper title  
            System.out.print("res" + res);  
            Thread.sleep(3000);  
            //res=ai.WinActivate("Opening autoit-v3-setup.exe")  
            ai.AU3_WinActivate("Opening autoit-v3-setup.exe","");  
            ai.AU3_Send("{TAB}",0);  
            Thread.sleep(3000);  
            ai.AU3_Send("{TAB}",0);  
            Thread.sleep(3000);  
            ai.AU3_Send("{ENTER}",0);  
            //Start file downloading...
      }  
 }  

So you can use AutoIT in WebDriver to deal with your Win32(disgusting) pop-ups or other situations, easily! Want to know how powerful the AutoIT is, please refer to its function reference :)

BTW, just realize WebDriver is implementing a batch of GUI event driven APIs, Cool!

Check JVM runtime compiler by VisualVM

One of my friends asked me how to determine what kind of runtime compiler of a JVM is using, the 'Server' or 'Client'?

there is some default value listed on Ergonomics, but if you want to make sure of it on your server?

It is not that hard to check, the easier way is to use VisualVM on Overview tab:

Usually, you had better add "-server" parameter to ensure a server mode on your machine.

The very first OOM in 2011

Here is how my first OOM of 2011 being detected:

2011/01/04, 2PM:
My defualt JVM setting:

 
...
 wrapper.java.additional.5=-server  
 wrapper.java.additional.6=-Xss128k  
 wrapper.java.additional.7=-Xms1536m  
 wrapper.java.additional.8=-Xmx1536m  
 wrapper.java.additional.9=-XX:MaxPermSize=64m  
 wrapper.java.additional.10=-Xmn512m  
 wrapper.java.additional.11=-XX:ParallelGCThreads=5  
 wrapper.java.additional.12=-XX:+UseConcMarkSweepGC  
...

2011/01/04, 2:01PM:
Changes from -Xmn=512m to -XX:NewRatio=2

 
...
 wrapper.java.additional.5=-server  
 wrapper.java.additional.6=-Xss128k  
 wrapper.java.additional.7=-Xms1536m  
 wrapper.java.additional.8=-Xmx1536m  
 wrapper.java.additional.9=-XX:MaxPermSize=64m  
 wrapper.java.additional.10=-XX:NewRatio=2  
 wrapper.java.additional.11=-XX:ParallelGCThreads=5  
 wrapper.java.additional.12=-XX:+UseConcMarkSweepGC  
...

2011/01/04, 2:10PM:
Load testing running, OOM happened:

 INFO  | jvm 1  | 2011/01/03 23:24:13 | 2011-01-03 23:24:13, 19560420@qtp-9708927-58, ERROR, com.XXX.XXXXX.concept.ConceptMetadataServlet - ConceptMetadataServlet.doGet() exception  
 INFO  | jvm 1  | 2011/01/03 23:24:13 | java.lang.OutOfMemoryError  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at java.util.zip.Deflater.init(Native Method)  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at java.util.zip.Deflater.<init>(Deflater.java:123)  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at java.util.zip.GZIPOutputStream.<init>(GZIPOutputStream.java:46)  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at org.mortbay.servlet.GzipFilter$GzipStream.doGzip(GzipFilter.java:525)  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at org.mortbay.servlet.GzipFilter$GzipStream.<init>(GzipFilter.java:417)  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at org.mortbay.servlet.GzipFilter$GZIPResponseWrapper.newGzipStream(GzipFilter.java:392)  
 INFO  | jvm 1  | 2011/01/03 23:24:13 |      at org.mortbay.servlet.GzipFilter$GZIPResponseWrapper.getOutputStream(GzipFilter.java:342)  
....
NFO   | jvm 1    | 2011/01/03 23:24:14 | #
INFO   | jvm 1    | 2011/01/03 23:24:14 | # A fatal error has been detected by the Java Runtime Environment:
INFO   | jvm 1    | 2011/01/03 23:24:14 | #
INFO   | jvm 1    | 2011/01/03 23:24:14 | # java.lang.OutOfMemoryError: requested 87744 bytes for Chunk::new. Out of swap space?
INFO   | jvm 1    | 2011/01/03 23:24:14 | #

2011/01/04, 2:20PM:

reproduced OOM again, exactly the same one...

2011/01/04, 2:30PM:

buy some drink and continue to do the search on the web

2011/01/04, 2:36PM:

Search keywords:java.lang.OutOfMemoryError, org.mortbay.servlet.GzipFilter$GzipStream.doGzip
It is said to be a Known issue; and it is tagged as fixed already in Jetty version 6.1.23

2011/01/04, 2:45PM:

Changes back from -XX:NewRatio=2 to -Xmn=512m
No OOM happens again after a few rounds of tests....
An idea comes into my mind: "NewRatio seems to be another Factor to this OOM?!"

2011/01/04, 3:25PM:

Search KeyWords: NewRatio, Out of swap space
Another Known issue being stamped: "-XX:NewRatio completely ignored when combined with -XX:+UseConcMarkSweepGC"
Also a Regression bug after above issue is said to be "fixed"

Happy New Year!!

WOW,Continual Full GC Detected!

'Continual Full GC',I will KILL you...

Updated:
Root Cause: when using Axis as an implementation of the SOAP, when Web Service request coming with the attachment over 1MB, then it will make trouble with Heap Memory.
Lot of OOM issues after searching on the web, and no ideal solution with that right now on Apache Axis:

1. SAX2EventRecorder causes Out of memory during Deserialization https://issues.apache.org/jira/browse/AXIS-2698

2. SAX2EventRecorder throws Out of Memory exception during Deserialization https://issues.apache.org/jira/browse/AXIS-2749

Solution:Just Replace the Apache Axis which project has been suspended for quite long time since 2006.

Take Away:
1. When you find Memory Heap Issues, you need to get HeapDump as possible as you can, the other info can be useless... We use SendSingal.exe(for Windows) to trigger the JDK5 to get the Heapdump, when we noticed the Continual Full GC
2. Analysis HeapDump file together with Functional QA, they can give some idea from business perspective, in order to reproduce your issue on local much easier.
After reproducing the issue, then you will solve your pain sooner or later, there will be something wrong with your code :)

Track my Performance Data with WebDriver test by MongoDB

In my WebDriver test, I will always track performance data during automation test on daily bases, it helps to reuse automation scripts which QA team donated to track slowness cases, and guarantee the test coverage very well.
Here is a sample tracking diagram (X-axis: Date, Y-axis: Response time):

Meanwhile, the data is End-End response time,not only back-end html generation time, which load testing tool usually measures.

Testing Code:

MongoDBReader conn = new MongoDBReader();  
WebDriver driver = new FirefoxDriver();  

long starttime = new Date().getTime();  
driver.get("http://www.google.com/");  
conn.InstertPermLog(starttime, "Landing_Page");  

driver.close();  

MongoDBReader source code:

 import java.net.UnknownHostException;  
 import java.util.Date;  
 import java.text.DateFormat;  
 import java.text.SimpleDateFormat;  
 import com.mongodb.BasicDBObject;  
 import com.mongodb.DB;  
 import com.mongodb.DBCollection;  
 import com.mongodb.Mongo;  
 import com.mongodb.MongoException;  
 /* sample usage  
  * MongoDBReader conn = new MongoDBReader();  
  * ...  
  * long starttime = new Date().getTime();  
  * driver.get("http://www.google.com/");  
  * conn.InstertPermLog(starttime, "Landing_Page");  
  */  
 public class MongoDBReader {  
      private DB db;  
      private DBCollection coll;  
      public MongoDBReader(String hostname) throws UnknownHostException, MongoException{  
           // Connect to Mongo DB server  
           Mongo m = new Mongo( hostname );  
           // Get Mongo DB Connection  
           db = m.getDB("perfdb");  
           // Prepare Collection  
           coll = db.getCollection("perfTestColl");  
      }  
      public MongoDBReader() throws UnknownHostException, MongoException {  
           //Connect to the localhost MongoDB by Default  
           String hostname = "localhost";  
           // Connect to Mongo DB server  
           Mongo m = new Mongo( hostname );  
           // Get Mongo DB Connection  
           db = m.getDB("perfdb");  
           // Prepare Collection  
           coll = db.getCollection("perfTestColl");  
      }  
      public DB getDB(){  
           return db;  
      }  
      public DBCollection getCollection(){  
           return coll;  
      }  
      public void InstertPermLog(long startTime, String transactionName){  
           long endtime = new java.util.Date().getTime();  
           long responsetime = endtime - startTime;  
           DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss.S");  
        Date date = new Date();  
           BasicDBObject doc = new BasicDBObject();  
           doc.put("TimeStamp", dateFormat.format(date));  
           doc.put("Rtime", responsetime);  
           doc.put("transaction", transactionName);  
           coll.insert(doc);  
      }  
 }  

MongoDB Shell console:

> db.perfTestColl.find();

{ "_id" : ObjectId("4d078fbb3599030f8a412be1"), "TimeStamp" : "2010/12/14 23:39:

39.546", "Rtime" : NumberLong(4609), "transaction" : "Landing_Page" }

{ "_id" : ObjectId("4d078fc33599030f8b412be1"), "TimeStamp" : "2010/12/14 23:39:

47.953", "Rtime" : NumberLong(1641), "transaction" : "Landing_Page" }

{ "_id" : ObjectId("4d078fcb3599030f8c412be1"), "TimeStamp" : "2010/12/14 23:39:

55.578", "Rtime" : NumberLong(1360), "transaction" : "Landing_Page" }

O'Reilly Velocity China 2010

I am starting to do Performance Testing since 2006. And I have followed Steve about Front-end Performance optimization from 2007, although i am mainly focus on the testing and back-end tuning at most of time.
A big Fan of him, because always can get many fresh ideas and best practices after reading his blog, if you haven't read his blog, suggest to catch up with it :) （Seems twitter has become his major updated place right now...）

Sorting algorithm in Perl

My Girl Friend likes the language that easy to write but hard to read, just like Perl which makes me crazy :)

Here are 3 simple examples which is illustrating her algorithm basic skills(BTW, she is a math talented person i ever met during college)

Insertion Sorting：

 @arry1 = (12,32,3,25,67,24,56);  
 @arry2 = shift @arry1;  
 $j=$#arry2;  
 foreach $arry (@arry1){  
  if ($arry2[$j]<= $arry) {  
   push @arry2, $arry;  
  }  
  elsif ($j==0 ) {  
   unshift @arry2, $arry;  
  }  
  else {  
   foreach (reverse(@arry2)){  
    if ($_ > $arry){  
    $j-=1;  
    }  
   }  
   if ($j >=0){  
   $i = $j+1;  
   @arry2= (@arry2[0..$j], $arry, @arry2[$i..$#arry2]);  
   }  
   else {  
    unshift @arry2, $arry;   
   }  
  }  
  $j=$#arry2;  
 }  

 foreach (@arry2){  
   print "$_ ";   
  }  

Quick Sorting:

 @arry1=(12,32,3,25,67,24,56);  
 $i=0;  
 $j=$#arry1;  
 &quick_sorting($i,$j);  
  sub quick_sorting {  
   my($m,$n)= @_;  
   while ($m <$n){  
    if ($arry1[$m]<= $arry1[$n]) {  
     $n-=1;  
     }  
    else {  
      ($arry1[$m], $arry1[$n]) = ($arry1[$n], $arry1[$m]);  
      $m+=1;  
     }  
    }  
   if ($m - $_[0]> 1) {  
    &quick_sorting($_[0], $m-1);  
    }  
   if ($_[1] -$n > 1){  
    &quick_sorting($n+1, $_[1]);  
    }     
  }  
 foreach $arry (@arry1){  
  print "$arry\n";   
 }  

Bubble Sorting：

 @arry1 = (12,32,3,25,67,24,56);  
 $i=0;  
 $j=$#arry1;  
  for ( ; $j >=0; $j-=1){  
   for ($i=0; $i<$j; $i+=1 ){  
   if ($arry1[$i]-$arry1[$i+1] gt 0) {  
     ($arry1[$i], $arry1[$i+1])=($arry1[$i+1], $arry1[$i]);  
     }  
   }  
  }  
  foreach $arry (@arry1){  
   print "$arry\n";   
  }