Monitoring your load generator client, when the server CPU% is under utilized

Recently, I am using one Virtual Box(VMware/2 Cores / 8GB) to conduct performance/load testing. With more and more load Jmeter generate, It seems the server's CPU% is not able to go up...
However, I noticed the my vitual-box client's CPU% has reached to 90%+. After changing to a physical and more powerful machine, everything is fine, server side CPU% can go up with increasing payload. I am not often to monitoring load generator client during my testing previously, and they are usually works fine as its physical machine.
You have to monitor your load generator machine as well when you find the server CPU% is under utilized(stay the same with even higher user load) and Server response time keep increasing, especially if you are using a Virtual Box to do performance testing, then please be caution!!

Deal with Modal Dialog in my WebDriver test

If  you do not have an idea how to deal with the Modal Dialog in WebDriver Tests, here is my solution to it
For example the modal dialog pop-up after clicking "Logout" link, so the code like this:

 driver.findElement(By.linkText("Logout")).click();  
 driver.switchTo().activeElement().sendKeys(Keys.ENTER);  
 driver.switchTo().alert().accept();  

"Mouse" hover to your WebElement as a bonus in case you are looking for:

 //simulate the mouse hover  
 Actions builder = new Actions(ldriver);  
 Action action = builder.moveToElement(sportslink).build();  
 action.perform();  

Hope it helps!

Performance doc written by me and the team

First of all, Thanks to Neil Feng, if you are not pushing me hard, I won't write something down :) Thank you and the team!You are awesome guys!!

I think it is a good things can be shared with anyone who is interested in performance engineering stuff, so in case you want to take peek at it...:

https://docs.google.com/open?id=0B_hz_q1i5QXJTXZqd0FVblAzNms 

Hope it helps! Any comments or suggestions are welcomed!

--Cheng Chi

Add time stamp to your GC log file name

To avoid overwriting your gc log once the service restart expected, you had better to add a time stamp to your gc log file, simply as this:

 rem Tested FOR WIN7 OS:  
 rem set time format:  
 set hour=%time:~0,2%  
 if "%hour:~0,1%" == " " set hour=0%hour:~1,1%  
 set min=%time:~3,2%  
 if "%min:~0,1%" == " " set min=0%min:~1,1%  
 set secs=%time:~6,2%  
 if "%secs:~0,1%" == " " set secs=0%secs:~1,1%  
 rem set date format:  
 set year=%date:~-8,4%  
 set month=%date:~0,2%  
 if "%month:~0,1%" == " " set month=0%month:~1,1%  
 set day=%date:~3,2%  
 if "%day:~0,1%" == " " set day=0%day:~1,1%  
 set datetime=%year%%month%%day%_%hour%%min%%secs%  
 set GCLOG=-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc-%datetime%.log  

The output like this: gc-20120507_172101.log
Notice: "-XX:+PrintGCDateStamps" option is only available for JDK6u4+, otherwise, you need add "-XX:+PrintGCTimeStamps" instead. Never leave any space at the end of line in your batch script, it will trigger stupid result...

The Third Party Performance Problems

We live by the third parties, but do not expect them perfect...

We used a lot of third party stacks on each project, and they are almost used in the similar way most of time, so creating this lists to avoid such kind of traps in future, just give you some examples we met before:

1> HashMap.put() is not thread safe, however Axis manipulate it in parallel which leads to an infinite loop (it will cause 100% CPU usage unless restart the service)
Bug for tracking: https://issues.apache.org/jira/browse/AXIS-2792
Update:
a good post for explaining this problem in detail: http://coolshell.cn/articles/9606.html
here is an English version: http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html
and how Doug Lea said about this in Sun's bug tracking system a few years ago: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6423457

2> Jetty service no responding by Deadlock (The Jetty can not respond to any coming request ,you have to restart the service)
Bug for tracking: http://jira.codehaus.org/browse/JETTY-1264

3> A JDK regression results in an indefinite wait for a connection which is created by Jtds or other JDBC drivers, so do not upgrade JDK version 1.6.0_27+ (said to be fixed in _30)
Bug for tracking: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7105007

4> Jtds-1.2.2 has introduced a few performance issues , including memory leak and infinite loop issues, you may consider to upgrade it after checking the history release notes:
http://sourceforge.net/news/?group_id=33291

Feel free to add more findings ;)

Algorithem Matters to Performance

If we want to find all Prime number within a positive range, how would you figure it out? I want to share with you with a simple code on how algorithm matters to performance: I am using Ruby to demo the difference between 2 ways to organize your code and how they impact the performance: Without any optimization:

 require 'ruby-prof'  
 def findprimenumber(uppernumber)   
  count =0;   
  for i in 3..uppernumber   
   primebool = true;   
   for j in 2...i   
    if (i%j!=0)   
    next;   
    else   
    primebool = false;   
    break;   
    end   
   end   
   if(primebool)   
   count= count+1   
   end   
  end   
  puts("total prime count:#{count}");   
  puts RubyProf.measure_process_time   
  end   

With optimization:

 require 'ruby-prof'   
 def findprimenumberopt(uppernumber)   
  count =0;   
  for i in 3..uppernumber   
   primebool = true;   
   if i%2==0   
   primebool = false;   
   next;   
   else   
   *for j in 3..Math.sqrt(i).floor+1 *  
    if (i%j!=0)   
    next;   
    else   
    primebool = false;   
    break;   
    end   
    *j=j+2;*  
   end   
   end   
   if(primebool)   
   count= count+1   
   end   
  end   
  puts("total prime count:#{count}");   
  puts RubyProf.measure_process_time   
  end   

Given the uppernumber to 100000 The performance result without optimization code :47s The performance result with optimization code :0.9s Also you may try more optimization way such as divide and conquer by parallel computing...

The Parenthesis Matched?

 def parentmatched(inputstring)  
  while inputstring != nil  
   p inputstring  
   break if (inputstring.gsub!(/\(\)/, '')==nil)  
  end  
  if inputstring.eql?('')  
   puts "matched"  
  else  
   puts "non-matched"  
  end  
 end  

 parentmatched("((())(())()(()))()");  

MAT(Eclipse Memory Analyzer Tool) Usage Simple Check list

Those are some basic steps to do heap dump analysis using MAT when there comes an OOM or memory leak issue:

Pre-condition:Config MAT correctly(you may edit JVM Heap size if it is not sufficient to open a .hprof file, for example, modify "-vmargs-Xmx1280m" to MemoryAnalyzer.ini)
1. Quick Overview on Leak Suspects Report: it will give you some indicator to the leak suspects or the memory issue, it shows the biggest retained heap size after grouping by the same instance name
2. Quick Overview on Overview Pane: it lists the each biggest individual object by retained heap size
3. Open histogram to list number of instances per class, within the histogram, you can check:

• Order by Shallow heap size to see if there is a particular object taking too much memory size, then drill it down by showing its dominator tree view for further analysis
  
• Order by Retained heap size which represents the amount of memory that will be freed by the garbage collector when this object is collected, then drill it down by showing its dominator tree view for further analysis
  
• Order by Objects Count to see if a lot of objects are accumulated, and take the further analysis by viewing the "merge shortest paths to GC roots" (Who is referencing them)

4.Some specific class needs to be paid attention to, for example "org.apache.catalina.core.ApplicationHttpRequest", it includes the very useful attributes for trouble shooting and reproducing the problems which are RequestURI and queryString

--Cheng Chi

Extend Btrace

If you would like to build you own methods in Btrace, you would prefer to checking out Btrace source code with Netbeans6.9plus and Mercurial, for detail steps, please refer to:
http://netbeans.org/kb/docs/ide/mercurial.html
the Repository URL: https://hg.kenai.com/hg/btrace~hg

Basically, I have added several methods to help monitoring the JVM Performance, for example:

• currentProcessCpuTime
• currentProcessCpuUsage
• getTotalCollectionCount
• getGCThroughput
• getMinorGCDetailInfo
• getFullGCDetailInfo
• jstackAllFile(dump all thread stack trace to local file)

You can take a look at the source code i extend from my github, and extend yours if you like:
https://github.com/joychester/btrace_Ext

Question about benefits of Web automation

One of my friend asked a question about Automation recently :"is UI test automation actually providing anything useful?"

I Want to share some of thoughts and bias from my own experience and perspective:

The UI automation is a kind of faith. Once you believe it, you need to advocate a lot at first(need dedicate resource and patient from management team), then you get your pay back after a period of time gradually.

We should not exaggerate its effect/scope(So do expectation management to the managers are really important and critical). But it really helps in following situations i can think of:

1. Prevent the UI regression bugs (Yes, Prevent Regression not Find bugs, due to you need to write workable automation code based on the workable scenario, so once the automation test failed, then you have got a regression bug)

2. Reduce the execution time, so as the human resources.(You can run test anywhere, anytime without anyone, and then collect test result and analyze it afterwards) Also it can be scaled easily if you have tons of tests to be run. So the execution time is critical to the projects.

3. It is reliable and consistence(Not like Human, Computer always tell the truth :) )

4. QA can help with that.It is black box testing, so no need to know the detail implementation of "complex" technique stuff. (seems most of QA person would like to learn and write UI automation tests from my observation)

5.Easy to track the history data for Automation test.Automation test result can be kept easily, in which history data can reveal something about quality and people, who always break the build, who always write the bad code and who always write the bad test code, etc.

6. Help to do the application health check for QA. Health check is time consuming and always repeatable but not avoidable in some situation, so it is really helpful to do automation in this situation.

7.... many more can get from the web i am sure,so i will not mention that to waste of your time, lol

For Practical Way of doing Web UI Automation test, I have done some research about that in my previous days in 2010 although i am not an automation guy... So hope my thoughts can give you some clues to this tough question :)

It is deserved to take a look at the WebDriver project which has been merged to Selenium 2.0 project: http://code.google.com/p/selenium/ 

In this project, apart from the multiple drivers implementation and clean APIs, the most Important and Essential idea of how to do the Web Automation is introducing the Page Objects Pattern:http://code.google.com/p/selenium/wiki/PageObjects, which implement the Automation test as a OOD/OOP way. It is much easier to maintain the code and tests than traditional way (which is a record/playback, and step by step describe way to write the automation code).

Anyway, The manual test can be never replaced by Automation test, however, Web UI Automation is also great complement to improve your quality and deserve to put great effort to invest the future. Everything is from the initial purpose, so if Web Automation's strong points can be matched with the project real needs, so just do it with patient and no doubt. If they just want to produce instant results and cut down the bug number on production badly(as i said, Auto test is just preventing bugs not help to find new bugs), then it might loose them confidence i am afraid...

Apache Memory keep increasing due to Rotate logs (with so many Vitural hosts)

We have a lot of Private labels for our application. We managed their access log and Error log in each Virtual host with Apache Http server.

If CustomLog or ErrorLog directives are placed inside a section, all requests or errors for that virtual host will be logged only to the specified file.This technique is very useful for a small number of virtual hosts, but if the number of hosts is very large, it can be complicated to manage. In addition, it can often create problems with insufficient file descriptors.

We 'luckily' got its limitation recently by adding 2 more private labels recently. So the Apache memory is keeping increasing gradually even if there is no any requests coming in.

When using a large number of Virtual Hosts, Apache may run out of available file descriptors (sometimes called file handles)if each Virtual Host specifies different log files.The limit is typically 64, and may usually be increased up to a large hard-limit.

Solution to our case:

1. We can combine all the Error logs into one Common Error log instead of creating their own within each Virtual host. So it will reduce a lot of file handles at once.
Note: If it does not work for you , then you need have to reduce your number of Virtual Hosts with single box...

2. Or you can try to adjust file handles limit before starting Apache, for example 100:

    #!/bin/sh

    ulimit -S -n 100

    exec httpd

Apache official Reference:

http://httpd.apache.org/docs/2.2/logs.html#virtualhost

http://httpd.apache.org/docs/2.2/vhosts/fd-limits.html

It is worthy to take look at the official doc carefully before using its cool features, although it is hard :)