Performance engineering internal group(team) responsibility

Here is a draft of internal performance group(team) responsibility chart in my mind, so much things to do, we need to prioritize each section better to maximum its value :

Responsibility	Tasks	Role	Delivery
Capacity planning	1. Current user pattern/distribution 2. Gap analysis 3. Predict trend of usage 4. Top slow transactions 5. Top usage of transactions	QA lead, Performance QA(owner) PM, OPS(supportive) SDM, TL(consultant)	Access log analysis result RFP distribution pattern
Performance Environment Setup	1. Setup Performance environment 2. Data preparation 3. Server configuration	SCM/Dev (owner) Performance QA (supportive)	Testable performance environment
Performance test planning and design	1. Performance test strategy design 2. Performance testing scripts/scenario design 3. Keep performance test scripts/ load distribution update to date	Performance QA(owner) Performance Dev(supportive) SDM, TL, QA lead(consultant)	Performance test scripts/ scenarios
Code review and design inspection	1. code review and Performance inspection checklist	Performance Dev , TL(owner) Performance QA(supportive) Arch(consultant)	Code review retro
Performance testing Execution	1. User-story based performance test 2. Weekly performance test 3. Release performance testing 4. End-end performance test 5. Performance initiatives 6. Performance test report	Performance QA(owner) Performance Dev(supportive)	Performance test result report
Performance result analysis	1. Server log issues analysis 2. Performance test result analysis 3. Conclusion of performance Grade and suggestion of further investigation	Performance Dev(owner) Performance QA(supportive)	Identify any potential Performance bottlenecks Escalate Performance issues to the team
Performance improvement planning	1. making follow up plan for escalated performance issues	SDM, TL, QA lead(owner) Performance QA/Dev(supportive)	Create tech user stories for each performance issue Triage the priority of each performance user story
Performance tuning	1. Java back-end performance tuning 2. DB tuning 3. Front-end performance tuning 4. Performance Validation	SDM, TL, Dev (owner) Performance QA(supportive)	Tuning analysis summary Performance improvement comparison result Code review Check-in code into new release
Production Performance issues Follow up	1. Log performance issue in Rally 2. Server log analysis 3. Reproduce performance issue on local 4. Find the root cause of bottleneck	Performance Dev(owner) Performance QA(supportive)	Performance issue analysis result Reproduce suggestions

Analysis Jmeter result log with Google Charts by Ruby

A thought from long time ago, but never pick it up...

Thanks to http://googlecharts.rubyforge.org/, it give me the inspiration on how to combine ruby and Google Charts, then i just make it by ruby

Thanks to my intern Tom as well, he has helped me to re-factoring of my previous code, he made the charts a little bit fancy than mine:)

enjoy it!

Note: the data of bellowing Diagram is not real data , just for testing purpose :)



Ruby Source Code V1.0:

require 'gchart'

def getresdata(block1)
# scan for the first numeric token
str = block1.scan(/\d+/)[0];
return str;
end

# format RGB
def fill(str)
i = 6 - str.length();
while(i > 0)
str = '0' + str;
i -= 1;
end
return str;
end

def format_color(num, boundary)
return "000000" if num < boundary =" 0"> 16777215;

color = boundary;
increment = (16777216 - boundary)/num;

retStr = "%x" % [boundary];
retStr = fill(retStr);

while(num > 1)
color += increment;
str = "%x" % [color];
str = fill(str);

num -= 1;
retStr += ',' + str;
end

return retStr;
end

def Draw_Perf_data(file)
url_sum = Hash.new();
IO.foreach(file) { |line|
next if !line.include?("lb="); # skip when this is xml head line

line.strip!(); # trim
url = line.scan(/lb="\w+/)[0]; # fetch the 'lb=' token
puts url.sub!("lb=\"", ""); # fetch the value of lb

if url_sum.has_key?(url)
url_sum[url].push(getresdata(line).to_i); # exist, then add cell
else
url_sum[url] = Array.new(1, getresdata(line).to_i); #!exist, then create
end
}

datas = Array.new();
actions = Array.new();

url_sum.each { |key, val|
datas.push(val);
actions.push(key);
}

colors = format_color(3, 0);
arrayabc = Gchart.line(:size => "800x375",:title => "Request Log", :data => datas, :line_colors => colors, :legend => actions, :axis_with_labels => ['y'], :custom => "chg=100,25");

puts(arrayabc);
return arrayabc;
end


######################
# Call the method, just input your Jmeter result log as a parameter, note it should be the default format of your logs

Draw_Perf_data("123.txt");

here is my configuration for my JMeter result file:

Performance env health check file

Previously, before i start perf testing, i usually went through the application to see if application works well. Checking if each server's java process running successfully or not at first will save us time(based on my working experience)

By just one click, I can tell the specific server contains Java process or not.

here is source file(BTW, you should input Gawk into the PsTools as well):

set PS_TOOLS_HOME=D:\\PsTools
set JVM_IP1=192.168.1.1

%PS_TOOLS_HOME%\pslist \\%JVM_IP1% -u "domain\username" -p "pwd" -e java | %PS_TOOLS_HOME%\gawk "{if($1 !~ /java/) {} else {print $0, \"\nGot you Java! :-)\"}}"
echo %JVM_IP1% Health check finished

Web access log analysis with Awk for Performance test modeling

I try to extract some end users' actions from Web access log, so that i can design my performance testing scenarios more accurately, not by assumption.

The scripts' purpose is to filter all the .do actions from access.log and i can use the output file to do more complexed analysis.

I named my script as "W3A.txt", here is the source:

 Begin {i=1}  
 $7~/\.do/ {  
 {if (i==ARGIND) {  
 sub(/;jsessionid=[0-9a-zA-Z]+.worker[0-9]/,"",$7 );  
 sub(/\?[a-zA-Z0-9_=&-\/:\?%]+/,"?",$7);  
 print $4,$7,$11 > i  
 }  
 else i++  
 }  
 { if (i> ARGIND)  
 exit 1  
 }  
 }  
 END{}  

(P.S. This log will give me each action's date/time, URL, response time)

how to use it?

$ awk -f W3A.txt access.2009-11-*_perf.log

Generated result files:

1.log
2.log
...

P.S.it can process multiple log files, and also generating result logs based on each log you put into the augments separately. if you just want to generate single result file, i think it is much easier than i did here :)
Hope helps!

JMeter "leaks" by storing the raw data into the heap

almost 1 year ago i wrote about load generated by JMeter decreasing after long time running, looking at this :http://joychester.blogspot.com/2009/01/jmeter-load-get-decreasing-after-long.html

After looking at the Heapdump of JMeter after on day running, we got the suspect:



I prefered to using the Aggregate listener as always, so it indeed to store every single response data into Heap, it may be one of the reason cause Jmeter slower and slower to generate the load itself.

Jmeter __javascript() got blocked

One of my Friends are using JMeter 2.3.2 __javascript() func to generate Random number, but after using VisualVM to monitoring Jmeter, we found such a threaddump:

"Thread Group 1-18" prio=6 tid=0x379c3800 nid=0xf3c runnable [0x39f2f000..0x39f2fd18]
java.lang.Thread.State: RUNNABLE
at org.mozilla.javascript.gen.c123605._c0()
at org.mozilla.javascript.gen.c123605.call()
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:340)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2758)
at org.mozilla.javascript.gen.c123605.call()
at org.mozilla.javascript.gen.c123605.exec()
at org.mozilla.javascript.Context.evaluateString(Context.java:1132)
at org.apache.jmeter.functions.JavaScript.execute(JavaScript.java:94)
- locked <0x0b9991e0> (a org.apache.jmeter.functions.JavaScript)
at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138)
at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:107)
....

"Thread Group 1-17" prio=6 tid=0x390f4c00 nid=0x288 waiting for monitor entry [0x39edf000..0x39edfd98]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.jmeter.functions.JavaScript.execute(JavaScript.java:73)
- waiting to lock <0x0b9994a8> (a org.apache.jmeter.functions.JavaScript)
at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138)
at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:107)
at org.apache.jmeter.testelement.property.FunctionProperty.getStringValue(FunctionProperty.java:87)
at org.apache.jmeter.testelement.property.AbstractProperty.hashCode(AbstractProperty.java:221)
at java.util.HashMap.getEntry(Unknown Source)
at java.util.HashMap.containsKey(Unknown Source)
at java.util.HashSet.contains(Unknown Source)
at org.apache.jmeter.testelement.AbstractTestElement.isTemporary(AbstractTestElement.java:376)
at org.apache.jmeter.testelement.AbstractTestElement.recoverRunningVersion(AbstractTestElement.java:351)
at org.apache.jmeter.threads.JMeterThread.notifyTestListeners(JMeterThread.java:561)
at org.apache.jmeter.threads.JMeterThread.access$200(JMeterThread.java:60)
....

"Thread Group 1-15" prio=6 tid=0x37149000 nid=0x878 waiting for monitor entry [0x39e3f000..0x39e3fa98]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.jmeter.functions.JavaScript.execute(JavaScript.java:73)
- waiting to lock <0x0b998000> (a org.apache.jmeter.functions.JavaScript)
at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138)
at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:107)
....

Here is the source code for your reference :)

public synchronized String execute(SampleResult previousResult, Sampler currentSampler)
throws InvalidVariableException {

JMeterContext jmctx = JMeterContextService.getContext();
JMeterVariables vars = jmctx.getVariables();

String script = ((CompoundVariable) values[0]).execute();
// Allow variable to be omitted
String varName = values.length < 2 ? null : ((CompoundVariable) values[1]).execute().trim();
String resultStr = "";

then you know why...
__javascript() itself got Performance problem, So be Careful to use __javascript when you perform load testing

Generating Heapdump On JDK5(v16+)

Teammates and I are doing one thing to get our life a little bit easier--Generating Heapdump On JDK5(V16+); P.S. unfortunately you could not using this way if you are using lower version of JDK5, like JDK1.5.0_11... A bug reported here for your information, you have to use JDK5(V16+): http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6454676

Thanks Andy and Adams' great idea and Support!!
0. download PsTools and SendSignal and gawk into your target machine as PS_TOOLS_HOME
1. Add -XX:+HeapDumpOnCtrlBreak in your JVM arguments
2. Using SendSignal to simulate the Ctl+Break signal to the service

here is the HeapGen.bat file, hope this helps you too:

 @echo off  
 rem -------------------------------------------------------------------------  
 rem Invoking Remote JVM to do Heap Dump Script for Win32  
 rem 1. Please replace your directory of PsTools  
 rem 2. Please set IP before execute it  
 rem -------------------------------------------------------------------------  
 set PS_TOOLS_HOME=D:\\PsTools  
 set JVM_IP=10.279.33.33  
 set HPROF_HOME=D:\abc\efg\bin  
 echo ===================================================================================  
 echo .  
 echo Your Directory of PsTools is %PS_TOOLS_HOME%  
 echo .  
 echo The Remotely Controlled Machine is %JVM_IP%  
 echo .  
 echo Heap dump is created in a file in the working  
 echo directory of the VM namely %HPROF_HOME%  
 echo .  
 echo =================================================================================== 
 echo .  
 %PS_TOOLS_HOME%\pslist \\%JVM_IP% -u "${Domainname}\${yourusername}" -p "${yourpassword}" -e java | %PS_TOOLS_HOME%\gawk "{if($1 ~ /java/) system(\" %PS_TOOLS_HOME%\\psexec \\\\%JVM_IP% -d -c %PS_TOOLS_HOME%\\SendSignal \" $2)}"  
 echo .  
 echo ===================================================  
 echo .  
 echo Heap Dump file has been created  
 echo .  
 echo ===================================================  
 echo .   

How to Clean IE Cache with Watir?

we need some sanity test from performance perspective,basically need to get end-end performance result on daily build automatically:

1. end-end response time
2. each page size info
3. clean cache between each test round

so the first item is relative easy to achieve and more straight forward by Watir

for second item, after checking out http://www.newsqa.com/delete-cache-and-cookies-in-ruby/, it works well on Windows XP_SP3+IE6 browser (did not try any other platform)

for the third item, current Iｕｓｅ　"ie.html.length" to get page size, any other better idea? I did not dig this deeper right now.

Or i may consider install a HTTPWatch +Watir to do all the things, haha :)

A bonus get from QA who helps us test website Performance

I always borrow our QA team to help us validate performance on staging env when they do release testing as a bonus, here is an example email I sent to the group:

Hi, All QA friends,
Once you find slowness on your side(for example 60 seconds to login, …), please do so :

1. Can you reproduce it? Or just randomly replicate?
2. Username/password/account
3. Which server are you touching? (you need some sniffer tools to determine this if there is a cluster on back-end)
4. Time stamp you encounter the problem(PST time preferred/Beijing time zone is also OK)
5. What steps you are following? (replicate steps if have)

Of course we need to set up some monitoring/profiling tools behind or run ping -t IP to guarantee Network is OK during testing phase.

Performance test scope requirement collection table

I would like to provide a table which include a relative simple and high level categories of performance factors to PM/Arch/Dev leader/(Func)QA leader/ module owner for collecting comments or requirements in numbers/priority from Func team point of view.

Here is an example(Click to view big):



So it can help you to filter some work easily this way :) and then prioritize other remaining tasks/user stories.