Friday, May 21, 2010

"Firewall(?)" blocks "Accept-Encoding" header

recently, I did Gzip testing with loadrunner with "Accept-Encoding: gzip, deflate" request headers, so want to make sure that Gzip rocks as i told to my teammates.

My test scenario is running from remote server which located in US DC, it sent requests to China servers.

However, i noticed the test result is not changed, then i doubt about Gzip impact, keep look at apache access logs by adding \"%{Accept-Encoding}i\" to LogFormat in httpd.conf. There is no "gzip, deflate" header in access log found:
"GET / HTTP/1.1" 200 2788 0 "-" "Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1)" "-"


Then I did another round of test from local desktop, then everything is fine:
"GET / HTTP/1.1" 200 2788 0 "-" "Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1)" "gzip, deflate"


So "Firewall" blocks "Accept-Encoding" header is just my assumption, but not quite sure... At least notice your test result is making sense, otherwise, you are wasting of time, do not let such test result misleading!

Update:
Find a person with similar issue as mine: http://forums.mozillazine.org/viewtopic.php?f=9&t=1483185
There is Outpost Firewall which disable gzip...

Another latest post on "pushing beyond Gzipping" by Yahoo! talking about forcing Gzip when Accept-Encoding request header is mangled or stripped.

Wednesday, May 12, 2010

Simple Performance profiling samples by Btrace

If you want to dig into why certain method runs slow than you expected, then you can do performance profiling by Btrace, Here is some simple example i used before, hope this helps, BTW, I used JVisualVM Btrace plugin, it is really easy of use, but Be caution with bugs :)

Calculate method execution time:

 /* BTrace Script Template */
 package com.sun.btrace.samples;  

 import com.sun.btrace.annotations.*;  
 import static com.sun.btrace.BTraceUtils.*;  
 import com.sun.btrace.aggregation.*;  
 @BTrace  
 public class MethodResponseTime {  
      /* put your code here */  
   @TLS  
   private static long starttime;  
   private static Aggregation average = newAggregation(AggregationFunction.AVERAGE);  
   private static Aggregation globalcount = newAggregation(AggregationFunction.COUNT);  
      @OnMethod(  
      clazz = "com.myapp.service.pipeline.PipelineServlet",  
      method = "execute"  
      )  
      public static void onCall(){  
        println("enter this method!");  
     starttime = timeNanos();  
      }  
   @OnMethod(  
      clazz = "com.myapp.operations.PrePopulateResponse",  
      method = "execute",  
   location = @Location(Kind.RETURN)  
      )  
      public static void onReturn(){  
        println("Method End!");  
     int duration = (int)(timeNanos()- starttime)/1000000;  
     println(duration);  
     addToAggregation(average,duration);  
     addToAggregation(globalcount,duration);  
      }  
   @OnTimer(20000)  
   public static void onEvent(){  
     println("----------------");  
     printAggregation("Average", average);  
     printAggregation("Global Count", globalcount);  
     println("----------------");  
   }  
 }  


------------------------------------------------------------

Execution time between code lines:

 /* BTrace Script Template */  
 package com.sun.btrace.samples;  

 import com.sun.btrace.annotations.*;  
 import static com.sun.btrace.BTraceUtils.*;  
 import com.sun.btrace.aggregation.*;  
 @BTrace  
 public class CodeLines {  
 @TLS private static long startTime;  
 @OnMethod(  
 clazz="com.myapp.impl.CommonResourceChangePollingImpl",  
 location=@Location(value=Kind.LINE, line=97)  
 )  
 public static void onEnter(){  
 //println("enter this method");  
 startTime= timeNanos();  
 }  
 @OnMethod(  
 clazz="com.myapp.impl.CommonResourceChangePollingImpl",  
 location=@Location(value=Kind.LINE, line=100)  
 )  
 public static void onReturn(){  
 //println("method end!");  
 int duration = (int)(timeNanos()-startTime)/1000000;  
 println(duration);  
 addToAggregation(average,duration);  
 addToAggregation(globalCount, duration);  
 }  
 @OnTimer(30000)  
 public static void onEvent() {  
 println("---------------------------------------------");  
 printAggregation("Average", average);  
 printAggregation("Global Count", globalCount);  
 println("---------------------------------------------");  
 }  
 }  

PS: it calculates from code line 97 to 100 , but not including 100 execution time itself

--------------------------------------------------------------
Call dump trace:

 /* BTrace Script Template */  
 package com.sun.btrace.samples;  

 import com.sun.btrace.annotations.*;  
 import static com.sun.btrace.BTraceUtils.*;  
 import com.sun.btrace.aggregation.*;  
 @BTrace  
 public class CallTree {  
 @OnMethod(  
 clazz="com.myapp.impl.CommonResourceChangePollingImpl",  
 method="getResourceChanges",  
 location=@Location(value=Kind.LINE, line=-1)  
 )  
 public static void online(@ProbeClassName String pcn, @ProbeMethodName String pmn, int line) {  
 print(strcat(pcn, "."));  
 print(strcat(pmn, ":"));  
 println(line);  
 }  
 }  


-----------------------------------------------------------
Regular expression match on Clazz:

 /* BTrace Script Template */  
 package com.sun.btrace.samples;  

 import com.sun.btrace.annotations.*;  
 import static com.sun.btrace.BTraceUtils.*;  
 import com.sun.btrace.aggregation.*;  
 @BTrace  
 public class RegexMatch {  
 @OnMethod(  
 clazz="/com\\.mycompany\\.service\\..*/",  
 location=@Location(value=Kind.LINE, line=-1)  
 )  
 public static void online(@ProbeClassName String pcn, @ProbeMethodName String pmn, int line) {  
 print(strcat(pcn, "."));  
 print(strcat(pmn, ":"));  
 println(line);  
 }  
 }  

Monday, May 10, 2010

Create short URL by tinyurl service

Google charts usually create a long url which is really meaningless for us to read. and I want to convert by some external services instead of creating a new one, I tried Shorturl but failed due to network service problem or something.

here is one simple way to generate using http://tinyurl.com/ service, easy of use :)


require 'open-uri'
require 'uri'

retrun_URL = 'http://chart.apis.google.com/chart?chxl=0:|load+google+landing+page|click_suggestion|1:|0|1|2|3|4|5&chxt=x,y&chco=76A4FB&chd=s:94&chtt=Detail+Page+Response+Time+for+GoogleSuggest.html&cht=lc&chs=900x300&chxr=1,5.977,5.401'

encoding_url = URI.escape(retrun_URL, Regexp.new("[^#{URI::PATTERN::UNRESERVED}]"))

tiny_url = open("http://tinyurl.com/api-create.php?url=#{encoding_url}").read

puts tiny_url


Sample OUTPUT:
>>http://tinyurl.com/1c2

Note: If you did not encoding/escape the URL, then the service will call it "bad URI(is not URI?)...URI::InvalidURIError"

Friday, May 07, 2010

Class loading issue by using Jaxb

After Monitoring our JVM by Jconsole/JvisalVM, I found the Perm Gen is not that stable and will perform Full GC after a short interval, meanwhile, the trend of total class loaded number is going up almost linear...

However Jconsole/JvisalVM only give me a rough idea that we got a problem on class loading or we just guess we may "dynamically creating new classes on the fly".




The simplest way to retrieve class loading information is to check the Verbose output check-box on Jconsole, so that you can trace the class loading event on command window where you started your service.


Here is some of loaded class information which caused the trouble we identify, Each new JAXB context would dynamically generate new classes instead of reusing the existing one:

[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_responseOID from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_respondDate from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_respondDate from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_statusRespondDate from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_statusRespondDate from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_statusOID from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_statusOID from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_statusCode from __JVM_DefineClass__]
[Loaded com.whatever.v2_0.RespondedDateRangeSummary$JaxbAccessor
F_statusCode from __JVM_DefineClass__]


Here is one potential solution on this similar problem, you can try that:
http://fleets.wordpress.com/2010/01/08/jaxb-memory-leak-with-metro-too-many-loaded-classes/

And also in HornetQ performance tuning " Avoiding Anti-Patterns", the first item is that "Re-use connections / sessions / consumers / producers", so we should Always re-use them instead of create new one. And also you could add -XX:+TraceClassLoading and -XX:+TraceClassUnloading for your running production to trace those information instead.

Sunday, April 25, 2010

Class Relation diagram in ExperT automation suite

I have posted "ExperT" automation for end-end performance test with Watir and Httpwatch for a while: http://joychester.blogspot.com/2010/02/preview-of-expert-end-to-end.html,
Here I made one simple class diagram for their relationship between these main classes:


And created Github account already, hope to open source the code soon...

Friday, April 23, 2010

Infinite Getting "Cannot open connection" error, high CPU%

I met one exception on our env recently:
2010-04-22 00:20:44,836 ERROR (WorkManager(4)-1767:)[org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.JDBCConnectionException: Could not execute JDBC batch update
......
Caused by: java.sql.BatchUpdateException: I/O Error: Read timed out
at net.sourceforge.jtds.jdbc.JtdsStatement.executeBatch(JtdsStatement.java:966)


And this will cause infinite loop to get a connection, taking over 90% CPU time:
....
org.hibernate.exception.GenericJDBCException: Cannot open connection
....


after looking around the jtds site,it quoted "jTDS 1.2.3 has fixed this issue" from its release notes:

[1843801] infinite loop if DB connection dies during a batch

Seems it is exactly the bug I met!

Did not have chance to evaluate this fix, however, jtds-1.2.3 is a 2-year release after all since jtds-1.2.2.

Upgrade to latest version maybe a great choice if you met same problem.

Wednesday, April 21, 2010

Typical performance data degradation over time

It becomes slower and slower over time...



After restarting the SQL server service, everything becomes normal:


note:
series 1-- before restarting SQL server service
series 2-- after restarting SQL server service

It seems "SQL cache Memory" keeping growth leads to this degradation from my monitoring result, but even not quite sure why...

Gather SQL execution time stat from DMV within performance testing period

after each round of performance testing, i will gather SQL execution time stat from DMV during testing period. it gives me insight on top slowness SQLs which i can focus on first:

SELECT creation_time
,last_execution_time
,total_physical_reads
,total_logical_reads
,total_logical_writes
, execution_count
, total_worker_time
, total_elapsed_time
, total_elapsed_time / execution_count avg_elapsed_time
,SUBSTRING(st.text, (qs.statement_start_offset/2) + 1,
((CASE statement_end_offset
WHEN -1 THEN DATALENGTH(st.text)
ELSE qs.statement_end_offset END
- qs.statement_start_offset)/2) + 1) AS statement_text

FROM sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) st

where creation_time > '2010-04-20 14:30:57.553' and creation_time <'2010-04-20 15:35:57.553'

ORDER BY total_elapsed_time / execution_count DESC;


thanks for this thread, i take the SQL from here and just add creation_time filter: http://www.sql-server-performance.com/articles/per/tsql_statement_performance_p1.aspx

Sunday, April 11, 2010

HornetQ JMS Queue testing using Sender and Consumer

Here is a dummy demo of how to test JMS performance by JMeter sender(Point-to-Point) and Consumer.
I am taking ExampleQueue as an example in HornetQ itself, by using JMeter sender to add load to the ExampleQueue, and write some code to create a dummy consumer listening to ExampleQueue to print out received text:
first you need to add .jar into jmeter\lib which you can get from HornetQ lib:
hornetq-core-client.jar
hornetq-jms-client.jar
hornetq-transports.jar
netty.jar
jnp-client.jar
jboss-jms-api.jar


Sender by Jmeter(click to see big pic):


Consumer by Jmeter(You need to come up with JMSCorrelationID in JMS header otherwise you may get an Error):
ERROR - jmeter.protocol.jms.sampler.FixedQueueExecutor: Correlation id is null. Set the JMSCorrelationID header

Here is my configuration(click to see big pic):


multi-thread Sender by Java code sample:

package msq;
import java.util.Hashtable;
import javax.jms.JMSException;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.NamingException;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;


public class JMS_Q_Sender_threads {

static int TASK_NUM=10;


static class MyTask implements Runnable{

public void run(){

try{
// Step 1. Create an initial context to perform the JNDI lookup.
Hashtable env = new Hashtable();
env.put(Context.PROVIDER_URL, "jnp://localhost:1099");
env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
env.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces");
Context ctx = new InitialContext(env);

// Step 2. Lookup the connection factory
ConnectionFactory cf = (ConnectionFactory)ctx.lookup("/ConnectionFactory");

// Step 3. Lookup the JMS queue
Queue queue = (Queue)ctx.lookup("/queue/ExampleQueue");

// Step 4. Create the JMS objects to connect to the server and manage a session
Connection connection = cf.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

// Step 5. Create a JMS Message Producer to send a message on the queue
MessageProducer producer = session.createProducer(queue);

// Step 6. Create a Text Message and send it using the producer
final int numMessages = 5;
for (int i = 0; i < numMessages; i++){
TextMessage message = session.createTextMessage("Hello, HornetQ!");
producer.send(message);
System.out.println(Thread.currentThread().getId()+": Sent message: " + message.getText());
}
// Finally, we clean up all the JMS resources
connection.close();
} catch (Exception e){}

}


}
public static void main(final String[] args) throws NamingException, JMSException
{
MyTask task=new MyTask();
ExecutorService pool=Executors.newFixedThreadPool(5);
for (int i=0;i < TASK_NUM ; i++){
pool.submit(task);
}
pool.shutdown();
}

}

Consumer by Java code sample:
package msq;
import java.util.Hashtable;
import javax.jms.JMSException;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.MessageConsumer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.NamingException;

public class JMS_Q_Consumer {
public static void main(final String[] args) throws NamingException, JMSException
{
// Step 1. Create an initial context to perform the JNDI lookup.
Hashtable env = new Hashtable();
env.put(Context.PROVIDER_URL, "jnp://localhost:1099");
env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
env.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces");
Context ctx = new InitialContext(env);

// Step 2. Lookup the connection factory
ConnectionFactory cf = (ConnectionFactory)ctx.lookup("/ConnectionFactory");

// Step 3. Lookup the JMS queue
Queue queue = (Queue)ctx.lookup("/queue/ExampleQueue");

// Step 4. Create the JMS objects to connect to the server and manage a session
Connection connection = cf.createConnection();
Session session2 = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
MessageConsumer messageConsumer = session2.createConsumer(queue);

// Step 5. Start the Connection so that the server starts to deliver messages
connection.start();

int j = 0;
// final int num = 5;
// while(j <= num){
final long duration = 60000;

long start = System.currentTimeMillis();

while (System.currentTimeMillis() - start <= duration){
// Step 6. Receive the message
TextMessage messageReceived = (TextMessage)messageConsumer.receive(5000);
if (messageReceived != null) {
j++;
System.out.println(j + "Received message: " + messageReceived.getText());
}
else {
System.out.println("no message detected");
break ;
}

}
long end = System.currentTimeMillis();

double rate = 1000 * (double)j / (end - start);

System.out.println("We consumed " + j + " messages in " + (end - start) + " milliseconds");

System.out.println("Actual consume rate was " + rate + " messages per second");
// Finally, we clean up all the JMS resources
connection.close();
}

}
You can create any type of consumer based on your requirement and get the performance data accordingly.
And also I used Jconsole Mbeans to help monitoring the Queue JMS message at the mean time. So that we can notice if there is anything wrong with the Queue ASAP.

Thursday, April 08, 2010

Translate From Hash to Array in ruby

it is code for putting transaction name and its response time in pair , so that it can be used to show on the Google chart afterwords.

#Hash implementation:

@Detail_CHART = Hash.new
if @Detail_CHART.has_key?(transname)
new_name = transname + i.to_s
while @Detail_CHART.has_key?(new_name)
i += 1;
new_name = transname + i.to_s
end
@Detail_CHART.store(new_name, restime);
else @Detail_CHART.store(transname, restime);
end

However, i found ruby Hash keep its own order instead of original order of input, so i have to change hash to Array implementation.

#Array implementation:

@Detail_action = Array.new
@Detail_time = Array.new
if @Detail_action.include?(transname)
new_name = transname + i.to_s;
while @Detail_CHART.has_key?(new_name)
i += 1;
new_name = transname + i.to_s;
end
@Detail_action.push(new_name);
@Detail_time.push(restime);
else
@Detail_action.push(transname);
@Detail_time.push(restime);
end