Monday, November 28, 2011

Dealing with "Not enough storage is available to process this command" exception by PsExec

When I am using Btrace to profiling JVM started with JavaServiceWrapper, I got such an annoyed exception:
CMD> C:\btrace\bin>btrace 3272 Test.java
Not enough storage is available to process this command
1> Login as a admin user anyway (remotely or locally)
2> Download PsTools which includes PsExec.exe , and "PsExec is a light-weight telnet-replacement that lets you execute processes on other systems, complete with full interactivity for console applications, without having to manually install client software", or you can download directly from : http://technet.microsoft.com/en-us/sysinternals/bb897553
3> Get your target PID(java.exe) with processExplorer.exe, download from: http://technet.microsoft.com/en-us/sysinternals/bb896653
4> Then Run your command line again using PsExec help:

CMD> D:\ProcessExplorer\PsTools>PsExec.exe -s "C:\btrace\bin\btrace.bat" 3272 C:\Test.java
 Not only for Btrace, the other external tools can also be run like this way to avoid such a wired Error...

Monday, November 21, 2011

如何计算消息队列中的消息处理时间


求某一时刻进入队列的消息,计算其出队列时间 EL? 

定义: 
S 入队列速率 (根据load的情况,S随不同时间段而变化)
P 出队列吞吐量, 通常在给定的系统配置条件下可以认为是常量
  TKK时刻
 TN是指对应K时刻进入消息队列的消息出队列所到达的时刻
  QD 为初始时刻Queue中剩余的Message的数量

情况1> 假设在一定时间段内,S为定值
S 总是<=P时, 在一段时间后任意时刻均无剩余message堆积, 这里暂时不考虑这类问题
S > P 时, Queue depth 会不断增长
所以,TN = (S * TK)/P + QD/P 
EL
TNTK
EL
  (S * TK)/P + QD/P TK (S/P - 1)*Tk + QD/P
其中,SP均为固定值,并且可以被measure
(一般可以通过SQL语句来统计S和P的值, 例如-- select count(*) from dbo.myqueue where ms.timestamp == ctime

情况2> 假设S为不断变化的正玄函数(斜率即为某时刻的进队速率S‘),则公式可变换为:
 EL = (∫ sin(cx)dx) /P + QD/P -TK , 其中
积分的范围是0-TK

 
因此,S需要不断的采样,利用曲线拟合方法(Curve fitting),确定它符合什么样的函数,但不管什么情况,都可以套用类似的模型解决此类问题

Remove the value of the param in URL

It should be formatted your URL when doing aggregation for further analysis, here is Ruby code to be copy with the URL which includes params within it:
 string1 = "http://www.google.com.hk/search?sclient=psy-ab&hl=en&newwindow=1&safe=strict&biw=1440&bih=693&noj=1&source=hp&q=wicket+1.5+URL&oq=wicket+1.5+URL&aq=f&aqi=g2&aql=&gs_sm=e&gs_upl=3231l4087l0l4226l4l4l0l0l0l2l302l832l2-2.1l3l0";  
 if (string1.include? "?")  
  string1.insert(-1, "&")  
  string1.gsub!(/=(.*?)&/, "&")  
  puts string1.chomp!("&")  
 end  
The output result:
 http://www.google.com.hk/search?sclient&hl&newwindow&safe&biw&bih&noj&source&q&oq&aq&aqi&aql&gs_sm&gs_upl  

Tuesday, November 15, 2011

Essential Counters for Performance Monitoring

The goals of performance monitoring i always think about are:

- Health indicator: it will show you if the current status of your application farm is in good shape or not; or let you predict the trend;
- Find out the bottleneck of your system from a high level, it is really helpful for Engineers to get start to get issue fixed;
- Help to do trouble shooting: the counters will drive us to the right direction from different angles to find the root cause instead of "smart guess";
- Operation management: Capacity Planning and Risk management, try to resolve any performance or stability issues before it comes!





Counters For OSExplaination
Server Uptimeelapse time since server recent start up
Processor--Total CPU%the percentage of elapsed time that the processor spends to execute a non-Idle thread.
Processor--% User Timethe percentage of elapsed time the processor spends in the user mode.
System--Processor Queue Lengththe number of threads in the processor queue. There is a single queue for processor time even on computers with multiple processors. Therefore, if a computer has multiple processors, you need to divide this value by the number of processors servicing the workload. A sustained processor queue of less than 10 threads per processor is normally acceptable, dependent of the workload.
Memory--% MEM in usethe ratio of Memory\\Committed Bytes to the Memory\\Commit Limit.
Memory--pages/secthe rate at which pages are read from or written to disk to resolve hard page faults.
Disk Space--DISK C:the percentage of Used space on Disk C
Disk Space--DISK D:the percentage of Used space on Disk D
Disk Space--DISK E:the percentage of Used space on Disk E
DISK IO --Avg. Disk Queue Lengththe average number of both read and write requests that were queued for the selected disk during the sample interval.
DISK IO --% Disk Timethe percentage of elapsed time that the selected disk drive was busy servicing read or write requests.
DISK IO --Avg. Disk sec/Readthe average time, in seconds, of a read of data from the disk.
DISK IO --Avg. Disk sec/Writethe average time, in seconds, of a write of data to the disk.
DISK IO --Disk Reads/secthe rate of read operations on the disk.
DISK IO --Disk Writes/secthe rate of write operations on the disk.
NetWork IO--Packets Sent/secthe rate at which packets are sent on the network interface.
NetWork IO--Packets received/secthe rate at which packets are received on the network interface.
TCP--TCP Connections ESTthe number of TCP connections for which the current state is either ESTABLISHED or CLOSE-WAIT
Total Process cntTotal number of processes on the server


Counters For Specific Process instanceExplaination
Apache Http Server -- Apache Process
Process Uptimeelapse time since apache process recent start up
Process CPU%the percentage of elapsed time that the processor spends on Apache Process.
Process Memory in useMemory usage by apache process
Busy workers cntthe number of threads which are in use for requests
Idle workers cntthe number of threads which are not receiving any request
requests/secthroughput of apache Http server
KB/secthroughput of apache Http server
Application Server -- Java Process
Process Uptimeelapse time since java process recent start up
Process CPU%the percentage of elapsed time that the processor spends on java Process.
Private Bytesthe current size, in bytes, of memory that this process has allocated that cannot be shared with other processes.
Working SetWorking Set is the current size, in bytes, of the Working Set of this process. The Working Set is the set of memory pages touched recently by the threads in the process
Used Heap SizeJVM Heap size is in use currently
Live Threads cntThe number of threads currently active in this process
Accumulate GCTimethe total time taken by GC activities
Async Server--JMS MQ Process
Process Uptimeelapse time since JMS process recent start up
Process CPU%the percentage of elapsed time that the processor spends on JMS Process.
Private Bytesthe current size, in bytes, of memory that this process has allocated that cannot be shared with other processes.
Working SetWorking Set is the current size, in bytes, of the Working Set of this process. The Working Set is the set of memory pages touched recently by the threads in the process
Message Queue Depththe current number of messages that are waiting on the queue
DLQ cntthe current number of messages which in Dead Letter Queue
Live Threads cntThe number of threads currently active in this process
DB Connections cntthe current number of open database connections used by this process
DB Server -- SQL Server Process
Process Uptimeelapse time since SQL Server process recent start up
Process CPU%the percentage of elapsed time that the processor spends on SQL Server Process.
Process Memory in useMemory usage by SQL Server process
Free Space in Temp DBTracks free space in tempdb in kilobytes
User Connectionsthe current number of connections (and users) are using the server
Buffer Cache Hit Ratioindicates how often SQL Server goes to the buffer, not the hard disk, to get data. Had better larger than 90%
Full Scans/Secthe number of unrestricted full scans during unit time. These can either be base table or full index scans.
Transactions/SecThe number of transactions started for the database during unit time
Average Wait Timethe average amount of wait time (milliseconds) for each lock request that resulted in a wait

More counters you add and higher granularity you made will bring more overhead to your system, you should make trade offs on selecting counters and sample interval.There is no silver bullet after all...

Wednesday, October 26, 2011

Add time stamp to GC log to ease your pain

When analyzing Gc log like these:
170609.479: [GC 170609.479: [ParNew: 1699544K->17689K(1887488K), 0.0537160 secs] 1728778K->47004K(5033216K) icms_dc=0 , 0.0538078 secs] [Times: user=0.08 sys=0.00, real=0.06 secs] 
172980.190: [GC 172980.190: [ParNew: 1695513K->4310K(1887488K), 0.0147373 secs] 1724828K->33657K(5033216K) icms_dc=0 , 0.0148245 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
174228.363: [Full GC 174228.363: [CMS: 29346K->31421K(3145728K), 0.5658221 secs] 955586K->31421K(5033216K), [CMS Perm : 58212K->57316K(58536K)] icms_dc=0 , 0.5660007 secs] [Times: user=0.51 sys=0.00, real=0.56 secs] 
174228.930: [GC [1 CMS-initial-mark: 31421K(3145728K)] 31421K(5033216K), 0.0013385 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 174228.931: [CMS-concurrent-mark-start] 
175694.396: [GC 175694.396: [ParNew: 1677824K->25622K(1887488K), 0.0726440 secs] 1709245K->57044K(5033216K) icms_dc=0 , 0.0727338 secs] [Times: user=0.06 sys=0.00, real=0.06 secs] 
177309.350: [GC 177309.350: [ParNew: 1703446K->23200K(1887488K), 0.0730725 secs] 1734868K->54622K(5033216K) icms_dc=0 , 0.0731623 secs] [Times: user=0.06 sys=0.00, real=0.06 secs] 179729.274: [GC 179729.274: [ParNew: 1701024K->4428K(1887488K), 0.0138419 secs] 1732446K->35850K(5033216K) icms_dc=0 , 0.0139156 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
I am always confused and painful to get what the exactly time stamp of each GC activity is(BTW, currently i am using GCViewer to do the analysis), so that i can not easily to correlate the GC log with Access log when doing further analyzing and trouble shooting the memory problems... So i am writing a ruby program to calculate the time stamp for your further investigation, called gcanalyzer.rb:

Pre-Condition: you should have your Ruby1.9.2 installed on your computer, Ruby 1.8.* may not work properly.

 require 'csv'  
 def gcanalyzer(filename)  
 i = File.new(filename);  
 arr1 = [0];  
 arr2 = [];  
 @@maxpasuetime = 0.0000;  
 @@fullgc_count = 0;  
 m = 0;  
 endstamp = i.mtime  
 if (!endstamp.isdst) #Summer Time  
  zone = 15  
 else  
  zone = 16  
 end  
 a = i.readlines()  
 endsec = a[a.count-1].to_i  
 outputfilename = 'GC_log' + endstamp.strftime("_%m-%d-%H%M%S") + '.csv';  
 j = File.new(filename);  
 open(outputfilename, "a"){|f|  
  j.each_line { |line|  
   if (s = line.split('->')[2])!= nil  
    offset = endsec - line.split(':')[0].to_i  
    ctime = endstamp - offset -zone*3600  
    timestamp = ctime.strftime("%m/%d-%H:%M:%S")  
    usedheap = s.split('K(')[0]  
    f << timestamp + "," + usedheap + "\n"  
    #prepare pause time  
    # This is for +UseParallelGC  
    if line =~/PSYoungGen/  
     if line =~ /Full/  
       @@fullgc_count = @@fullgc_count + 1  
       arr2.push(timestamp);  
     end  
     t1 = line.split(',')[1]  
    elsif # This is for CMS case  
     if line =~ /Full/  
       @@fullgc_count = @@fullgc_count + 1  
       arr2.push(timestamp);  
       t1 = line.split(',')[3]  
     elsif  
       t1 = line.split(',')[2]  
     end  
    end  
    pausetime = t1.split('secs')[0].strip.to_f  
    arr1[0] = arr1[0] + pausetime  
    #prepare maxpausetime  
    if pausetime > @@maxpasuetime  
      @@maxpasuetime = pausetime;  
    end  
   end  
  }  
 }  
 elapsedday = endsec/(24*3600);  
 ed = (endsec - elapsedday*24*3600);  
 elapsedhour = ed/3600;  
 elapsedmin = (ed - elapsedhour*3600)/60;  
 printf("Total Elapse Time: %d (%dday-%dhour-%dmin)\n", endsec, elapsedday, elapsedhour,elapsedmin) ;  
 printf("Total GC Puase Time: %.3f \n" , arr1[0]) ;  
 printf("GC Throughput: %.3f \n" ,(1-arr1[0]/endsec)*100) ;  
 printf("Max Pause Time: %.3f \n",@@maxpasuetime) ;  
 printf("Avg Puase Time: %.3f \n", (arr1[0]/(a.count-1)));  
 printf("Total GC Occurency: %d \n", (a.count-1));  
 printf("GC Frenquency: %.3f \n", endsec/(a.count-1));  
 printf("Full GC Occurency: %d \n", @@fullgc_count);  
 printf("Full GC Time stamp: #{arr2}");  
 # consider to make a chart based on new file in the future  
 end  

How to use it:
>ruby gcanalyzer.rb "D:\\app01_garbage_collection.log"
There will be a new file created with accurate time stamp of each GC, and also Output Summary info to the console, something like this:
Total Elapse Time: 2341621 (27day-2hour-27min)
Total GC Puase Time: 636.061
GC Throughput(%): 99.973
Max Pause Time: 0.566
Avg Puase Time: 0.162
Total GC Occurency: 3936
GC Frenquency(sec): 594.000
Full GC Occurency: 1
Full GC Time stamp: ["09/25-05:51:15"]

Thursday, September 01, 2011

Find Optimal Concurrency for your Application

Although I found this article is more "academic", it is really helpful to understand how computer works when there are high concurrency coming in. So not always good idea for setting a very high number of workers/threads to handle large volume of concurrent requests , by given the certain hardware environment. Want to know why? please take a look at the post which is from book :

Finding the Optimal Concurrency [From High Performance MySQL]

Every web server has an optimal concurrency—that is, an optimal number of concurrent
connections that will result in requests being processed as quickly as possible,
without overloading your systems. A little trial and error can be required to find this
“magic number,” but it’s worth the effort.

It’s common for a high-traffic web site to handle thousands of connections to the
web server at the same time. However, only a few of these connections need to be
actively processing requests. The others may be reading requests, handling file
uploads, spoon-feeding content, or simply awaiting further requests from the client.

As concurrency increases, there’s a point at which the server reaches its peak
throughput. After that, the throughput levels off and often starts to decrease. More
importantly, the response time (latency) starts to increase.

To see why, consider what happens when you have a single CPU and the server
receives 100 requests simultaneously. One second of CPU time is required to process
each request. Assuming a perfect operating system scheduler with no overhead,
and no context switching overhead, the requests will need a total of 100 CPU seconds
to complete.

What’s the best way to serve the requests? You can queue them one after another, or
you can run them in parallel and switch between them, giving each request equal time
before switching to the next. In both cases, the throughput is one request per second.
However, the average latency is 50 seconds if they’re queued (concurrency = 1), and
100 seconds if they’re run in parallel (concurrency = 100). In practice, the average
latency would be even higher for parallel execution, because of the switching cost.
[Ideally]For a CPU-bound workload, the optimal concurrency is equal to the number of
CPUs (or CPU cores).
[Normally]However, processes are not always runnable, because they
make blocking calls such as I/O, database queries, and network requests. Therefore,
the optimal concurrency is usually higher than the number of CPUs. [That's why we are able to add more threads to handle concurrent requests]

You can estimate the optimal concurrency, but it requires accurate profiling.
[Conclusion:]It’s usually easier to experiment with different concurrency values and see what gives the
peak throughput without degrading response time.

Monday, August 15, 2011

Performance books I have or I wish to have

Overall Performance Engineering Methodology:
* Performance Analysis for Java Web Sites By Stacy Joines
* Software Performance and Scalability: A Quantitative Approach By Henry Liu
* Improve .Net Application Performance and Scalability By Microsoft
* Building Scalable Web Sites By Cal Henderson

Performance Testing Process & Practice:
* Performance Testing Guidance for Web Application By Microsoft
* The Art of Application Performance Testing By Ian Molyneaux

DB Performance tuning:
* Inside SQL Server 2005 Query Tuning and Optimization By Kalen Delaney
* High Performance MySQL By Baron Shwartz

Programming Language Performance Tuning:
* Effective Java By Joshua Bloch
* Java Concurrency in Practice By Brian Goetz
* The Art of Concurrency by Clay Breshears
* Java Performance Tuning by Jack Shirazi

Front-End Performance Optimization Practice:
* High Perforamnce WebSites By Steve Souders
* Even Faster WebSites By Steve Souders
* High Performance Javascript By Zakas

Web Operations and Capacity Planning:
* Web Operations By John Allspaw
* The Art of Capacity Planning By John Allspaw

Tuesday, August 09, 2011

C3p0 infinity wait for "availble" connections by default

If you have connection leaks on C3p0 which means the Connections are being checked-out that don't make it back into the pool. You will be in trouble...

By default, C3p0 will wait for the connection forever if there is no one available in the pool currently.

When testing against the application, i found some threads are wait for the connections for a long time, and after monitoring the JMX console in VisualVM, all the connections are taken, here is the thread dump i took:
 "1658227@qtp-25018827-56" - Thread t@321  
   java.lang.Thread.State: WAITING  
      at java.lang.Object.wait(Native Method)  
      - waiting on <103344b> (a com.mchange.v2.resourcepool.BasicResourcePool)  
      at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)  
      at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)  
      at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)  
      at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)  
      at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)  
      at org.apache.ibatis.session.defaults.DefaultSqlSessionFactory.openSessionFromDataSource(DefaultSqlSessionFactory.java:72)  
      at org.apache.ibatis.session.defaults.DefaultSqlSessionFactory.openSession(DefaultSqlSessionFactory.java:32)  

I am not discussing why the code makes connection leaks here in this post.(Actually, we did not invoke session.close from our code finally ), i just want to make sure if you meet this kind of similar pattern of problem, then you can temperately change some C3p0 configuration before you fix your bad code...

we are using C3p0 0.9.1.2 version now, here is the source code which have the infinity loop to wait for the connection, FYI
 1297        while ((avail = unused.size()) == 0)   
  1298        {  
  1299          // the if case below can only occur when 1) a user attempts a  
  1300          // checkout which would provoke an acquire; 2) this  
  1301          // increments the pending acquires, so we go to the  
  1302          // wait below without provoking postAcquireMore(); 3)  
  1303          // the resources are acquired; 4) external management  
  1304          // of the pool (via for instance unpoolResource()   
  1305          // depletes the newly acquired resources before we  
  1306          // regain this' monitor; 5) we fall into wait() with  
  1307          // no acquires being scheduled, and perhaps a managed.size()  
  1308          // of zero, leading to deadlock. This could only occur in  
  1309          // fairly pathological situations where the pool is being  
  1310          // externally forced to a very low (even zero) size, but   
  1311          // since I've seen it, I've fixed it.  
  1312          if (pending_acquires == 0 && managed.size() < max)  
  1313            _recheckResizePool();  
  1314    
  1315          this.wait(timeout);  
  1316          if (timeout > 0 && System.currentTimeMillis() - start > timeout)  
  1317            throw new TimeoutException("A client timed out while waiting to acquire a resource from " + this + " -- timeout at awaitAvailable()");  
  1318          if (force_kill_acquires)  
  1319            throw new CannotAcquireResourceException("A ResourcePool could not acquire a resource from its primary factory or source.");  
  1320          ensureNotBroken();  
  1321        }  

timeoutvalue is set to 0 by default in C3p0, which means wait indefinitely
checkoutTimeout
Default: 0
The number of milliseconds a client calling getConnection() will wait for a Connection to be checked-in or acquired when the pool is exhausted. Zero means wait indefinitely. Setting any positive value will cause the getConnection() call to time-out and break with an SQLException after the specified number of milliseconds.

And also if you have connection leaks with your code, and you hardly find it in short time, then you should pay attention to this parameter:
unreturnedConnectionTimeout
Default: 0
Seconds. If set, if an application checks out but then fails to check-in [i.e. close()] a Connection within the specified period of time, the pool will unceremoniously destroy() the Connection. This permits applications with occasional Connection leaks to survive, rather than eventually exhausting the Connection pool. And that's a shame. Zero means no timeout, applications are expected to close() their own Connections. Obviously, if a non-zero value is set, it should be to a value longer than any Connection should reasonably be checked-out. Otherwise, the pool will occasionally kill Connections in active use, which is bad. This is basically a bad idea, but it's a commonly requested feature. Fix your $%!@% applications so they don't leak Connections! Use this temporarily in combination with debugUnreturnedConnectionStackTraces to figure out where Connections are being checked-out that don't make it back into the pool!

So you can try to set these two parameters to the same time as your goal keeper, say 30 seconds. So that you can prevent some extremely leak cases, but dev need to find the root cause for sure :)
checkoutTimeout= 30000
unreturnedConnectionTimeout=30

also adjust your maxPoolSize to 30 or more instead of 15 by default under a heavy load situation.

Friday, July 08, 2011

Weekly Health check for SQL Server 2005 using DMV

Every Week, you can set up a benchmark by running following DMV, it can help to provide high level view to show whether your DB is healthy or not currently without any monitoring tools.

While, Some limitations you may have to pay attention to when you are using following DMV Queries:
(Thanks for a good reference provided by Vance: http://www.mssqltips.com/tip.asp?tip=1843)

1. Limitation with DMV queries: Keep this in mind when you are using the DMVs for query usage and performance stats. If you are using inline T-SQL and sp_executesql you may not be capturing all of the data that you need.
—Suggestion : think about using stored procedures for all data related operations instead of using inline T-SQL or sp_executesql in your application code.
2. Limitation with dbid column: there is a problem that it is limiting the result data to queries with a database id. The reason for this is that the dbid column is NULL for ad hoc and prepared SQL statements, So you can comment out the where condition which having dbid in (...);
—Suggestion : you may just comment out the dbid constrain or using “dbid = null” instead of assign a dbid in “where clause”


-- DMV FOR CHECKING CPU USAGE:
 
 SELECT TOP 50   
      DB_Name(dbid) AS [DB_Name],  
      total_worker_time/execution_count AS [Avg_CPU_Time],  
      total_elapsed_time/execution_count AS [Avg_Duration],  
      total_elapsed_time AS [Total_Duration],  
      total_worker_time AS [Total_CPU_Time],  
      execution_count,  
   SUBSTRING(st.text, (qs.statement_start_offset/2)+1,   
     ((CASE qs.statement_end_offset  
      WHEN -1 THEN DATALENGTH(st.text)  
      ELSE qs.statement_end_offset  
      END - qs.statement_start_offset)/2) + 1) AS statement_text  
 FROM sys.dm_exec_query_stats AS qs  
 CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st  
 WHERE dbid in (  
           SELECT DB_ID('yourtablename') AS [Database ID]  
      )  
 ORDER BY Avg_CPU_Time DESC;  

-- DMV FOR CHECKING I/O USAGE
 SELECT TOP 50  
      DB_Name(dbid) AS [DB_Name],  
      Execution_Count,  
      (total_logical_reads/Cast(execution_count as Decimal(38,16))) as avg_logical_reads,  
      (total_logical_writes/Cast(execution_count as Decimal(38,16))) as avg_logical_writes,  
      (total_physical_reads/Cast(execution_count as Decimal(38,16))) as avg_physical_reads,  
      max_logical_reads,  
      max_logical_writes,  
      max_physical_reads,  
   SUBSTRING(st.text, (qs.statement_start_offset/2)+1,   
     ((CASE qs.statement_end_offset  
      WHEN -1 THEN DATALENGTH(st.text)  
      ELSE qs.statement_end_offset  
      END - qs.statement_start_offset)/2) + 1) AS statement_text  
 FROM sys.dm_exec_query_stats AS qs  
 CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st  
 WHERE dbid in (  
           SELECT DB_ID('yourtablename') AS [Database ID]  
      )  
 ORDER BY avg_logical_reads DESC;  

-- DMV FOR CHECKING INDEX USAGE
 SELECT     top 50   
           idx.name as Index_name  
           ,obj.name   
           ,dmv.object_id  
           ,sampledatetime=Getdate()  
           ,dmv.index_id  
           ,user_seeks  
           ,user_scans  
           ,user_lookups   
 FROM sys.dm_db_index_usage_stats dmv  
 INNER JOIN sys.indexes idx on dmv.object_id = idx.object_id and dmv.index_id = idx.index_id  
 Cross Apply sys.objects obj  
 WHERE dmv.object_id = obj.object_id and database_id in (  
 SELECT DB_ID('yourtablename') AS [Database ID]  
 )  
 ORDER BY user_scans desc  

-- DMV FOR CHECKING OBJECT BLOCKING/WAITING
 SELECT TOP 50  
      DB_NAME(qt.dbid),  
      [Average Time Blocked] = (total_elapsed_time - total_worker_time) / qs.execution_count,  
      [Total Time Blocked] = total_elapsed_time - total_worker_time,  
      [Execution count] = qs.execution_count,  
      SUBSTRING(qt.text, (qs.statement_start_offset/2)+1,   
     ((CASE qs.statement_end_offset  
      WHEN -1 THEN DATALENGTH(qt.text)  
      ELSE qs.statement_end_offset  
      END - qs.statement_start_offset)/2) + 1) AS statement_text,
      [Parent Query] = qt.text
 FROM sys.dm_exec_query_stats qs  
 CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt  
 WHERE DB_NAME(qt.dbid) = 'yourtablename'  
 ORDER BY [Average Time Blocked] DESC;  

-- DMV FOR CHECKING TEMPDB USAGE
 SELECT getdate(),   
      SUM(user_object_reserved_page_count) * 8 as user_objects_kb,  
      SUM(internal_object_reserved_page_count) * 8 as internal_objects_kb,  
      SUM(version_store_reserved_page_count) * 8 as version_store_kb,  
      SUM(unallocated_extent_page_count) * 8 as freespace_kb  
 FROM sys.dm_db_file_Space_Usage  
 where database_id = 2  

Thursday, June 30, 2011

Performance Toolkit in My Pocket

Please Note, this is a Draft Version, the list will be updated on the fly.

Kindly Warning:
Don’t be a slave of tools, but you can not leave without tools!
Be a master of your job with tools :)

Performance Toolkit in My Pocket:

1> Perf Testing Tools:
- Jmeter (Load generate tool for different Protocol)
- Loadrunner (Load generate tool for different Protocol)
- SoapUI (WebService load Testing preferred, or help to create mock services)
- Traffic Shaper XP (Network Bandwitch limiter)
- Badboy (HTTPS recording supported for .jmx scripts)
- JMeter plugin : http://code.google.com/p/jmeter-plugins/
- WebDriver Automation Framework for End-End Performance measurement
- ^Unit Performane testing tool need to be filled in...$ (Method level performance testing)

2> Perf Monitoring Tools:
- JConsole
- JVisualVM
- Task manager/PerfMon
- Process Explorer
- Hyperic HQ
- NetXMS
- Netstat
- typeperf Command line (with Ruby Programming)

3> Perf Profiling Tools:
- Jprofiler
- Btrace
- Jmap
- SQL Profiler
- Perf4j
- Guice AOP Profiling methods for Automation test
- HttpWatch
- Firebug
- Chrome Developer Tools
- Fiddler
- Charles
- Wireshark
- DBCC Command

4> Perf Analysis and Tuning Tools:
- Dynatrace Ajax
- MemoryAnalyzer
- TDA
- DB tuning adviser
- Yslow
- Page Speed
- Image Opertimazer : http://www.imageoptimizer.net/Pages/Home.aspx
- jpegmini - Sprite Me :http://spriteme.org/
- Minify JS :http://www.minifyjs.com/
- WebPageTest : http://www.webpagetest.org/

5> MISC:
- Text Editor/IDE you perferred: Netbeans with Ruby for me
- Windows Grep
- Regular Expression
- T-SQL
- Ruby/Python/Perl/Shell/Awk : http://www.ibm.com/developerworks/cn/education/aix/au-gawk/index.html
- STAF/STAX :
- Excel
- LogBack/Log4j
- JSLint (looks for problems in JavaScript programs)
- User Agent analysis: http://www.useragentstring.com/index.php