Isolation issues on performance test and tuning

This message will be focusing on how we isolate performance issue from testing perspective
Found many errors (from Loadrunner/Jmeter or from back-end server log)

* Try single user see if it works fine then add load slowly till the error happens
* Try different data (e.g. account / user)
* Remove certain scenarios from the complete set and see which scenario is the trouble maker. E.g. in reporting performance test, with / without scenario 6, 7, 8 the error rate is very different

Note for sure we should analyze loadrunner/jmeter/server error log as early as possible. and I always look at Loadrunner/Jmeter errors from the very first occurrence(look at the error message time stamp), usually the following erros are caused related to the first one.
Certain scenario / step is slow in formal performance test

* Tune the scenario / step using single user
E.g. When tuning website publish module, we first focus on single user to go through every method/SQL and optimized the code (especially remove unnecessary SQL). After this step we begin to tune the performance under multiple user.
Note tuning under single user is helpful at the beginning phase but analyze multi user result (profiling result, etc) is also critical for performance tuning.
* Disable other scenarios / steps and focus on few cases with multiple user
E.g. we have a combined case of login, sign up, search, vendor detail. When executing the full set of test we find vendor detail is relatively slow. So we decided to run the vendor detail only – all the threads running login once then loop forever to view vendor detail. Thus, the issue inside view vendor detail got highlighted and easily find the bottleneck and fixed.
Note often the performance issue is caused by contention between different cases but before that we can try to tune single step first with multiple users.

if you have memory leak or connections being killed situation, then you also can do the isolation work to narrow down the issue and do the certian step tests with dev or DBA, then you can seize the killer quickly.
when doing tests you can ignore the think time and add more load to reproduce this kind of issues, do not be shy, be rude :)