Sunday, February 09, 2014

Lessons learned druing one of my recent projects

1.       Prepare a good Planning and Clear target in advance, and we share the common purpose

2.       Understand the envs/software/settings (Design, Mechanical sympathy, VMware DRS, Gateway Throttling Replication mechanism), doing some Research before doing any testing

3.       A simple but straightforward tests, complex ones make things complex and hard to narrow down the problem

4.       A Clean up the env

5.       A repeatable, accurate and detailed Baseline first.. Do not rush to do any optimization before a repeatable baseline captured

6.       Get Fully real time Monitoring, even the load generators

7.       Make changes One by One, not all in once, do not mix things up, checklist all the stuff we have been made

8.       Logs is not free, turn DEBUG logs off if nobody look at it, if we really need them, make them into INFO

9.       Check the Disk space when you do multiple rounds and write tons of logs.. Response time will be slowed down suddenly

10.   Visibility to the VMXhost who is using VMware to do PE tests and get to know how your hosts are distributed is critical to the tests(resource utilization should be balanced(CPU,IO,Mem) and not touching the ceiling of each host), Disable the DRS feature for the PE cluster, it is not good for getting repeatable data, but we can optimize the hosts based on our observations by monitoring the hosts utilization

11.   Do not trust open source, need to dig into it when you are making use of it intensively!!