When you met "java.lang.OutOfMemoryError: unable to create new native thread..."

A good post to read: https://blog.fastthread.io/2016/07/06/troubleshoot-outofmemoryerror-unable-to-create-new-native-thread/

 

Another One: http://www.mastertheboss.com/jboss-server/jboss-monitoring/how-to-solve-javalangoutofmemoryerror-unable-to-create-new-native-thread

 

Good posts to share on Detecting Connection Leak and How to test connection leak

In case you have the following exceptions:

"org.apache.http.conn.ConnectionPoolTimeoutException: 
Timeout waiting for connection from pool at 
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:412) at 
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298) at 
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238) at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422) at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at 
...... " 

Here are  some Good posts on Detecting HTTP Connection Leak and How to test connection leak
Detect HTTP Connection leak Post: http://phillbarber.blogspot.com/2014/02/lessons-learned-from-connection-leak-in.html
How to test Connection leak: http://phillbarber.blogspot.co.uk/2015/02/how-to-test-for-connection-leaks.html

How to find High CPU% Threads in Jboss through JMX Console

1. Open your JBoss JMX Console page
2. Find the Path : [jboss.system]: [ServerInfo] : [listThreadCpuUtilization]
3. Then you can Dump your threads by listThreadDump() operation to find out what the threads are doing

NMT and Java Native memory leak

Java application process used memory usually include JVM Heap, non-heap(PermGen/Metaspace) and Native code which including JVM internals and native OS libs. As we noticed our Physical memory is out of memory after application running a while, however, the Heap usage is fine and normal, using TOP cmd, we found it has eaten almost all the physical memoy and even much bigger than -Xmx heap size we assign to the heap , the first thought come into my mind is maybe Native Memory come into trouble...
But how to make a conclusion to figure point to Native memory?
using -XX:NativeMemoryTracking=summary to help (after JDK7_40?)
after you add above option into your JVM startup config file, first to make sure you have a root permission or switch to root user to run following cmd, for example:

sudo -u {UID} /opt/java/bin/jcmd {PID} VM.native_memory baseline

after the testing running for a while, run another cmd to show your difference comparing with baseline:

sudo -u {UID} /opt/java/bin/jcmd {PID} VM.native_memory summary.diff

PS. if you do not using sudo -u {UID}, you may get exceptions like following:

java.io.IOException: Operation not permitted

or

com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

This could give you some high level idea if you have the Native memory leak or not, but which Object brings you trouble ,  Since NMT doesn't track memory allocations by non-JVM code,  you can use jemalloc / pmap to detect memory leaks in native code, few good posts for your reference : http://jenshadlich.blogspot.com/2016/08/find-native-memory-leaks-in-java.html
or http://lysu.github.io/blog/2015/02/02/how-to-deal-with-non-heap-or-native-memory-leak/

Questionnaire Template for API Performance Review Process

• API Owner: Product Owner, Dev Lead and QA Lead
• Release Target Date
• Business Impact(GMS or Save Cost), Impacted flows and User types
• How much traffic expected During Peak hour(TPS/TPM/TPH)
• API Name/EndPoint & Method/Sample Request & Response?
• API priority based on its traffic and importance
• High level Design and Workflow diagram for the API or API dependencies
• How many Roles get deployed, and their .war names/versions
• Existing or New API, If an existing API, any Monitoring Dashboard and Baseline captured against PROD?
• JDBC queries
• Third parties dependencies and End points
• Firewall Ruleset/Gateway Synapse changes
• Project Wiki page Link
• Dev API testing plan/scripts/results for reference

Make a Performance budgeting chart for measuring page performance

You should have a performance goal before you measure your page performance, i would suggest you had better have a performance budget for each component, then fight against the one who has overdrawn.
Using navigation timing API , User Timing API and Resource Timing API to do the measurement, in both synthetic and RUM way!!

Besides measure Duration of each Component as a Main KPI, The Content Downloaded Size, # of Requests for each Component need to be considered meanwhile. 

Another point of view to explain the key page performance metrics, please remember every page design differently, you can not set up a rule to fit all, considering from end user perspective is always a great start:

Get Page performance result by Combining User and Resource timing API

 //calculate page loading time by resource timing API  
 var perfEntries = performance.getEntries();  
 var end_probe = perfEntries.filter(function(item) {  
   //page load finish request indicator  
   if (item.name.indexOf('/req_url_end') > -1) {  
     console.log(item.name);  
     return true;  
   }  
 });  
 if(end_probe.length > 0) {  
   var end_time = end_probe[0].responseEnd;  
   var start_probe = perfEntries.filter(function(item) {  
     //page load start indicator  
     if (item.name.indexOf('/req_url_start') > -1) {  
       console.log(item.name);  
       return true;  
     }  
   });  
   var start_time = start_probe[0].startTime;  
   var duration = end_time - start_time;  
   console.log(duration);  
 } else {  
   console.log("page loading end indicator is not found, please double check all perf Entries");  
 }  

There is a better way to combine User Timing API and Resource Timing API to get accurate page performance, i am using Nightmare APIs as a sample to do the automated page tests, which can help our continuous page performance test process on daily basis:

 var url = "http://www.yourhost.com/abc/def/";  
 var page_complete_idy = 'key_request_name'; //page indicator by resource name  
 var tag = 'ReviewPage';  
 var env_name = 'prod'; 

 const RENDER_TIME_MS = 2000;  

var Nightmare = require('nightmare'),
  nightmare = Nightmare({show: true, switches: {
    'ignore-certificate-errors': true
  }});

nightmare
  .goto(url)
  .wait(function(idy){
    var perfEntries = window.performance.getEntries();
    if (perfEntries.length > 20) {
      return perfEntries.some( function (item) {
        if (item.name.includes(idy)) {
          return true;
        }
      });
    } else {
      return false;
    }
  }, page_complete_idy)
  .evaluate(function(idy){
    var perfEntries = window.performance.getEntries();
    var perf_obj = perfEntries.find(function (item) {
      return item.name.includes(idy)
    });
    if (perf_obj) {
      return perf_obj.responseEnd.toFixed(1);
    } else {
      return 'undefined'
    }
  }, page_complete_idy)
  .end()
  .then( function (duration) {
    console.log(tag + ":" + duration);
  })

Replaying Your [access] log by JMeter

Replaying [apache access] log by JMeter to mimic the real user load..
This is inspired by blazemeter post: Learn How to Replay Your Production Traffic With JMeter, but I made my own optimization and enhancement, check it out if you are interested in it:

https://github.com/joychester/Doraemon

PS: the first row in the formatted log file will be ignored, due to fetch the log started timestamp.

Upgrade Ruby version on your Mac OS X

Finally I got a 15'' Mac Pro as my working laptop, it is kind of Chinese new year gift :)
First, the OS embedded a Ruby 2.0 version, which is kind of out of date, so upgrade to the 2.2.0 is my first stuff to do with my Mac.
The main steps i follow is This
However, after doing that, it is not working properly on mine, so i will simplify my steps as following:

1. Install Homebrew: ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
2. Install rbenv to help you manage ruby versions: brew install rbenv ruby-build
3. Adding extra Path to ~/.bash_profile : export PATH=/Users/cchi/.rbenv/shims:$PATH
4. Open Terminal-> preference -> Shell -> Startup : source ~/.bash_profile
5. Check all existing package you can install : rbenv list -l
6. Install Ruby 2.2.0: rbenv install 2.2.0
7. Sets the global version to Ruby 2.2.0: rbenv global 2.2.0
8. Check the ruby Version: ruby -v

Start to Taste your Ruby and Mac!!


Update: i found this link which is great steps and explanation to install  Ruby by rbenv:
https://cbednarski.com/articles/installing-ruby/

2 links to fix the black screenshots on your WebPageTest private instance for Windows

How to make RDP window session never expired:
http://technet.microsoft.com/en-us/library/cc754272.aspx

How to fix the black screen when you minimize your RDP:
http://serverfault.com/questions/48650/remote-desktop-session-black-after-minimize