Wednesday, November 22, 2017

How to find High CPU% Threads in Jboss through JMX Console

1. Open your JBoss JMX Console page
2. Find the Path : [jboss.system]: [ServerInfo] : [listThreadCpuUtilization]
3. Then you can Dump your threads by listThreadDump() operation to find out what the threads are doing

Wednesday, October 18, 2017

NMT and Java Native memory leak

Java application process used memory usually include JVM Heap, non-heap(PermGen/Metaspace) and Native code which including JVM internals and native OS libs. As we noticed our Physical memory is out of memory after application running a while, however, the Heap usage is fine and normal, using TOP cmd, we found it has eaten almost all the physical memoy and even much bigger than -Xmx heap size we assign to the heap , the first thought come into my mind is maybe Native Memory come into trouble...
But how to make a conclusion to figure point to Native memory?
using -XX:NativeMemoryTracking=summary to help (after JDK7_40?)
after you add above option into your JVM startup config file, first to make sure you have a root permission or switch to root user to run following cmd, for example:
sudo -u {UID} /opt/java/bin/jcmd {PID} VM.native_memory baseline

after the testing running for a while, run another cmd to show your difference comparing with baseline:
sudo -u {UID} /opt/java/bin/jcmd {PID} VM.native_memory summary.diff

PS. if you do not using sudo -u {UID}, you may get exceptions like following:

java.io.IOException: Operation not permitted

or
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
This could give you some high level idea if you have the Native memory leak or not, but which Object brings you trouble ,  Since NMT doesn't track memory allocations by non-JVM code,  you can use jemalloc / pmap to detect memory leaks in native code, few good posts for your reference : http://jenshadlich.blogspot.com/2016/08/find-native-memory-leaks-in-java.html
or http://lysu.github.io/blog/2015/02/02/how-to-deal-with-non-heap-or-native-memory-leak/

Tuesday, March 07, 2017

Questionnaire Template for API Performance Review Process


  • API Owner: Product Owner, Dev Lead and QA Lead
  • Release Target Date
  • Business Impact(GMS or Save Cost), Impacted flows and User types
  • How much traffic expected During Peak hour(TPS/TPM/TPH)
  • API Name/EndPoint & Method/Sample Request & Response?
  • API priority based on its traffic and importance
  • High level Design and Workflow diagram for the API or API dependencies
  • How many Roles get deployed, and their .war names/versions
  • Existing or New API, If an existing API, any Monitoring Dashboard and Baseline captured against PROD?
  • JDBC queries
  • Third parties dependencies and End points
  • Firewall Ruleset/Gateway Synapse changes
  • Project Wiki page Link
  • Dev API testing plan/scripts/results for reference