Debugging, Profiling Servlets
|What tools/techniques are available for profiling my servlets/JSPs/JVM/IIS?
This technique takes a "snapshot" of the memory
usage of a currently running JVM:
Here is a simple JSP that outputs some basic information
about the state of the JVM's heap. If you suspect that
your application may be suffering from a memory or resource
leak then manually request this JSP (periodically) to see if
your heap is growing and growing over time, yet never shrinking.
Note that if you are using SE 6.0 or higher then you do not need
to use this JSP since the SE admin UI outputs this information
on the JVM Settings page.
<%@ page import = "java.text.*" %>
long maxMemory = Runtime.getRuntime().maxMemory();
long totalMemory = Runtime.getRuntime().totalMemory();
long freeMemory = Runtime.getRuntime().freeMemory();
DecimalFormat df = new DecimalFormat("00.00%");
double currentCapacity = (double)totalMemory / (double)maxMemory;
String s_currentCapacity = df.format(currentCapacity);
double currentFree = (double)freeMemory / (double)totalMemory;
String s_currentFree = df.format(currentFree);
double growthRoom = ((double)maxMemory - (double)totalMemory) / (double)totalMemory;
String s_growthRoom = df.format(growthRoom);
StringBuffer buf = new StringBuffer();
buf.append("max heap size = " +maxMemory+ " bytes = " +maxMemory/1024+ " Kb = " +maxMemory/(1024*1000)+ " Mb");
buf.append("<br>current heap size = " +totalMemory+ " bytes = " +totalMemory/1024+ " Kb = " +totalMemory/(1024*1000)+ " Mb");
buf.append("<br>free/unused heap = " +freeMemory+ " bytes = " +freeMemory/1024+ " Kb = " +freeMemory/(1024*1000)+ " Mb (if zero it just means <1Mb)");
buf.append("<br>The heap is currently "+ s_currentCapacity + " of its maximum possible capacity.");
buf.append("<br>Of that, "+ s_currentFree + " is available for use.");
buf.append("<br>The current heap will grow by, "+ s_growthRoom + " if neccessary before reaching its maximum possible size.");
If on Windows, consider using the Windows Task manager to monitor the memory usage and/or the CPU usage of the appropriate java.exe process and/or w3wp.exe process. Details about the Windows Task manager given in BlueDragon FAQ #393 may be adapted for use with ServletExec.
Do a thread dump of the JVM in which your SE AS instance is running at the time the problem is occurring. Here are the general steps for doing that:
- stop the SE AS Windows Service
- make a backup copy of StartServletExec.bat
- edit StartServletExec.bat to remove the -Xrs option (if present)
- use the keyboard to invoke StartServletExec.bat
- reproduce the problem
- use Ctrl-Break to get a thread dump of the JVM when it is in the problem state. For example if the problem is that java.exe goes to 100% CPU when requesting a specific JSP page, then request that page and then do the Ctrl-Break.
- When finished, capture the resulting thread dump (shown in the DOS window) to a file. Then restore the -Xrs as it was before and then stop SE running from the batch file (i.e. close the DOS window, or manually invoke StopServletExec.bat). Then start the SE AS instance up as a service so that your application will be up and running for your users.
- Study the thread dump
Here is how we were able to configure the JVM 1.6 being used by SE 6.0 AS on Windows to write a dump file to the hard-drive anytime an OutOfMemoryError occurs:
- Turn off the SE AS 6.0 instance
- Make a backup copy of the StartServletExec.bat file
- Open StartServletExec.bat file in a plain text editor and edit it as follows:
Find this line in the file:
set jvmOptions=%jvmOptions% -Djava.naming.factory.initial=com.newatlanta.servletexec.InitialContextFactory
Directly under that line, add these lines:
:: telling the JVM to write a Heap Dump file to the hard-drive when an OutOfMemoryError occurs
:: the folder must exist else the JVM won't be able to write the dump file
set jvmOptions=%jvmOptions% -XX:+HeapDumpOnOutOfMemoryError
set jvmOptions=%jvmOptions% -XX:HeapDumpPath="%heapDumpsFolder%"
- Save the change.
- Start SE by double-clicking the batch file instead of starting it as service. If the DOS window comes up and stays up, and you can access your servlets or the SE admin pages then you know you have not added syntax errors in your StartServletExec.bat.
- Turn it off by either double-clicking the StopServletExec.bat file, or by just closing the DOS window.
- Now start the SE AS 6.0 instance as a Windows Service (which is how it would typically be running)
- Optional: Write a simple servlet that will cause an OutOfMemoryError and request it to confirm that a dump file was written. Note that using a JSP to test that won't likely work since all JSPs catch Throwable by default, so the OOM Error would never make it up to the JVM for a heap dump file to be written. Likewise for a servlet that is running inside a webapp that is configured to catch the java.lang.OutOfMemoryError or java.lang.Error, or java.lang.Throwable (via configured error pages inside the webapp for example)
If you have reason to believe that the problem is NOT on the java side of SE, but is rather on the native side (within IIS for example) then consider using the Microsoft Debug Diagnostic Tool v1.1 to both generate and analyze a dump of the w3wp.exe process.
Try using the Java Interactive Profiler (JIP).
We like it because it is very lightweight and can be "turned on" and "turned off" against a continuously running JVM.
When you download it, for simplicity, be sure to download the non-source ZIP file. For example, at the time of this writing, the link above offers a file named
jip-src-1.1.1.zip for download. We are suggesting that you do not download that ZIP, but rather click on the "View All Files" link there and download
jip-1.1.1.zip instead. The reason we make this recommendation is because the src distribution contains an example.bat file which does not work without modification (paths are not surrounded with double-quotes, a reference to an ANT class that does not exist, etc...) so having the src distribution could cause unnecessary confusion and frustration. The non-source distro has all you need to use JIP.
After you have downloaded that ZIP, unpack its contents to a new, empty folder.
Then read the readme.html file that is in the "doc" folder.
Use the information there to perform these suggested steps:
Turn off your Application Server (ServletExec, BlueDragonJX, etc...).
If on Windows, turn off the App server from the Services control panel so that it is no longer running as a Windows service (and thus no longer running at all). For ServletExec this would mean turning off the Windows Service whose name is "ServletExec-<instanceName>".
Then use your browser (and/or the list of java.exe processes shown in the Windows Task Manager) to confirm that the Application Server is truly turned off.
Find the batch file (or unix shell script) that is used to start your Application Server. Keep in mind that one batch/shell may simply call another (which may call another and another, etc....). What you are really looking for is the 1 batch/script that is cranking up the java interpreter (java.exe for example).
For ServletExec this is typically StartServletExec.bat.
Once you have found that file, make a backup copy of it.
Now start your Application Server using the batch file and use your browser to make sure that your application works.
If it does, then turn the App Server off via the Stop batch/script file (or by closing the command window in which it had been running) and then make sure the App Server is truly turned off (check this in the same manner that you did earlier).
Now edit that batch/script file to add the 2 JVM options that are discussed in the JIP readme.html file. Here we are talking about
-Dprofile.properties. The locations specified for each of those will depend on where you unpacked the JIP ZIP file.
The profile.properties file that we recommend you point to comes with the JIP ZIP and it's name is "webapp.profile.properties".
For example, to hook JIP into ServletExec 5, where the JIP ZIP has been unpacked to C:\jip\ one might do the following:
- locate the
-Djava.factory.naming.initial JVM option that is already in StartServletExec.bat.
- add the following (including the double-quotes that are shown just before that -D JVM option so that that section of the file looks like this:
"-javaagent:C:\jip\profile\profile.jar" -Dprofile.properties="C:\jip\profile\webapp.profile.properties" -Djava.factory.naming.initial
- Save the change and then start your app server (ServletExec for example) by manually invoking the start script/batch file. You should see JIP-related startup messages in the command window. Then make sure your application is working and is accessible from a browser.
When you are ready to turn on JIP, follow the simple steps given in the "Interactive profiling" section of the readme.html file that comes with JIP.
To help you with this step, here are the contents for a batch file that could be used to setup, turn on, and then turn off the profiler. As it is written now, it would need to be placed inside the "client" folder:
call file.bat localhost 15599 "%outputFile%"
echo The next step will be to turn on the profiler...
call start.bat localhost 15599
echo Run your test case, or put things into the state you wish to analyze. Then return here and continue...
call finish.bat localhost 15599
echo JIP has been turned back off, go analyze the output file [%outputFile%].
This technique takes a "snapshot" of the state of the objects in a currently running JVM:
JDK 1.6 (for both windows & linux) comes with a cool tool (jmap) to
help with debugging out-of-memory issues. This tool is also available in
earlier versions of the JDK for linux only.
Here's what you do to use it (using BlueDragon JX as an *example*):
- Invoke jps to find the BlueDragon Server JX process id.
Here's example output:
- Invoke the jmap command on that process id, sending the output to a file.
Here's an example command:
jmap -histo 6484 > jmap_output.txt. Here's
num #instances #bytes class name
1: 28095 3794824 <constMethodKlass>
2: 28095 2251856 <methodKlass>
3: 52698 1998680 <symbolKlass>
4: 2187 1469592 <constantPoolKlass>
5: 19260 1445280 [C
45: 299 7176 java.security.Permissions
46: 270 6480 java.lang.ref.WeakReference
47: 252 6048 java.util.Vector
48: 147 5880 com.newatlanta.bluedragon.Color
Other profiling tools we are aware of are:
OptimizeIt (see SE FAQ #127 for important info. about OptimizeIt.)