Links of the Week (weekly)

Posted from Diigo. The rest of my favorite links are here.

Fix HtmlUnit Memory Leak

The Problem

While testing pages with heavy JavaScript usage in HtmlUnit, great amounts of heap memory are allocated but never released.

Solutions

There are numerous solutions, which can be applied to fix an OutOfMemory error related to HtmlUnit:

Call webClient.closeAllWindows();

The first often suggested remedy to fix a HtmlUnit memory leak is to call webClient.closeAllWindows() after a page has been processed completely.

Increase the JVM Heap Size

One should consider that HtmlUnit is very memory hungry and therefore an increase of the heap size might be in order. Just add a VM argument such as:

-Xmx1024m

Explicitly Stop All JavaScript Jobs

It might also be helpful to explicitly stop all JavaScript background jobs (Note: this is not be required in newer versions of HtmlUnit). For this, add the following code before calling .closeAllWindows:

final List<WebWindow> windows = webClient.getWebWindows();

for (final WebWindow wd : windows) {

    wd.getJobManager().removeAllJobs();

}

webClient.closeAllWindows();

Don’t Forget to Trigger GC

In some configurations the JVM might delay clearing up objects after a test case has been completed. If you monitor the memory usage, make sure to request an explicit garbage collection run (e.g. by clicking [Perform GC] on the ‘Monitor’ page of VisualVM). Trigger it multiple times, too.

Analysis

Processing pages with complex JavaScript procedures appears to result in a large number of heavily memory consuming objects of the class ‘net.sourceforge.htmlunit.corejs.javascript.ScriptableObject$Slot’.

A possible strategy to avoid creation of many of these objects is to limit the usage of JavaScript within an application.

Background

As reported here earlier, a complex test suite for the Appjangle platform is constantly executed using the opsunit framework.

To test the implementation of the Nextweb JavaScript API a number of these test cases use the qunit-htmlunit-java wrapper to run quite extensive QUnit test suites with HtmlUnit. This has been proven to be quite troublesome on a memory-restricted server. The main reason for this appeared to be the complexity of the JavaScript exercised during the test.

Get Free Memory on Unix System in Java

Background

The opsunit framework allows to continuously run a number of JUnit unit tests and do some maintenance/repair work upon any failed test.

The latest addition to the test suite for the Appjangle platform is a simple unit test, which checks the remaining free memory on the server node. If the free memory goes beyond a certain threshold (e.g. 150 mb), some maintenance work is conducted in order to free memory on the node.

Since the server nodes are running a UNIX derivate, the Java unit test therefore needs to be able to determine the currently available system memory.

Problem

How to determine the available free system memory on a UNIX system from a Java program?

Solution

The first starting point is the ‘vmstat’ application. This application prints out all different kinds of information for the current system. Together with a few other GNU programs, it can be moulded to return the available free memory:

vmstat -s -S M | egrep -ie ‘memory|swap’ | grep ‘free memory’

It is possible to run this shell script from Java. There are multiple ways to achieve this, many of them troublesome and I ALWAYS do it wrong. Therefore I have assembled my hard-learned best practices (and those of others I could find) in the project java-start-process. Using this project, we can determine the free memory for a UNIX system using the following function:


public static int getFreeMemoryUnixInMb() {
 try {

final String expr = Spawn
 .runCommand(
 new String[] { "/bin/bash", "-c",
 "vmstat -s -S M | egrep -ie 'memory|swap' | grep 'free memory'" },
 null);
 // expr sth like " \t \t778 M free memory "

final String[] elems = expr.split(" |\t");
 for (final String elem : elems) {

try {
 return Integer.valueOf(elem);
 } catch (final Throwable t) {

}
 }

throw new RuntimeException(
 "Could not find free memory within: Elements="
 + Arrays.asList(elems) + " Raw Result=[" + expr
 + "]");

} catch (final Throwable t) {
 throw new RuntimeException(t);
 }
}

 

Resources

vmstat :: Detect Problems That Can Adversely Affect System Performance

Executing shell commands in java

executing shell script from java