Fix HtmlUnit Memory Leak

The Problem

While testing pages with heavy JavaScript usage in HtmlUnit, great amounts of heap memory are allocated but never released.

Solutions

There are numerous solutions, which can be applied to fix an OutOfMemory error related to HtmlUnit:

Call webClient.closeAllWindows();

The first often suggested remedy to fix a HtmlUnit memory leak is to call webClient.closeAllWindows() after a page has been processed completely.

Increase the JVM Heap Size

One should consider that HtmlUnit is very memory hungry and therefore an increase of the heap size might be in order. Just add a VM argument such as:

-Xmx1024m

Explicitly Stop All JavaScript Jobs

It might also be helpful to explicitly stop all JavaScript background jobs (Note: this is not be required in newer versions of HtmlUnit). For this, add the following code before calling .closeAllWindows:

final List<WebWindow> windows = webClient.getWebWindows();

for (final WebWindow wd : windows) {

    wd.getJobManager().removeAllJobs();

}

webClient.closeAllWindows();

Don’t Forget to Trigger GC

In some configurations the JVM might delay clearing up objects after a test case has been completed. If you monitor the memory usage, make sure to request an explicit garbage collection run (e.g. by clicking [Perform GC] on the ‘Monitor’ page of VisualVM). Trigger it multiple times, too.

Analysis

Processing pages with complex JavaScript procedures appears to result in a large number of heavily memory consuming objects of the class ‘net.sourceforge.htmlunit.corejs.javascript.ScriptableObject$Slot’.

A possible strategy to avoid creation of many of these objects is to limit the usage of JavaScript within an application.

Background

As reported here earlier, a complex test suite for the Appjangle platform is constantly executed using the opsunit framework.

To test the implementation of the Nextweb JavaScript API a number of these test cases use the qunit-htmlunit-java wrapper to run quite extensive QUnit test suites with HtmlUnit. This has been proven to be quite troublesome on a memory-restricted server. The main reason for this appeared to be the complexity of the JavaScript exercised during the test.

4 thoughts on “Fix HtmlUnit Memory Leak

  1. Do you have a reproducible example of a leak in HtmlUnit (missing WebClient.closeAllWindows() causes a leak in the code using HtmlUnit, not in HtmlUnit itself)?

    If you run extensive test suites, it’s not surprising that you need some amount of memory.
    I don’t think that there is any value in explicitly stopping JavaScript jobs: WebClient.closeAllWindows() already does it.

    1. I did some more tests with and without stopping the JavaScript jobs explicitly. It appears to me that it does not make a difference whether the Jobs are stopped explicitly or not.

      I therefore highlighted in the post that this procedure is not required when using newer versions of HtmlUnit. As for older versions, I’m not sure but I saw some discussion posts that it helped some people in some cases. So I guess it doesn’t harm to leave it in the post?

      For the testing framework qunit-htmlunit-java though I have removed stopping the jobs, since it does not appear to have any affect with the used HtmlUnit version.

      Thank you for your suggestions, which were all correct. Happy to hear further ideas! :)

  2. Currently I m struggling with memory Leaks in My HtmlUnit-System
    developed on HtmlUnit that is the module of our application, so I have
    applied the suggestions given above, and one more
    concern is that in my module in various functions I have not cleanedUp
    the HtmlPage Object, is that can also be the concern for that, I have
    not cleaned those HtmlPage Objects as I had read in the javadocs of cleanUp method that “Clean up this page. This method gets called by the web client when an other page is
    loaded in the window and you should probably never need to call it
    directly”, please suggest what things I will do, would I cleanedUp
    those HtmlPages Object or not.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s