GWT Object Serialization with gwt-storage

Problem

The built in Java Serialization and various serialization frameworks available for Java allow to serialize Java objects conveniently into binary or textual representations. Unfortunately, most of these frameworks are not available for Google Web Toolkit client applications.

There is no easily available built in solution for serializing GWT objects into text on the GWT client side. This is surprising since GWT evidently has a mechanism to serialize and deserialize objects to support GWT RPC.

Solution

Seanchenxi has kindly created a convenient wrapper around the GWT RPC serialization mechanism, which makes it very easy to serialize and deserialize objects on a GWT client application into String representations. The project, gwt-storage, is available on github:

https://github.com/seanchenxi/gwt-storage

You can download the source code and import it to your Java project. I’ve also uploaded the project to a public Maven repository. You can add it to your Maven project as follows:


<dependencies>

<dependency>

<groupId>com.seanchenxi.gwt</groupId>

<artifactId>gwt-storage</artifactId>

<version>1.2</version>

</dependency>

</dependencies>

…

<repositories>

<repository>

<id>Appjangle</id>

<url>http://maven.appjangle.com/appjangle/releases/</url>

</repository>

</repositories>

With gwt-storage available for your project, you can do the following in GWT client side code:


StorageSerializer serializerImpl = new com.seanchenxi.gwt.storage.client.serializer.StorageRPCSerializerImpl();

try {

MyClass obj = new MyClass("1234");

String serialized = serializerImpl.serialize(Serializable.class, obj);

GWT.log("Serialized Object: "+serialized);

MyClass deserialized = (MyClass) serializerImpl.deserialize(

Serializable.class, serialized);

} catch (SerializationException e) {

throw new RuntimeException(e);

}

Note:

  • If you download the gwt-storage source code, you might have to change the visibility of the class StorageRPCSerializerImpl() to public (from package).
  • MyClass needs to be serializable by GWT RPC; that is, implement Serializable and have non-transient references that are Serializable as well.
  • If you change the implementation of the class MyClass, deserialization of Strings created with older versions of the class will probably not be possible.

References

Serializing objects in GWT and deserializing them in servlet

GWT Google Groups ‘GWT client side Java object serialization’

Setting Up SSL with Netty

SSL is unquestionably a very useful and powerful technology to make systems more secure. Java in general and Netty in specific provide strong out-of-box support for supporting SSL encryption and authentication. However, since SSL is a complex technology it can often be cumbersome and difficult to set it up correctly.

The following steps describe how an SSL certificate can be obtained and configured to be used in Netty.

Step 1: Create a Keystore

Create a keystore using Java’s keytool (it’s in the [path to java]/bin directory if it’s not on your classpath):

keytool -keysize 2048 -genkey -alias tomcat -keyalg RSA -keystore tomcat.keystore

Step 2: Create a Certificate Signing Request (CSR)

You need to generate this request to submit it to whichever party you would like to issue your SSL certificate (e.g GoDaddy or StartSSL). Again, use Java’s keytool:

keytool -certreq -keyalg RSA -alias tomcat -file yourdomain.csr -keystore tomcat.keystore

Step 3: Upload Certificate Singing Request to Issuer

Go to your issuer’s website and upload the CSR where possible. For GoDaddy, the necessary form looks as follows:

Just copy and paste the contents from the yourdomain.csr file.

Step 4: Download Certificate from your Provider

Download the certificate from your provided/issuer. For instance, for godaddy.com, select the certificate for Tomcat.

Step 5: Installing the Certificates into your Keystore

You will need to install both the certificate for your server as well as some certificates from your issuer (see intermediary certificates).

Below we will install the certificates provided from GoDaddy. First, install the root certificates from GoDaddy into your keystore (the names for the crt files are different for different providers).

keytool -import -alias root -keystore tomcat.keystore -trustcacerts –file gd_bundle.crt

Then, install the intermediary certificates:

keytool -import -alias intermed -keystore tomcat.keystore -trustcacerts –file gd_intermediate.crt

The tool might prompt you that the “Certificate already exists in keystore under alias <root>”. Type “yes” to add it in any case.

Lastly, install the certificate for your server:

keytool -import -alias tomcat -keystore tomcat.keystore -trustcacerts –file yourdomain.com

This time, the tool should report “Certificate reply was installed in keystore”.

The result of this procedure should be that the tomcat.keystore file should have grown in size to about 8 kb (from 3 kb after step 2).

Optional: Also download the ‘Go Daddy Class 2 Certification Authority Root Certificate‘ from the godaddy website and add it to your store. Add it with the alias ‘root2’.

Step 6: Double-Check you have Installed all Certificates

Use to keytool to verify that you have all certificates installed correctly.

keytool –list –keystore tomcat.keystore

This should show you that your keystore contains three entries: ‘root’, ‘intermed’ and ‘tomcat’. (And ‘root2’ if you have added the Go Daddy Class2 certificate).

Step 7: Prepare Keystore for Netty

There are many ways to provide the keystore we have just create to Netty. The following is just one option (but one I find particularly handy):

1. Convert your keystore into a String using Base64Coder and the OneUtils.

final File original = new File(pathtoyourkeystore);

System.out.println(Base64Coder.encode(

OneUtilsJre.toByteArray(new FileInputStream(original))));

2. Copy the String and place it into a static variable, for instance:

class MyKeystore {

public static data = "[generated base 64 data]";

}

3. Create an SSLContext from the data of your keystore.

SSLContext serverContext = SSLContext.getInstance("TLS");

final KeyStore ks = KeyStore.getInstance("JKS");

ks.load(new ByteArrayInputStream(Base64Coder

.decode(MyKeystore.data)),

"yourkeystorepassword".toCharArray());

final KeyManagerFactory kmf = KeyManagerFactory

.getInstance(algorithm);

kmf.init(ks, "yourkeystorepassword".toCharArray());

serverContext.init(kmf.getKeyManagers(), null, null);

4. Create a Netty SSLHandler from the created context:

final SslHandler sslHandler = new SslHandler(serverContext);

5. Add the SSL handler to your Netty pipe (remember this must be the first handler):

pipeline.addLast("ssl", sslHandler);

Now deploy to server and enjoy the SSL security.

References

Generating a CSR and Installing an SSL Certificate in Tomcat 4.x/5.x/6.x

Tomcat 5 SSL – Install GoDaddy Wildcard Certificate JKS / PKCS12 ? What?

Eclipse Wiki – Generating a Private Key and a Keystore

GWT RPC Serialization for LocalStorage

GWT RPC is a great technology for sending ‘Java’ objects from a Java sever to a JavaScript client and vice averse. Since GWT RPC provides facilities for serializing and deseralizing Java objects, it seems like a good option, too, for preparing objects to be stored in a browsers LocalStorage.

Unfortunately, the devil lies in the details here, since GWT RPC is implemented in an asymmetrical way:

  • The client can only deseralize objects serialized on the server and
  • Only the server can deseralize objects serialized on the client.

Fortunately, however, with a few modifications the GWT RPC serialization mechanism can be adjusted to support client-client serialization.

The gwt-storage project available on GitHub does exactly that for us. Check out the getting started guide for this project here. Here a simple example:

StorageExt localStorage = StorageExt.getLocalStorage();

localStorage.put(key, gwtObject);


GwtClass gwtObject = localStorage.get(key);

Note that for this code to work, GwtClass must be involved in one of your GWT RPC services.

Further Information

Stackoverflow ‘How to apply SerializationStreamWriter for storage’

GWT Docs class RPC

SerializationStreamWriter (client)

Tutorial: Upload from Java, Download from JavaScript

This tutorial.describes how a simple unit of data, the text ‘Hello, Java!’ can be uploaded from a Java application and retrieved with a JavaScript application. 

Upload from Java

This section lists all steps required to create a small application that store the text ‘Hello, World’ in the Appjangle cloud database.

Step 1: Download Client Libraries

First download the latest version of the Appjangle client libraries from this page.
Chose the ‘Appjangle Client with Nextweb API’. The Nextweb API is more declarative in nature than the more low-level onedb API and therefore allows in most cases to write more concise code.
If you use Maven in your project, you can also use the provided Maven dependencies to link the library to your project.

Step 2: Link Client Libraries

Create a new project in your favorite IDE. This example will make use of eclipse.
Copy the downloaded .jar file into a your project directory (e.g. ‘lib’) and add the library to the classpath/ build path.

Step 3: Write App to Upload Data

Create a new class with the name ‘Upload’. Add a Java main method to the class as follows:
import io.nextweb.Query;
import io.nextweb.Session;
import io.nextweb.jre.Nextweb;

public class Upload {

    public static void main(String[] args) {
        Session session = Nextweb.createSession();

        Query hello = session.seed().append("Hello, Java!");

        System.out.println("Created:\n"+hello.get());

        session.close().get();
    }

}
Running this application should result in an output such as the following:
Created:
node("http://slicnet.com/seed1/seed1/2/1/2/h/sd/Hello__1", 
  class java.lang.String)
You can now access the created node using the reported URI. For instance, by opening http://slicnet.com/seed1/seed1/2/1/2/h/sd/Hello__1 
in a web browser.

Download from JavaScript

This section lists all steps required to create a small application that store the text ‘Hello, World’ in the Appjangle cloud database using JavaScript.

Step 1: Download and Extract Appjangle Bootstrap

Head over to github to download the Appjangle Bootstrap project or, even better, fork it if you are familiar with github.
Extract the project and open app.html in your faverioute HTML/JS editor.

Step 2: Write App to Download Data

Replace the text // Your JavaScript here with the following application:
<body>
    <script
        src="http://appjangle.com/js/v01/appjangle/appjangle.nocache.js">
    </script>

    <script>
        window.onNextwebOnedb = function() {
            var session = Nextweb.createSession();
            var hello = session
                    .node("http://slicnet.com/seed1/seed1/2/1/2/h/sd/Hello__1");
            hello.get(function(node) {
                document.body
                        .appendChild(document.createTextNode(node.value()));
            });
        }
    </script>
</body>
Save app.html and close your editor.

Step 3: Upload App to Appjangle

Open the TextSync JAR file in the appjangle-bootstrap directory (for instance, by double clicking on the file if supported by your OS).
Drag and drop the file app.html onto the files list in the TextSync App and click [Synchronize].
Note: You will need to get an Appjangle account in order to use the TextSync App. All your uploaded applications will be stored under this account.

Step 4: Open Application

Open app.html again. In the beginning of the file, copy the URL to the right of one.upload .
Past the URL into your web browser. Add to the end of the URL .value.html and replace https with http at the beginning of the URL.
Loading the page should result in the output (also see an example deployed app here):

Hello, Java

It’s not ‘Hello, JavaScript’ since here we are loading the node created by the Java Hello, World application described above. Check out the Nextweb.io API docs to find out how to change the text stored in the Appjangle cloud to a more fitting ‘Hello, JavaScript’.
This tutorial is also published on the Appjangle blog.

Start JTreeMap from Command Line

In the course of writing up my thesis, I wanted to perform a simple source code analysis on the Appjangle code base. Since most of the code is as of now stored in a SVN repository, I used StatSVN for this purpose. StatSVN is really easy to use and generate some very nice statistics and diagrams.

However, one of the visualization, the Repo Heatmap, requires a Java Applet based on JTreeMap to be run on the page. Most current browsers are a bit picky about running Java Applets, and also I have quite a number of Java versions floating about in my machine – in consequence, it would be quite a hassle to configure my browsers to start the Java applet.

Thus, I wanted to load the Repo Heatmap using the JTreeMap standalone Swing Application. Now, most of the documentation on the JTreeMap homepage focusses on its use as an Applet, but it’s really quite easy to launch it as standalone application as well.

JTreeMap is bundled in a single JAR file. However, plugging this file into a call to ‘java –jar’ does not yield much good; since there is no MANIFEST.MF file is not configured appropriately for this. Luckily, the application can easily be started using the ‘-cp’ parameter:

java –cp jtreemap-1.1.0.jar net.sf.jtreemap.swing.example.JTreeMapExample

Once the application is launched, you can open a data file. For instance, the data file generated by StatSVN: repomap-data.txt.

Then, you can view the heat map, even without having to install the Java plugin:

Fix HtmlUnit Memory Leak

The Problem

While testing pages with heavy JavaScript usage in HtmlUnit, great amounts of heap memory are allocated but never released.

Solutions

There are numerous solutions, which can be applied to fix an OutOfMemory error related to HtmlUnit:

Call webClient.closeAllWindows();

The first often suggested remedy to fix a HtmlUnit memory leak is to call webClient.closeAllWindows() after a page has been processed completely.

Increase the JVM Heap Size

One should consider that HtmlUnit is very memory hungry and therefore an increase of the heap size might be in order. Just add a VM argument such as:

-Xmx1024m

Explicitly Stop All JavaScript Jobs

It might also be helpful to explicitly stop all JavaScript background jobs (Note: this is not be required in newer versions of HtmlUnit). For this, add the following code before calling .closeAllWindows:

final List<WebWindow> windows = webClient.getWebWindows();

for (final WebWindow wd : windows) {

    wd.getJobManager().removeAllJobs();

}

webClient.closeAllWindows();

Don’t Forget to Trigger GC

In some configurations the JVM might delay clearing up objects after a test case has been completed. If you monitor the memory usage, make sure to request an explicit garbage collection run (e.g. by clicking [Perform GC] on the ‘Monitor’ page of VisualVM). Trigger it multiple times, too.

Analysis

Processing pages with complex JavaScript procedures appears to result in a large number of heavily memory consuming objects of the class ‘net.sourceforge.htmlunit.corejs.javascript.ScriptableObject$Slot’.

A possible strategy to avoid creation of many of these objects is to limit the usage of JavaScript within an application.

Background

As reported here earlier, a complex test suite for the Appjangle platform is constantly executed using the opsunit framework.

To test the implementation of the Nextweb JavaScript API a number of these test cases use the qunit-htmlunit-java wrapper to run quite extensive QUnit test suites with HtmlUnit. This has been proven to be quite troublesome on a memory-restricted server. The main reason for this appeared to be the complexity of the JavaScript exercised during the test.

Get Free Memory on Unix System in Java

Background

The opsunit framework allows to continuously run a number of JUnit unit tests and do some maintenance/repair work upon any failed test.

The latest addition to the test suite for the Appjangle platform is a simple unit test, which checks the remaining free memory on the server node. If the free memory goes beyond a certain threshold (e.g. 150 mb), some maintenance work is conducted in order to free memory on the node.

Since the server nodes are running a UNIX derivate, the Java unit test therefore needs to be able to determine the currently available system memory.

Problem

How to determine the available free system memory on a UNIX system from a Java program?

Solution

The first starting point is the ‘vmstat’ application. This application prints out all different kinds of information for the current system. Together with a few other GNU programs, it can be moulded to return the available free memory:

vmstat -s -S M | egrep -ie ‘memory|swap’ | grep ‘free memory’

It is possible to run this shell script from Java. There are multiple ways to achieve this, many of them troublesome and I ALWAYS do it wrong. Therefore I have assembled my hard-learned best practices (and those of others I could find) in the project java-start-process. Using this project, we can determine the free memory for a UNIX system using the following function:


public static int getFreeMemoryUnixInMb() {
 try {

final String expr = Spawn
 .runCommand(
 new String[] { "/bin/bash", "-c",
 "vmstat -s -S M | egrep -ie 'memory|swap' | grep 'free memory'" },
 null);
 // expr sth like " \t \t778 M free memory "

final String[] elems = expr.split(" |\t");
 for (final String elem : elems) {

try {
 return Integer.valueOf(elem);
 } catch (final Throwable t) {

}
 }

throw new RuntimeException(
 "Could not find free memory within: Elements="
 + Arrays.asList(elems) + " Raw Result=[" + expr
 + "]");

} catch (final Throwable t) {
 throw new RuntimeException(t);
 }
}

 

Resources

vmstat :: Detect Problems That Can Adversely Affect System Performance

Executing shell commands in java

executing shell script from java

Java Get Process Id: Three Approaches

Java provides no robust way to obtain the id of the OS process in which the application is running.

That does not mean, however, that there is no way. There are actually a couple of ways, each with its own advantages and disadvantages. I will describe three possible ways in the following:

1. ManagementFactory.getRuntimeMXBean().getName()

The class ManagementFactory in the package java.lang.management provides access to the “managed bean for the runtime system of the Java virtual machine”. The getName() method of this class is described as:

Returns the name representing the running Java virtual machine.

This name, as it happens, contains the process id in the Sun/Oracle JVM implementation of this methods in a format such as.


System.out.println(ManagementFactory.getRuntimeMXBean().getName());

// --> 742912@localhost

Through applying a simple String split, this thus allows to obtain the pid of the current process through the following expression:


ManagementFactory.getRuntimeMXBean().getName().split("@")[0]

Advantages:

  • Quick and dirty

Disadvantages:

  • Not guaranteed to work on all JVM implementations

2. Use Java Native Interface

At least on most UNIX based systems, it is relatively easy to write a quick JNA wrapper to the C library.


// Alternative 1: interface-mapped class, dynamically load the C library
public interface CLibrary extends Library {
 CLibrary INSTANCE = (CLibrary)Native.loadLibrary("c", CLibrary.class);
}

You can then add a mapping to the getpid function:


public interface CLibrary extends Library {
CLibrary INSTANCE = (CLibrary)Native.loadLibrary("c", CLibrary.class);

int getpid ();

}

And call this mapper as follows:


int mypid = CLibrary.INSTANCE.getpid();

Advantages:

  • Guaranteed to work as long as C library is available
  • Depends on JNA under com.sun. ….

Disadvantages:

  • Only works on UNIX based systems

3. Use the Java Virtual Machine Process Status Tool (jps)

If your JVM comes with the utility Java Virtual Machine Process Status Tool you can call this tool by spawning a new process, and analyse its output to obtain the id of your Java process.

Since there are a million ways to get starting a process from Java and reading its output wrong, I will use the utility java-start-process for this:


public static void getProcessId(final Class<?> mainClass,
 final Callback<String> callback) {

Spawn.startProcess("jps -l", null, new ProcessListener() {

@Override
 public void onProcessQuit(final int returnValue) {

}

@Override
 public void onOutputLine(final String line) {
 final String[] parts = line.split(" ");
 if (parts.length > 1 && parts[1].endsWith(mainClass.getName())) {
 callback.onDone(parts[0]);
 }
 }

@Override
 public void onErrorLine(final String line) {

}

@Override
 public void onError(final Throwable t) {

}
 });

}

Advantages:

  • Works on both Windows and Linux when Oracle/Sun Java is installed and on classpath

Disadvantages:

  • Complex
  • Requires Sun JVM

Resources

stackoverflow – How can a Java program get it’s own process ID?

stackoverflow – How do I obtain the PID of a spawned java process

 

Jenkins StackOverflow Exception during JUnit Test

Problem

The build of a Maven job in Jenkins fails due to a StackOverflowException or a OutOfMemoryException during a JUnit test.

Analysis

Most likely, there is a bug in your application, which leads to infinite recursion or an infinite allocation in resources. So first try to run your test in a different environment (for instance in eclipse, directly through Maven or as plain Java application).

However, sometimes, as in my case, the application works correctly and just requires more memory/stack size than is provided by the JVM during default settings. In this case, you will need to adjust the JVM settings in order to allow for more memory/stack size.

Solution

The JVM parameters, which allow to allocate more memory or stack size to a JVM are:

1. 

-Xss(x)m

where x is the size of the stack you would like to use. For instance: -Xss4m for a 4 MB stack size.

2.

-Xmx(x)m

where x is the size of the Java heap space (=memory) you would like to use. For instance: -Xmx1024m for a 1024 MB heap space.

Unfortunately, there are multiple places in which these parameters can be configured. Essentially, if the increased heap space requirement is in your application, you will need to adjust the parameters for the JUnit test runner in Maven:


<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-surefire-plugin</artifactId>
 <version>2.12.4</version>
 <configuration>
 <argLine>-Xss18m</argLine>
 </configuration>
 </plugin>

However, if the adjusted parameters are required for building your application, you might have to adjust them for the Maven Build Process.

This can be done by adding the parameters to MAVEN_OPTS either:

  • for your job: [Your Job] / Configure / Build / Advanced / MAVEN_OPTS
  • or for the Jenkins instance: Jenkins / Manage Jenkins / Configure System / Maven Project Configuration / Global MAVEN_OPTS

The last possible culprit might be the Jenkins server itself.

You can set the JVM parameters for Jenkins either by specifying them when:

  • starting Jenkins through java [your JVM parameters here] -jar jenkins.war or
  • by editing the jenkins.xml in your JENKINS_HOME folder.

Resources

Increasing Memory of junit Testcases

Stackoverflow – Jenkins build fails after running Cucumber tests on Java heap space exception

Increase heap size in java

Stackoverflow – How to give Jenkins more heap space when it´s started as a service under Windows?