Deploy Java Lambda using SAM and Buildkite

I’ve recently covered how to deploy a Node JS based Lambda using SAM and Buildkite. I would say that this should cover most use cases, since I believe a majority of AWS Lambdas are implemented with JavaScript.

However, Lambda supports many more programming languages than just JavaScript and one of the more important ones among them is certainly Java. In principle, deployment for Java and JavaScript is very similar: we provide a SAM template and an archive of a packaged application. However, Java uses a different toolset than JavaScript, so the build process of the app will be different.

In the following, I will briefly explain how to build and deploy a Java Lambda using Maven, SAM and Buildkite. If you want to get to the code straight away, find a complete sample project here:

https://github.com/mxro/lambda-java-sam-buildkite

First we define a simple Java based Lambda:

package com.amazonaws.handler;

import com.amazonaws.services.lambda.runtime.Context; 
import com.amazonaws.services.lambda.runtime.LambdaLogger;

public class SimpleHandler {
    public String myHandler(int myCount, Context context) {
        LambdaLogger logger = context.getLogger();
        logger.log("received : " + myCount);
        return String.valueOf(myCount);
    }
}

Then add a pom.xml to define the build and dependencies of our Java application:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.amazonaws</groupId>
    <artifactId>lambda-java-sam-buildkite</artifactId>
    <version>1.0.0</version>
    <packaging>jar</packaging>
    <name>Sample project for deploying a Java AWS Lambda function using SAM and Buildkite.</name>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <maven.compiler.plugin.version>3.8.0</maven.compiler.plugin.version>
        <aws.lambda.java.core.version>1.1.0</aws.lambda.java.core.version>
        <junit.version>4.12</junit.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>${aws.lambda.java.core.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>    
    </dependencies>

    
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
    
</project>

We define the lambda using a SAM template. Note that we are referencing the JAR that is assembled using Maven target/lambda-java-sam-buildkite-1.0.0.jar.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    sam-app

Globals:
    Function:
        Timeout: 20
        Environment: 

Resources:
  SimpleFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: target/lambda-java-sam-buildkite-1.0.0.jar
      Handler: com.amazonaws.handler.SimpleHandler::myHandler
      Runtime: java8

Then we need Dockerfile that will run our build. Here we simply start with an image that has Maven preinstalled and then install Python and the AWS SAM CLI.

FROM zenika/alpine-maven:3-jdk8

# Installing python
RUN apk add --update \
    python \
    python-dev \
    py-pip \
    build-base \
  && pip install virtualenv \
  && rm -rf /var/cache/apk/*

RUN python --version

# Installing AWS CLI and SAM CLI
RUN apk update && \
    apk upgrade && \
    apk add bash && \
    apk add --no-cache --virtual build-deps build-base gcc && \
    pip install awscli && \
    pip install aws-sam-cli && \
    apk del build-deps

RUN mkdir /app
WORKDIR /app
EXPOSE 3001

The following build script will run within this Dockerfile and first package the Java application into a Jar file using mvn package and then uses the SAM CLI to deploy the template and application.

#!/bin/bash -e

mvn package

echo "### SAM Deploy"

sam --version

sam package --template-file template.yaml --s3-bucket sam-buildkite-deployment-test --output-template-file packaged.yml

sam deploy --template-file ./packaged.yml --stack-name sam-buildkite-deployment-test --capabilities CAPABILITY_IAM

Finally we define the Buildkite template. Note that this template assumes the environment variables AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided by Buildkite.

steps:
  - name: Build and deploy to AWS
    command:
      - './.buildkite/deploy.sh'
    plugins:
      - docker-compose#v2.1.0:
          run: app
          config: 'docker-compose.yml'
          env:
            - AWS_DEFAULT_REGION
            - AWS_ACCESS_KEY_ID
            - AWS_SECRET_ACCESS_KEY

Now we simply need to create a Buildkite pipeline and link it to a repository with our source code.

Upgrade to Oracle JDK 10 on CentOS/RHEL

With the release of Java 10 only a few days ago, it seems only prudent to update to Java 10 on suitable systems since the support for Java 9 official ends with the release of Java 10. (Note that Java 8 still enjoys long-time support, so it might be the best choice to stick with that on systems which are difficult to change)

  • Go to the official download site and indicate you agree to their terms.
  • Copy the link for jdk-10_linux-x64_bin.rpm
  • Log into your CentOS machine
  • Download the RPM file using the following command (Don’t forget to provide the link you have copied)

wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" [paste copied link here]

  • Install the JDK

sudo yum localinstall jdk-*_linux-x64_bin.rpm

  • Set the default Java version to 10 using alternatives

sudo alternatives --config java

  • Lastly, make sure you are running the correct version of Java:

java -version

 

PlantUML (Open Source Awesomeness)

I’ve always had a soft spot for diagrams. I think that representing information in various visual ways tremendously helps our thinking and understanding. Unfortunately it is often a big headache to create (and maintain) diagrams.

So I was very pleased today when I came across PlantUML. PlantUML is a Java library and web service which renders UML diagrams from text input. Take the following text definition for example:


@startuml
object Object01
object Object02
object Object03
object Object04
object Object05
object Object06
object Object07
object Object08

Object01 <|-- Object02
Object03 *-- Object04
Object05 o-- "4" Object06
Object07 .. Object08 : some labels
@enduml

This will be rendered into the following diagram:

diagram

PlantUML does not just support object diagrams but also many other types of diagrams. There is another service, called WebSequenceDiagrams which focusses on only sequence diagrams (and is not open source) but can be useful if more visually pleasing sequence diagrams are required,

Java Logging – The Ultimate, Easy Guide

On first glance, logging looks like an exceedingly simple problem to solve. However, it is one of these problems which unfortunately become more and more complex the longer one looks at it.

I think because of this, there are many frameworks in Java to support logging (since everyone seems to have thought they have found a solution) with many of them being less than optimal, especially under load.

In effect, for someone who wants to start with logging in Java, there is an overwhelming, confusing and often contradictory wealth of resources available. In this guide, I will provide an introduction to Java logging in three simple steps: First, to choose the right framework. Second, to get your first log printed out onto the screen. And, third, to explore more advanced logging topics. So, without further ado, here the steps to get you started with Java logging:

Framework

The first question to sort out when considering logging for Java is to decide which logging framework to use. Unfortunately, there are quite a few to choose from.

The standard Java logging seems to be very unpopular. Further, it seems that Log4j and Logback both have architectural disadvantages to Log4j 2. In specific in respect to the performance impact which logging has on the host app. Loggly ran some tests on the different logging frameworks and the theoretical advantages of Log4j 2 also seem to be reflected in cold, hard data.

Thus, I think the prudent choice is to go with log4j2 in any but exceptional circumstances.

How To Get Started

The official documentation for Log4j 2 is not very approachable. Simply speaking, you only need to do two things to get ready for logging with Log4j 2.

The first is to add the following Maven dependency:

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.10.0</version>
</dependency>

The second is to create the file src/main/resources/log4j2.properties in your project with the following content:

status = error
name = PropertiesConfig

filters = threshold

filter.threshold.type = ThresholdFilter
filter.threshold.level = debug

appenders = console

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

rootLogger.level = debug
rootLogger.appenderRefs = stdout
rootLogger.appenderRef.stdout.ref = STDOUT

(Note, you may also provide the configuration in XML format. In that case, simply create file named log4j2.xml in src/main/resources)

Now you are ready to start logging!

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class OutputLog { 
  public static void main(String[] args) { 
    Logger logger = LogManager.getLogger(); 
    logger.error("Hi!"); 
  } 
}

Master Class

The real power of using a logging framework is realised by modifying the properties file created earlier.

You can, for instance, configure it to log into a file and rotate this log file automatically (so it doesn’t just keep on growing and growing). The following presents a properties file to enable this:


status = error
name = PropertiesConfig

property.filename = ./logs/log.txt

filters = threshold

filter.threshold.type = ThresholdFilter
filter.threshold.level = debug

appenders = rolling

appender.rolling.type = RollingFile
appender.rolling.name = RollingFile
appender.rolling.fileName = ${filename}
appender.rolling.filePattern = ./logs/log-backup-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=10MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 20

loggers = rolling

logger.rolling.name = file
logger.rolling.level = debug
logger.rolling.additivity = false
logger.rolling.appenderRef.rolling.ref = RollingFile

#rootLogger.level = debug
#rootLogger.appenderRefs = stdout
rootLogger.appenderRef.stdout.ref = RollingFile

This configuration will result in a log file being written into the logs/ folder. If the application is run multiple times, previous log files will be packed into gzipped files:

output

For even more sophisticated logging, you would want to set up a Graylog server and then send the logs there. This can be achieved using the logstash-gelf library. Add the following Maven dependency:

<dependency>
<groupId>biz.paluch.logging</groupId>
<artifactId>logstash-gelf</artifactId>
<version>1.11.1</version>
</dependency>

And then provide a log4j.xml configuration file like the following (replace yourserver.com with your Graylog server):


<Configuration>
<Appenders>
<Gelf name="gelf" host="udp:yourserver.com" port="51401" version="1.1" extractStackTrace="true"
filterStackTrace="true" mdcProfiling="true" includeFullMdc="true" maximumMessageSize="8192"
ignoreExceptions="true">
<Field name="timestamp" pattern="%d{dd MMM yyyy HH:mm:ss,SSS}" />
<Field name="level" pattern="%level" />
<Field name="simpleClassName" pattern="%C{1}" />
<Field name="className" pattern="%C" />
<Field name="server" pattern="%host" />
<Field name="server.fqdn" pattern="%host{fqdn}" />

<DynamicMdcFields regex="mdc.*" />
<DynamicMdcFields regex="(mdc|MDC)fields" />
</Gelf>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="gelf" />
</Root>
</Loggers>
</Configuration>

Then create a new GELF UDP input in Graylog (& don’t forget to open the firewall for udp port 51401) and you are ready to receive messages!

message

Finally, I personally find the logging frameworks with all their dependencies and insistence on configuration files exactly where they expected them a bit intrusive. Thus, I developed delight-simple-log – this very simply project can be used as a dependency in your reusable component; and then linked with Log4j 2 in the main package for an app. That way, the Log4j dependencies will only be present in one of your modules.

 

 

Determine Which JDK Version a JAR/Class File Was Compiled With

Today I came across a nasty error which occurred in a deployed Java application only but not during development or integration tests. The error went something like the following:

java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
at nx.serializerkryo.internal.InternalKryoSerialzer.performToStream(InternalKryoSerialzer.java:33)
at nx.serializerkryo.internal.InternalKryoSerialzer.serialize(InternalKryoSerialzer.java:63)
at nx.serializerkryo.internal.InternalKryoSerialzer.serialize(InternalKryoSerialzer.java:21)
at nx.persistence.jre.internal.OptimizedPersistedNodeSerializer.serialize(OptimizedPersistedNodeSerializer.java:47)
at nx.persistence.jre.internal.OptimizedPe<span 				data-mce-type="bookmark" 				id="mce_SELREST_end" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>rsistedNodeSerializer.serialize(OptimizedPersistedNodeSerializer.java:21)

Now I had a feeling that this had something to do with me trying to be ahead of the curve and use a Java 9 JDK to compile the application. In order to debug this, I had to confirm which with JDK the classes I was using were compiled with. Thankfully I found a handy thread of StackOverflow.

Unfortunately, it wasn’t immediately obvious to me which solution listed there would work best, so I decided to provide the solution here in a more condensed form. Simply use the following command:

javap -v [path to your class file]

The output will then contain the following line (towards the top of the file):

public class ...
minor version: 0
major version: 50
flags: ACC_PUBLIC, ACC_SUPER

The major version and minor version indicates which version of Java the class was compiled with. The following contains a list of which Java versions which major versions relate to.

Java SE 9 = 53 (0x35 hex),
Java SE 8 = 52 (0x34 hex),
Java SE 7 = 51 (0x33 hex),
Java SE 6.0 = 50 (0x32 hex),
Java SE 5.0 = 49 (0x31 hex),
JDK 1.4 = 48 (0x30 hex),
JDK 1.3 = 47 (0x2F hex),
JDK 1.2 = 46 (0x2E hex),
JDK 1.1 = 45 (0x2D hex).

Interestingly my files were apparently compiled for Java 6 (Maven compiler plugin was responsible). The problem was that the files were compiled with JDK 9 (though they were compiled for 1.6). Downgrading the JDK used to do the compilation to JDK8 fixed the problem.

A Handy Reference of Maven Parameters

I cannot count the times I have looked up the following through Google. Thus I decided to put together a few handy parameters (or properties or whatever is the correct term) for Maven builds.

All the below are given with the goal install but they can safely be used with any other Maven goal as well.

Skip Tests

mvn install -DskipTests

Build Only From Specified Project

This is only relevant in a multi pom.

mvn install -rf :[artifactId]

Don’t Compile JavaDoc

-Dmaven.javadoc.skip=true

Don’t Compile GWT

-Dgwt.compiler.skip=true

 

Installing Jenkins on Centos 7

I set up a Jenkins server on a brand new Centos 7 VPS. In the following the instructions for doing this in case you are looking at doing the same:

Setting up Jenkins Server

sudo yum install java-1.8.0-openjdk
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins

Or for stable version (link did not work for me when I tried it)

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum install jenkins
  • Start Jenkins server
sudo systemctl start jenkins

You should now be able to access Jenkins at yourserver.com:8080 (if not, see troubleshooting steps at the bottom).

If you want to access your server more securely on port 80, you can do so by installing ngnix as outlined in this article in step 4: How to Install Jenkins on CentOS 7.

Connecting to a Git Repo

You will probably want to connect to a git repository next. This is also somewhat dependent on the operating system you use, so I provide the steps to do this on CentOS as well:

  • Install git
sudo yum install git
  • Generate an SSH key on the server
ssh-keygen -t rsa
  • When prompted, save the SSH key under the following path (I got this idea from reading the comments here)
/var/lib/jenkins/.ssh
  • Assure that the .ssh directory is owned by the Jenkins user:
sudo chown -R jenkins:jenkins /var/lib/jenkins/.ssh
  • Copy the public generated key to your git server (or add it in the GitHub/BitBucket web interface)
  • Assure your git server is listed in the known_hosts file. In my case, since I am using BitBucket my /var/lib/jenkins/.ssh/known_hosts file contains something like the following
bitbucket.org,104.192.143.3 ssh-rsa [...]
  • You can now create a new project and use Git as the SCM. You don’t need to provide any git credentials. Jenkins pulls these automatically form the /var/lib/jenkins/.ssh directory. There are good instructions for this available here.

Connecting to GitHub

  • In the Jenkins web interface, click on Credentials and then select the Jenkins Global credentials. Add a credential for GitHub which includes your GitHub username and password.
  • In the Jenkins web interface, click on Manage Jenkins and then on Configure System. Then scroll down to GitHub and then under GitHub servers click the Advanced Button. Then click the button Manage additional GitHub actions.

additional actions

  • In the popup select Convert login and password to token and follow the prompts. This will result in a new credential having been created. Save and reload the page.
  • Now go back to the GitHub servers section and now click to add an additional server. As credential, select the credential which you have just selected.
  • In the Jenkins web interface, click on New Item and then select GitHub organisation and connect it to your user account.

Any of your GitHub projects will be automatically added to Jenkins, if they contain a Jenkinsfile. Here is an example.

Connect with BitBucket

  • First, you will need to install the BitBucket plugin.
  • After it is installed, create a normal git project.
  • Go to the Configuration for this project and select the following option:

BitBucket trigger

  • Log into BitBucket and create a webhook in the settings for your project pointing to your Jenkins server as follows: http://youserver.com/bitbucket-hook/ (note the slash at the end)

Testing a Java Project

Chances are high you would like to run tests against a Java project, in the following, some instructions to get that working:

Troubleshooting

  • If you cannot open the Jenkins web ui, you most likely have a problem with your firewall. Temporarily disable your firewall with: `sudo systemctl stop iptables` and see if it works then.
  • If it does, you might want to check your rules in `/etc/sysconfig/iptables` and assure that port 8080 is open
  • Check the log file at:
sudo cat /var/log/jenkins/jenkins.log

 

Continuous Integration Server Overview

Since I plan to set up a continuous integration server in the near future, I had a quick look around for open source and cloud-based solutions; my main concern was finding something which will work for a small scale project and result in reasonable costs.

Jenkins (Open Source)

The best choice if you are looking for an open source CI server. If you are familiar with Java, setting up and running Jenkins on your own is in all likeliness much cheaper than any cloud-based alternative.

Buildbot (Open Source)

Jenkins looks to be more widely used than Buildbot. However, if you have a Python project, Buildbot might be worth considering.

Travis CI (Cloud)

My top choice for open source projects. For commercial projects, however, the costs seem to be quite high starting with US$69 per month.

Circle CI (Cloud)

They offer one build container for free which seems like a very generous offer to me. I haven’t explored though how powerful this container is and how long builds would take.

AWS CodePipeline and AWS CodeDeploy (Cloud)

The best choice if you are using an AWS environment.

Codeship (Cloud)

They offer 100 builds per month for free which seems to be quite reasonable. However, since builds are triggered automatically this figure can be reached relatively quickly even with smaller projects.

 

Selenium ChromeDriver Hangs on Startup

Problem

ChromeDriver and Headless Chrome are a great solution for running automated JavaScript tests.

Today I wanted to run some tests on a Linux Server (Centos 7) and although Chrome and ChromeDriver were installed correctly, my Java app would just hang after ChromeDriver is started:

Starting ChromeDriver 2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4) on port 31042
Only local connections are allowed.

After a while, it would show the following error:

Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
 (Driver info: chromedriver=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4),platform=Linux 2.6.32-042stab112.15 x86_64) (WARNING: The server did not provide any stacktrace information)

Solution

It turns out that the problem was that Chrome does not like to run on Linux operating systems when it is started by the root user.

The quick solution is to add the following argument when launching Chrome:

ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--no-sandbox");

However, this unfortunately has security implications and thus it would be best to find a way to run the Java app which launches ChromeDriver without the root user account.

 

 

 

 

Install Latest JDK on Linux Server

Edit: Updated with new script version provided by MaxdSre.

To install the Oracle JDK on a Linus server is often a tricky proposition. For one, the download page requires to confirm a prompt and only unlocks the download link after this prompt has been confirmed (via a cookie, I think). This makes it difficult to download the binary in the first place!

Thankfully, MaxdSre has created the following handy script to download and extract the JDK:

OracleSEJDK.sh

If you run this script, you are presented with a prompt as follows:

select

Just select the version you require, and the script will download and install the Oracle JDK.

Finally, you might have existing JDK versions installed on your machine which are managed using alternatives. The updated version of the script should already trigger the alternatives command for you to configure which Java version to use.

For reference how to point your ‘java’ command to the new installation manually, please see this article.