Setting up Prometheus and Grafana for CentOS / RHEL 7 Monitoring

As mentioned in my previous post, I have long been looking for a centralised solution for collecting logs and monitoring metrics. I think my search was unsuccessful since I was looking for too many things in one solution. Instead I found now two separate solutions, one for log management (using Graylog) and one for metrics (using Prometheus and Grafana). I deployed both of these on very inexpensive VPS machines and so far I am very happy with them.

In this post, I provide some pointers how to set up the metrics solution based on Prometheus and Grafana. I assume you are using a RHEL / CentOS system as the server hosing Prometheus and Grafana and you are interested in the OS metrics for a CentOS system. This tutorial will guide you through setting up the Prometheus server, collecting metrics for it using node_exporter and finally how to create dashboards and alerts using Grafana for the collected metrics.

Installing Prometheus Server

  • Follow the excellent instructions here with the following modifications.
    • Make sure to download the latest version of Prometheus (the link can be obtained on the Prometheus download page, this guide works with version 2.1.0)
    • For the systemd service, use the following file:

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
[Service]
User=root
Restart=on-failure
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml
[Install]
WantedBy=multi-user.target

Note: This configuration will run the Prometheus server as root. In a production environment, it is highly recommended to run it as another user (e.g. ‘prometheus’)

  • If you are using a firewall, add the following rule to /etc/sysconfig/iptables and restart service iptables:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 9090 -j ACCEPT

Viewing a Metric

You should the data of http requests served by the Prometheus server itself. If you click on the tab graph, you can see the data as a graph.

Installing Node Exporter on the Server to be Monitored

tar xvfz node_exporter-*.tar.gz
  • Create link (replace 0.15.2 with the version you have downloaded)

ln -s node_exporter-0.15.2.linux-amd64 node_exporter

  • Define a service for node_exporter in /etc/systemd/system/node_exporter.service

(or if you are using init.d, please see this article).


[Unit]
Description=Node Exporter

[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter --no-collector.diskstats

[Install]
WantedBy=default.target

Note: This configuration will run the Grafana server as root. In a production environment, it is highly recommended to run it as another user (e.g. ‘prometheus’)

(The –no-collector.diskstats is added above since diskstats often does not work in virtualized environments. If that is not an issue for you, be free to leave this argument out.)

  • Enable and start service

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

  • Tell Prometheus to scrape these metrics by adding the following to /opt/prometheus/prometheus.yml
 - job_name: "node"
static_configs:
- targets: ['localhost:9100']

Now if you got to yourserver.com:9090/graph you can for instance enter the expression node_memory_MemFree and see the free memory available on the server.

You can also install node_exporter on another server. Simply point the job definition then to this servers address; and of course remember to open port 9100 on the server.

Installing Grafana

The default Prometheus interface is quite basic. Thankfully Grafana offers excellent integration with Prometheus and will result in a much nicer UI.

You can easily install Grafana on your own server or use a free cloud-based instance (limited to one user and five dashboards).

To install Grafana locally:

  • First follow these instructions.
  • Graphana by default runs on port 3000, so make sure you add the following firewall rule after you install it:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 3000 -j ACCEPT

  • In the file /etc/grafana/grafana.ini, provide details for an SMTP connection which can be used for sending emails (section [smtp]).
  • Also update the host name in the field domain to the address at which your server can be reached on the internet.

Configuring Grafana

  • Go to yourserver.com:3000
  • The default login is username ‘admin’ and password ‘admin’. Create a new user with a good password and delete the admin user.
  • First connect with your Prometheus instance as a data source.
  • Then go to Dashboards and select import

dashboards

Done! You should now be able to see the metrics for your server such as CPU usage or free memory.

dashboard

If you monitor multiple servers, you can switch between them by clicking next to the text ‘node’.

switch

Additional servers will appear here if you add them to the Prometheus configuration:

- job_name: "node"
 static_configs:
 - targets: ['localhost:9100', 'xxxxx']

Configure Alerting

While Prometheus has some build in alerting facilities, alerting in Grafana is much easier to use. To set up altering for the dashboard you have created:

  • Go to Alerting / Notification Channels
  • Click on New Channel
  • Provide a name for the channel and your email address and click Save.

channel

  • Next go to the dashboard you have created: Node Exporter Server Metrics
  • Click on the first panel to select it

panel

  • Next click on Edit in the menu which is shown above the panel
  • Go to the Alert tab and click on Create Alert

create alert

  • Configure the following condition(for more details about this, please see this page):

condition

  • Select the Notification page on the right
  • In Send to, select the notification channel you have created earlier.
  • Provide a message such as: CPU usage high.
  • Save the dashboard (Ctrl+S)

Done! You should now receive notifications if the CPU usage on any of the servers monitored on this dashboard is too high.

Further Reading

Determine Which JDK Version a JAR/Class File Was Compiled With

Today I came across a nasty error which occurred in a deployed Java application only but not during development or integration tests. The error went something like the following:

java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
at nx.serializerkryo.internal.InternalKryoSerialzer.performToStream(InternalKryoSerialzer.java:33)
at nx.serializerkryo.internal.InternalKryoSerialzer.serialize(InternalKryoSerialzer.java:63)
at nx.serializerkryo.internal.InternalKryoSerialzer.serialize(InternalKryoSerialzer.java:21)
at nx.persistence.jre.internal.OptimizedPersistedNodeSerializer.serialize(OptimizedPersistedNodeSerializer.java:47)
at nx.persistence.jre.internal.OptimizedPe<span 				data-mce-type="bookmark" 				id="mce_SELREST_end" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>rsistedNodeSerializer.serialize(OptimizedPersistedNodeSerializer.java:21)

Now I had a feeling that this had something to do with me trying to be ahead of the curve and use a Java 9 JDK to compile the application. In order to debug this, I had to confirm which with JDK the classes I was using were compiled with. Thankfully I found a handy thread of StackOverflow.

Unfortunately, it wasn’t immediately obvious to me which solution listed there would work best, so I decided to provide the solution here in a more condensed form. Simply use the following command:

javap -v [path to your class file]

The output will then contain the following line (towards the top of the file):

public class ...
minor version: 0
major version: 50
flags: ACC_PUBLIC, ACC_SUPER

The major version and minor version indicates which version of Java the class was compiled with. The following contains a list of which Java versions which major versions relate to.

Java SE 9 = 53 (0x35 hex),
Java SE 8 = 52 (0x34 hex),
Java SE 7 = 51 (0x33 hex),
Java SE 6.0 = 50 (0x32 hex),
Java SE 5.0 = 49 (0x31 hex),
JDK 1.4 = 48 (0x30 hex),
JDK 1.3 = 47 (0x2F hex),
JDK 1.2 = 46 (0x2E hex),
JDK 1.1 = 45 (0x2D hex).

Interestingly my files were apparently compiled for Java 6 (Maven compiler plugin was responsible). The problem was that the files were compiled with JDK 9 (though they were compiled for 1.6). Downgrading the JDK used to do the compilation to JDK8 fixed the problem.

Installing Jenkins on Centos 7

I set up a Jenkins server on a brand new Centos 7 VPS. In the following the instructions for doing this in case you are looking at doing the same:

Setting up Jenkins Server

sudo yum install java-1.8.0-openjdk
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins

Or for stable version (link did not work for me when I tried it)

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum install jenkins
  • Start Jenkins server
sudo systemctl start jenkins

You should now be able to access Jenkins at yourserver.com:8080 (if not, see troubleshooting steps at the bottom).

If you want to access your server more securely on port 80, you can do so by installing ngnix as outlined in this article in step 4: How to Install Jenkins on CentOS 7.

Connecting to a Git Repo

You will probably want to connect to a git repository next. This is also somewhat dependent on the operating system you use, so I provide the steps to do this on CentOS as well:

  • Install git
sudo yum install git
  • Generate an SSH key on the server
ssh-keygen -t rsa
  • When prompted, save the SSH key under the following path (I got this idea from reading the comments here)
/var/lib/jenkins/.ssh
  • Assure that the .ssh directory is owned by the Jenkins user:
sudo chown -R jenkins:jenkins /var/lib/jenkins/.ssh
  • Copy the public generated key to your git server (or add it in the GitHub/BitBucket web interface)
  • Assure your git server is listed in the known_hosts file. In my case, since I am using BitBucket my /var/lib/jenkins/.ssh/known_hosts file contains something like the following
bitbucket.org,104.192.143.3 ssh-rsa [...]
  • You can now create a new project and use Git as the SCM. You don’t need to provide any git credentials. Jenkins pulls these automatically form the /var/lib/jenkins/.ssh directory. There are good instructions for this available here.

Connecting to GitHub

  • In the Jenkins web interface, click on Credentials and then select the Jenkins Global credentials. Add a credential for GitHub which includes your GitHub username and password.
  • In the Jenkins web interface, click on Manage Jenkins and then on Configure System. Then scroll down to GitHub and then under GitHub servers click the Advanced Button. Then click the button Manage additional GitHub actions.

additional actions

  • In the popup select Convert login and password to token and follow the prompts. This will result in a new credential having been created. Save and reload the page.
  • Now go back to the GitHub servers section and now click to add an additional server. As credential, select the credential which you have just selected.
  • In the Jenkins web interface, click on New Item and then select GitHub organisation and connect it to your user account.

Any of your GitHub projects will be automatically added to Jenkins, if they contain a Jenkinsfile. Here is an example.

Connect with BitBucket

  • First, you will need to install the BitBucket plugin.
  • After it is installed, create a normal git project.
  • Go to the Configuration for this project and select the following option:

BitBucket trigger

  • Log into BitBucket and create a webhook in the settings for your project pointing to your Jenkins server as follows: http://youserver.com/bitbucket-hook/ (note the slash at the end)

Testing a Java Project

Chances are high you would like to run tests against a Java project, in the following, some instructions to get that working:

Troubleshooting

  • If you cannot open the Jenkins web ui, you most likely have a problem with your firewall. Temporarily disable your firewall with: `sudo systemctl stop iptables` and see if it works then.
  • If it does, you might want to check your rules in `/etc/sysconfig/iptables` and assure that port 8080 is open
  • Check the log file at:
sudo cat /var/log/jenkins/jenkins.log

 

Versioning WordPress with Git and Revisr

WordPress is a powerful platform to just get a simple website up and running. Unfortunately, some things which are considered best practice in software development projects are a bit difficult to realize with WordPress. One of these things is versioning. Thankfully, there is a powerful plug in which enables versioning WordPress using a git repository: Revisr.

As of writing this, one is first greeted by an error message when visiting the Revisr website (something to do with SSL). It is safe to ignore this and to instruct your browser to show this website irrespective of the error message displayed (you won’t be giving any confidential information to this webiste, just browsing around).

In any case, you can download Revisr from WordPress with the following link:

https://wordpress.org/plugins/revisr/

Following the steps for setting it up:

  • Go to your WordPress Admin console
  • Install the plugin
  • Activate it
  • Reload  WordPress Admin console
  • Click on Revisr on the Sidebar
  • Instruct it to create a new repository

That’s already it for setting up a local git repo that will enable some versioning. However, you can also use this plugin to backup your site and all versions to a remote git repository, for instance using BitBucket. The instructions for this come as follows (assuming you are using an Apache Web server):

  • Login to your server using SSH with a user with sudo rights
  • Execute the following
sudo ssh keygen
  • Follow the prompts to create the key
  • Execute the following (where /var/www is the root dir of your Apache server)
sudo cp -r /root/.ssh /var/www

sudo chown -R apache:apache /var/www/.ssh
  • Create the file /var/www/.ssh/.htaccess and put in the following content (this is just a security measure)
Deny from all
  • Grab the public key and save it somewhere
sudo cat /var/www/.ssh/id_rsa.pub
  • Create a new account for BitBucket if you don’t have one already.
  • Add a public SSH key for your account. Add the SSH key you saved earlier.
  • Create a new repository. Grab the git SSH link for this repository.
  • Go back to your WordPress Admin console and select the Revisr plugin from the sidebar
  • Go to Settings / General. Set your username to git and define an email. Click Save
  • Go to Settings / Remote. Set Remote URL to the SSH link for the repository you saved earlier. Click Save.

Now you can go back to the main Revisr page and start committing changes and pushing to the remote repository!

 

Resize EC2 Volume (without Resizing Partition)

Problem

You would like to resize a volume attached to an EC2 instance.

Solution

Do the following:

  • Create a snapshot of your volume (instructions)
  • Stop your instance
  • Go to EBS / Volumes and select Actions / Modify Volume

Modify Vol

  • Enter the new size for your volume (note you can only ever make the volume larger) and click on Modify

size

  • Wait for the modification to be complete (this might take a while, like 30 min or so)
  • Start your instance

Now, if everything went well, you should have more space available on the disk for the virtual machine. To confirm this, run:

df -h

You should see the new size of the volume as the size of your main partition:

size2

Notes

  • If the size of your partition, does not match the size of the volume, you probably need to resize your partition (instructions).
  • Resizing the partition is a very painful process, that I think should best be avoided at all costs. I think for this it helps if the EC2 instance attached to the volume is stopped when the resize is performed. Assure that this is the case before you do the resize.
  • If you forgot to stop your instance, and need to do a partition resize, there is a little workaround. Wait for six hours, then resize your volume again (this time while the instance is stopped). Then, it hopefully adjusts your partition size to the correct size.
  • In the above, you might be able to start up your instance even while the new volume is still optimizing. I haven’t tested this though but my guess is that it would work.

 

Library for Parsing multipart File Upload with Java

One of the most convinient ways to upload files from the Web Browser to the server is by using file inputs in HTML forms.

Many web servers come with preconfigured modules for parsing this data on the server-side. However, sometimes, your HTTP server of choice might not offer such a module and you are left with the task of parsing the data the browser submits to the server yourself.

I specifically encountered this problem when working with a Netty-based server.

The form will most likely submit the files to your server as part of a multipart/form-data request. These are not that straightforward to parse. Thankfully, there is the library Apache Commons FileUpload which can be used for this purpose.

Unfortunately, processing some arbitrary binary data with this library is not very straightforward. This has motivated me to write a small library – delight-fileupload –  which wraps Commons FileUpload and makes parsing multipart form data a breeze. (This library is part of the Java Delight Suite).

Just include the library and let it parse your data as follows:

FileItemIterator iterator = FileUpload.parse(data, contentType);

Where data is a binary array of the data you received from the client and contentType is the content type send via HTTP header.

Then you can iterate through all the files submitted in the form as follows:

while (iter.hasNext()) {
 FileItemStream item = iter.next();
 if (item.isFormField()) {
   ... some fields in the form
 } else {
   InputStream stream = item.openStream();
   // work with uploaded file data by processing stream ...
 }
}

You can find the library on GitHub. It is on Maven Central. Just add the following dependency to your Java, Scala etc. application and you are good to go:

<dependency>
 <groupId>org.javadelight</groupId>
 <artifactId>delight-fileupload</artifactId>
 <version>0.0.3</version>
</dependency>

You can also check for the newest version on the JCenter repostiory.

I hope this is helpful. If you have any comments or suggestions, leave a comment here or raise an issue on the javadelight-fileupload GitHub project.

 

 

Java: Find all instances of System.out.println()

A good Java application should print only the absolute necessary to standard out. Not to do so can mean a serious hit in performance and can make it difficult to debug issues.

Unfortunately either we ourselves or our colleagues put System.out.println() statements in the code and then forget to remove them.

Here are two solutions how to find all those nasty statements.

Solution 1: Do a Full Text Search

Use your IDE and search for all occurrences of the string “System.out.println()” in your code. In eclipse this can be done through Menu / Search / File …

Make sure to use *.java for the file name pattern.

search

Solution 2: Put a Breakpoint in PrintStream

The above solution might not be practical if there are many System.out statements in the code which are legitimately there. In that case, you can also put a break point into the Java standard class PrintStream.

In eclipse, open the search for class dialog (Ctrl+Shift+t) and type in the class name PrintStream from the package java.io.

printstream.PNG

In this class, find the print(x) method and add a breakpoint:

breakpoint

Then run your program in debug mode. You can then find the places where System.out is called from the stack trace.

 

Understanding Creation of GWT Serialization Policy Files (.gwt.rpc)

Today I deep-dived a bit into how GWT creates and reads Serialization Policy files. These are the .gwt.rpc files which are generated beside the JavaScript files for GWT modules.

Chiefly, I learned two things:

  • The .gwt.rpc files are only used by the server. The client never reads them.
  • The .gwt.rpc files are generated in the ProxyCreator class.

I have listed some further classes and links below.

GWT Framework Classes

ProxyCreator: Creates the .gwt.rpc file

RemoteServiceProxy: Manages calls to services

ClientSerializationStreamWriter: Write serialization on client for server

ClientSerializationStreamReader: Read responses from server on client

Serializer: Interface for serialization contract for class

Links

The GWT RPC Wire Protocol

 

Sandboxing JavaScript in Java App – Link Collection

The JVM is by design an insecure environment and it is generally difficult to run untrusted code in a sandboxed environment.

However, it seems that is relatively easy to sandbox JavaScript code running in Oracle Nashorn. The instructions are here. Strangely, this was not easy to find through a Google search.

Below I have listed some further sources on Sandboxing JavaScript and Java code. Although there is plenty of material on Rhino, I would not recommend using this engine. I think Nashorn has been designed with support for Sandboxed code in mind from the very beginning while in Rhino the functionality feels kind of bolted on.

UPDATE I have implemented two little libraries which takes care of the grunt work of sandboxing Nashorn and Rhino code in Java:

Nashorn Sandbox (on GitHub)

Rhino Sandbox (on Github)

Sandboxing JavaScript

Nashorn

Restricting Script Access to Specified Java Classes: From the Oracle Nashorn docs. Shows how to restrict access to specific Java classes.

Rhino

Class ContextFactory: Useful for monitoring and setting restrictions on Rhino code.

Method initSafeStandardObjects: Useful for creating sandboxed Rhino code.

Rhino Sandbox: A small library for sandboxing JavaScript code running in Rhino.

Sandboxing Rhino in Java: Blog post

Securing Rhino in Java6: Blog post

DynJS

Sandboxing JavaScript Execution in Java: Blog post

Sandboxing Java

Example Code Monitoring Threads: Example code how thread CPU usage can be monitored.

The Java Sandbox: A library for sandboxing any Java code. Might be useful to sandbox the Java code with runs the script.

Use Signed SSL Certificate with Java

It seems to me like every time I have to do something that has to do with SSL certificates – be they self-signed or signed by a certificate authority – things do not go smoothly. I only do this from time to time, so I am by no means an expert; but I do believe my difficulties result to some degree from the intrinsic complexities within SSL and the systems which support it.

I have created another guide which walks step by step through the process of configuring a Java key store with a signed SSL certificate. If you do want this to succeed, note that you have to follow every step precisely. Even minor omissions can lead to errors (believe me, I’ve tried it myself).

  • Download Portecle (from here) and Unzip it
  • Start portecle.jar by double clicking it
  • Go to File / New Keystore

  • Select JKS and click [OK]
  • Go to Tools / Generate Key Pair

  • Select Algorithm RSA and Key size 2048
  • Increase validity from the default 356 to 1000 or more days

  • In Common Name provide the domain or subdomain of the domain you want to protect
  • Provide some input for all other fields – do not leave any empty
  • Provide a password and remember it (This can be the same password as for the the whole store)
  • Provide an alias – best the name of your domain

  • You should see the following:

  • Right click the key pair you have create and select ‘Generate Certification Request’

  • Portecle will generate file ‘XYZ.csr‘ for you.
  • Provide the contents of this file to the SSL provider of your choice (see a brief comparison here – I’ve had good experiences with RapidSSL certificates from GoGetSSL).
  • Your SSL provider should supply you with an SSL certificate. This file should end with ‘.crt‘. Download it.
  • Go back to Portecle and right click your key pair again. Select ‘Import CA Reply’.

  • Import the .crt file you got from your SSL provider.
  • If this does not work, first proceed to import the certificates as listed in the next steps, then try again to import the CA Reply.
  • You can import the ROOT certificate of your SSL provider just in case.
  • Also, your SSL provider will supply you with an intermediate and server certificate. You can import these into your keystore as well.
  • Note that when importing the ROOT certificate of your provider, you might get a warning that no trust chain can be established to the certificate. However, when importing the intermediate and server certificates AFTER importing the root certificate, there should be no warning that no chain can be established.
  • Your keystore should look something like this now:

  • Now go to File / Save Keystore
  • Provide the same password you used before.

Now you can use the created key store in Java servers. For an easy way how to use a keystore with Java, check out step 7 in this post.