Automatically Make Snapshots for EC2

A quick Google search reveals that there are quite a few different approaches for automatically creating snapshots for EC2 images (such as herehere and here).

All of these are rather difficult to do.

Thankfully, after some more searching around I found a great way to schedule regular snapshots using AWS CloudWatch.

CloudWatch supports a built-in target for ‘Create a snapshot of an EBS volume’:

target

For details of how this can be set up, see the excellent step-by-step instructions on the CloudWatch Documentation.

Delete All Binary Logs for MySQL

Today I discovered that one of my servers mysteriously ran out of disk space.

I ran the following Linux command to find all the biggest files and folders on the server:

sudo du -a / | sort -n -r | head -n 100

… and found that it was the binary logs used for MySQL replication that were gobbling up all the disk space:

7375152	/
4691636	/var
4324880	/var/lib
4284952	/var/lib/mysql
1079420	/usr
1048588	/var/lib/mysql/mysql-bin.000004
1048584	/var/lib/mysql/mysql-bin.000006
1048584	/var/lib/mysql/mysql-bin.000003
802356	/var/lib/mysql/mysql-bin.000007

Now I first found some advise that using the PURGE BINARY LOGS should be the way to go. That is true if you want to delete the logs without hurting your ongoing MySQL replication.

However, I was just interested in deleting all the binary logs and the way to do that is by logging into your server with a user with SUPER privileges and executing the following command:

RESET SQL;

Now all those pesky ‘mysql-bin.*’ files should have disappeared!

 

Generate md5 Hash for Maven

Maven creates and checks MD5 checksums at various times. For instance, when downloading an artifact from a repository, Maven checks whether the checksum of the downloaded files (e.g. POM, JAR) is correct.

I am uploading Maven artifacts manually to a very simple, file-based repository. This requires me to calculate the Maven checksum for artifacts manually.

What sounds like a simple problem at first actually turned out to be quite tricky (as it often does with Maven? Well, I guess it’s still better than regular expressions!).

After digging around in the project checksum-maven-plugin, I finally figured out how to generate MD5 checksum files in a way that Maven will accept. The key here was to use the Hex class from Bouncy Castle to turn the MD5 digest into a String.

Following the rough-cut code to create a hash file for Maven (extracted from the maven-tools project, class WriteHashes):

public static void writeMd5(final Path baseFile) throws NoSuchAlgorithmException, IOException {
        final FileInputStream fis = new FileInputStream(baseFile.toFile());

        final MessageDigest messageDigest = MessageDigest.getInstance("MD5");
        messageDigest.reset();
        final byte[] buffer = new byte[1024];
        int size = fis.read(buffer, 0, 1024);
        while (size >= 0) {
            messageDigest.update(buffer, 0, size);
            size = fis.read(buffer, 0, 1024);
        }

        final String result = new String(Hex.encode(messageDigest.digest()));

        fis.close();

        final Path md5File = baseFile.getFileSystem().getPath(baseFile.toString() + ".md5");

        FilesJre.wrap(md5File.toFile()).setText(result);
    }

AWS Lambda: Cross-account pass role is not allowed.

Today I came across the following exception while working with the AWS SDK for Amazon Lambda:

com.amazonaws.AmazonServiceException: Cross-account pass role is not allowed. (Service: AWSLambda; Status Code: 403; Error Code: AccessDeniedException; Request ID: xxx)

At first I was a bit puzzled where this exception might come from; but when I found out what the problem was, it seemed to be pretty obvious:

I tried to upload a service to one AWS account while specifying an execution role that belonged to another AWS account.

So that could easily be fixed by providing a role belonging to the correct account!

UPDATE

As mentioned in the comments by rjhintz, if you require the to use the role from another user, you can do so by modifying the policy for the role as follows:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "AWS":[
               "arn:aws:iam::123456789012:user/user1",
               "arn:aws:iam::123456789012:user/user2"
            ],
            "Service":"ec2.amazonaws.com"
         },
         "Action":"sts:AssumeRole"
      }
   ]
}

What is Amazon Flourish (for AWS)

According to a recent article on the New Stack Blog, the Amazon Serverless Team (responsible for instance for Amazon Lambda) is about to release a new open source product called ‘Floruish’.

Currently, there are very few details available on this product. These are some points I could find:

  • It will be a platform to manage components of serverless applications.
  • This includes versioning lambda functions and packaging lambda functions with other components such as database dependencies.
  • It will be open source (under Apache license)

As more details become available, I will update this post.

For now, here are some related resources regarding serverless applications with Amazon:

Bulk Change ACL for Amazon S3

Using bucket policies, it is easy to set ACL settings for all new objects that are uploaded to Amazon S3.

However, I wanted to remove ‘public’ read rights for a whole bunch of objects at the same time and such policies do not apply to objects that are already stored on S3.

I found an easy way to change the ACL settings for many objects at the same time. To bulk change, ACL, do the following:

  • Download the free tool CloudBerry Explorer for Amazon S3
  • Install it
  • In the AWS management console, go to Security Credentials
  • Create a new user ‘s3-super’. Save the access and secret key.
  • Assign the role  ‘AmazonS3FullAccess’ to the user

full_access

  • Start CloudBerry Explorer and connect to your S3 with the access and secret key of the s3-super user
  • Now in this tool navigate to the bucket with the objects you would like to change
  • Select one or more objects for which you want to change the ACL settings in the left-hand column.
  • Click on the button ACL Settings

acl

  • In the dialog that pops up, change the settings to what you like and click OK.

acl_settings

The ACL settings for your objects should now be changed.

 

 

JQuery UI Droppable: Prevent Event Bubbling

JQuery UI Droppable is a great framework for implementing drag and drop features in a web application.

Here I will show two ways how it can be prevented that multiple droppable elements on the same page can receive the same drop events.

If the one droppable is the parent of the other:

In this case, it is sufficient to add the property greedy: true. Easy.

If there is no parent-child relationship between the elements:

This is a bit tricky, since setting the greedy property will only prevent events bubbling up to the parent. If the two elements are independent (but somehow one floats on top of the other), we need to add some extra code to the drop handlers for both elements:

elem.droppable({
 ...
 drop: function( event, ui ) {

   var elementAtPoint = document.elementFromPoint(event.pageX-1, event.pageY-1);
 
   if (!$.contains(elem[0], elementAtPoint)) {
     // not really meant for this element
     return;
   }

   // handle drop for this element

 }
 ...
});

Replace elem with the two respective elements that are droppable.

This code will assure that the event will only be triggered on the element that is visible for the user.