Express.js on Lambda Getting Started

AWS Lambda is a cost efficient and easy way to deploy server applications. Express.js is a very popular Node.js framework that makes it very easy to develop REST APIs. This post will go through the basics of deploying an Express.js application to AWS Lambda.

You can also check out the sample project on GitHub.

Develop Express.js Server

We first need to implement our Express.js server. Nothing particular we need to keep in mind here. We can simply define routes etc. as we normally would:

import express from 'express';
import cors from 'cors';
import helmet from 'helmet';
import { rootHandler } from './root';
export const app: express.Application = express();
app.use(helmet());
if (process.env.CORS) {
console.info(`Starting server with CORS domain: ${process.env.CORS}`);
app.use(cors({ origin: process.env.CORS, credentials: true }));
}
app.use(express.json());
app.get('/', rootHandler);
view raw server.ts hosted with ❤ by GitHub

In order to publish this server in a Lambda, we will need to add the aws-serverless-express to our project. Once that is done, we can define a new file lambda.ts with the following content:

require('source-map-support').install();
import awsServerlessExpress from 'aws-serverless-express';
import { app } from './server';
const server = awsServerlessExpress.createServer(app);
exports.handler = (event: any, context: any): any => {
awsServerlessExpress.proxy(server, event, context);
};
view raw lambda.ts hosted with ❤ by GitHub

Note that we are importing the app object from our server.ts file. We have also added the package source-map-support. Initialising this module in our code will result in much easier to read stack traces in the Lambda console (since we will package up our Lambda with Webpack).

Please see all files that are required for the server, including the handler in the sample project.

Package Server

In order to deploy our server to AWS lambda, we need to package it up into a ZIP file. Generally lambda accepts any Node.js application definition in the ZIP file but we will package up our application using Webpack. This will drastically reduce the size of our server, which results in much improved cold start times for our Lambda.

For this, we simply add the webpack package to our project and define a webpack.config.js as follows:

/* eslint-disable @typescript-eslint/no-var-requires */
const path = require('path');
const PnpWebpackPlugin = require('pnp-webpack-plugin');
module.exports = {
entry: './dist/src/lambda.js',
output: {
path: path.resolve(__dirname, 'distLambda'),
filename: 'lambda.js',
libraryTarget: 'umd',
},
target: 'node',
mode: 'production',
devtool: 'source-map',
resolve: {
plugins: [PnpWebpackPlugin],
},
resolveLoader: {
plugins: [PnpWebpackPlugin.moduleLoader(module)],
},
module: {
rules: [
// this is required to load source maps of libraries
{
test: /\.(js|js\.map|map)$/,
enforce: 'pre',
use: [require.resolve('source-map-loader')],
},
],
},
};
view raw webpack.config.js hosted with ❤ by GitHub

Note that we are adding some configuration here to include source maps for easy to read stack traces, as well as load an additional plugin to support Yarn Pnp which is used in the sample project.

Running webpack should result in the following files being generated:

lambda.js should be around 650 kb which includes the whole Express server plus Helmet which we included earlier. Cold-starts for lambda should be sub 1 s with this file size.

Deploy to AWS

Lastly, we need to deploy this Lambda to AWS. For this we first need to define the infrastructure for the Lambda. In the sample project, this is done in Terraform:

resource "aws_lambda_function" "main" {
function_name = var.lambda_name
filename = data.archive_file.empty_lambda.output_path
handler = "lambda.handler"
runtime = "nodejs12.x"
memory_size = 2048
timeout = 900
role = aws_iam_role.lambda_exec.arn
lifecycle {
ignore_changes = [
filename,
]
}
}
view raw lambda.tf hosted with ❤ by GitHub

Important here is handler which should match the file name and main method name for our packaged Node.js application. The filename is lambda.js and we defined an export handler in lambda.ts above. Therefore the handler we need to define for the Lambda is lambda.handler. Also note we set the runtime to nodejs12.x. This ensures that Lambda knows to run our application as a Node.js application. To see how to define a Lambda function manually, see this post.

Note that there is a bit more we need to configure, including an API gateway that will send through HTTP requests to our Lambda. To see all infrastructure definitions, see the AWS Infrastructure definitions in the sample project. One important thing to note is that we need to use a proxy integration in our API gateway. This will ensure that our lambda receives all HTTP calls for the gateway and allow our Express server to do the routing.

resource "aws_api_gateway_integration" "lambda" {
rest_api_id = aws_api_gateway_rest_api.main.id
resource_id = aws_api_gateway_method.proxy.resource_id
http_method = aws_api_gateway_method.proxy.http_method
# Lambdas can only be invoked via Post – but the gateway will also forward GET requests etc.
integration_http_method = "POST"
type = "AWS_PROXY"
uri = aws_lambda_function.main.invoke_arn
}
view raw api-gateway.tf hosted with ❤ by GitHub

Once our Lambda is created, we can simply ZIP up the folder distLambda/ and upload this to AWS using the Lambda console.

In the supplied sample project, we use @goldstack/template-lambda-express to help us with uploading our Lambda. Under the hood, this is using the AWS CLI. You can also use the AWS CLI directly using the update-function-code operation.

Next Steps

This post described a few fundamentals about deploying an AWS Lambda with an Express server. There are actually quite a number of steps involved to get a project working end to end. The easiest way I would recommend for getting a project with an Express server deployed on Lambda off the ground would be to use the tool Goldstack that I have developed, specifically the Express Lambda template. This has also been used to create the sample project for this post.

Otherwise, be welcome to check out the sample project on GitHub and modify it to your need. Note that one thing you will need to do is to update the goldstack.json configuration in packages/lambda-express/goldstack.json. Specifically you will need to change the domain configuration.

{
"$schema": "./schemas/package.schema.json",
"name": "lambda-express",
"template": "lambda-express",
"templateVersion": "0.1.19",
"configuration": {},
"deployments": [
{
"name": "prod",
"configuration": {
"lambdaName": "expressjs-lambda-getting-started",
"apiDomain": "expressjs-lambda.examples.goldstack.party",
"hostedZoneDomain": "goldstack.party"
},
"awsUser": "awsUser",
"awsRegion": "us-west-2"
}
]
}
view raw goldstack.json hosted with ❤ by GitHub

More details about the properties in this configuration can be found here

You will also need to create a config.json in config/infra/aws/config.json with AWS credentials for creating the infrastructure and deploying the Lambda.

{
"users": [
{
"name": "awsUser",
"type": "apiKey",
"config": {
"awsAccessKeyId": "your secret",
"awsSecretAccessKey": "your access key",
"awsDefaultRegion": "us-west-2"
}
}
]
}
view raw config.json hosted with ❤ by GitHub

If you simply use the Goldstack UI to configure your project, all these files will be prepared for you, and you can also easily create a fully configured monorepo that also includes modules for a React application or email sending.

Deploy Next.js to AWS

Next.js is becoming ever more popular these days and rightfully so. It is an extremely powerful and well-made framework that provides some very useful abstractions for developing React applications.

An easy way to deploy Next.js application is through using Vercel. However, it can also be useful to deploy Next.js application into other environments to simplify governance and billing. One popular way to to deploy frontend applications is through using the AWS services S3 and CloudFront.

This article describes how to set up the infrastructure required for running a Next.js application using Terraform on AWS, and some of the gotchas to keep in mind. You can also check out the code on GitHub.

Build Project

There are numerous ways to deploy a Next.js application. In our case, we will need to deploy our application as static website.

For this, we will need to define the following script.

"scripts": {
  "build:next": "next build && next export -o webDist/"
}

Running this script will output a bundled, stand-alone version of the Next.js application that can be deployed on any webserver that can host static files.

Next.js bundle files

S3

We will need an S3 bucket to store the files resulting from the Next.js build process. This essentially is just a public S3 bucket.

Below the Terraform for generating such a bucket using the aws_s3_bucket resource. Note here:

  • The permissions for public read are set by acl = "public-read" but we also need a public read bucket policy that is defined
  • The index_document and error_document correspond to those output in the previous step.
resource "aws_s3_bucket" "website_root" {
  bucket = "${var.website_domain}-root"

  acl = "public-read"

  # Remove this line if you want to prevent accidential deletion of bucket
  force_destroy = true

  website {
    index_document = "index.html"
    error_document = "404.html"
  }

  tags = {
    ManagedBy = "terraform"
    Changed   = formatdate("YYYY-MM-DD hh:mm ZZZ", timestamp())
  }

  policy = <<EOF
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "PublicReadForGetBucketObjects",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::${var.website_domain}-root/*"
    }
  ]
}
EOF

  lifecycle {
    ignore_changes = [tags]
  }
}

Next we need to be able to upload our files to this S3 bucket. There are many ways to do this, for instance using the AWS CLI. I have also created an open source package that provides an integrated package for handling the upload of the files (plus setting up infrastructure through Terraform): template-nextjs. This library will also be used when you choose to create a starter project on Goldstack.

One thing to keep in mind when uploading website resources to S3 is that we want to avoid errors on the user’s end during the deployment process. We for instance do not want to delete the files before we re-upload them, since this may result in a small window in which the files are unavailable for users.

This can be solved by uploading the new files first using the AWS S3 sync command and then uploading them a second time with the --delete flag. The resulting commands will look somewhat like this:

aws s3 sync . s3://[bucketname]/[path]
aws s3 sync . s3://[bucketname]/[path] --delete

Since Next.js in general generates hashed file names, you could also just keep all old files on the bucket. However, Next.js deployments can quickly become large (> 50 MB), so depending on how often you deploy, this could quickly result in a significant amount of unnecessary data stored in your S3 bucket.

See here for reference as well the utility used by the `template-nextjs’ package above: utilsS3Deployment.

CloudFront

While it is possible to use an S3 bucket by itself to host a static website, it is usually preferrable to also use a CloudFront distribution in front of the bucket. This results in significantly faster load times for the user, and also enables us make our website available through a secure https:// link.

There are quite a few moving pieces involved in setting up a CloudFront distribution, so the best reference point here will be the complete Terraform availabe in the example project on GitHub. However, I will put a few excerpts from the Terraform configuration below.

We first need to configure an origin that points to our S3 bucket defined earlier:

  origin {
    domain_name = aws_s3_bucket.website_root.website_endpoint

    origin_id   = "origin-bucket-${aws_s3_bucket.website_root.id}"
    
    custom_origin_config {
      http_port = 80
      https_port = 443
      origin_protocol_policy = "http-only"
      origin_ssl_protocols = ["TLSv1.2"]
    }
  }

And specify a root object that matches the index of our page:

  default_root_object = "index.html"

Then we need to set up the cache settings for CloudFront. A Next.js application consists of some files that should be loaded fresh by the user on every page load (e.g. index.html) but most files should only be downloaded once, and then can be cached safely. These pages can also be cached on CloudFront’s edge locations. This can be achieved in CloudFront using cache behaviours.

  # Priority 0
  ordered_cache_behavior {
    path_pattern     = "_next/static/*"
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD", "OPTIONS"]
    target_origin_id = "origin-bucket-${aws_s3_bucket.website_root.id}"

    forwarded_values {
      query_string = false
      headers      = ["Origin"]

      cookies {
        forward = "none"
      }
    }

    min_ttl                = 0
    default_ttl            = 86400
    max_ttl                = 31536000
    compress               = true
    viewer_protocol_policy = "redirect-to-https"
  }

We can also configure CloudFront to service the correct 404 page provided by Next.js:

  custom_error_response {
    error_caching_min_ttl = 60
    error_code            = 404
    response_page_path    = "/404.html"
    response_code         = 404
  }

Finally, we can link the CloudFront distribution with an AWS provided SSL certificate (free). There is more involved to that than the following – for more information, please have a browse around the source files.

  custom_error_response {
    error_caching_min_ttl = 60
    error_code            = 404
    response_page_path    = "/404.html"
    response_code         = 404
  }

In the example project, we also configure two CloudFront distributions. One to serve the main website, and one to allow forwarding users from an alternate domain. So if a user would go to the domain https://www.yourcompany.com, they can be forwarded to https://yourcompany.com. You can find the configuration for that here redirect.tf.

Route53

Lastly we need to be able to link our CloudFront distribtuion to a domain, and for this we can use the Route 53 service on AWS. This service works both for domains purchased on AWS and for domains purchased through another domain provider.

This can be defined in Terraform easily:

# Creates the DNS record to point on the main CloudFront distribution ID
resource "aws_route53_record" "website_cdn_root_record" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = var.website_domain
  type    = "A"

  alias {
    name                   = aws_cloudfront_distribution.website_cdn_root.domain_name
    zone_id                = aws_cloudfront_distribution.website_cdn_root.hosted_zone_id
    evaluate_target_health = false
  }
}

The example project assumes that a Route 53 hosted zone is already configured for the domain you would like to deploy your application to. You can find instructions how to achieve that in the Goldstack documentation.

Gotchas

When deploying a Next.js application to AWS using the method described in this post, there are a few gotchas we need to keep in mind.

Chiefly, this deployment will not support any Next.js API routes and server-side rendering. There are ways to provide some support for these, and there is a good Serverless module for that. However, things can quickly become complicated, so I would recommend to instead deploy an API separately using Lambdas. Goldstack projects make that easy by allowing to define the Next.js application and lambdas for providing the API in one monorepo.

There are further some issues related to pre-fetching pages in some cases. These can be avoided by not using the Next.js <Link> component.

Final Thoughts

If you are looking for a fully integrated experience for deploying Next.js applications and you do not worry too much about costs and system governance I would recommend deploying to Vercel. You can still use the Goldstack Next.js templates if you are interested in that (see Vercel Deployment).

However, there are some benefits in being able to have your entire infrastructure defined in one cloud provider, and hosting a Next.js website on AWS is very cost effective and provides a good experience for users. Also, AWS has a great reputation for service uptime, especially for the services involved in hosting this solution (S3, CloudFront, Route 53).

Feel free to clone the sample project on GitHub or be welcome to check out Goldstack which provides an easy UI tool to configure your Next.js project (plus link it to additional services such as email sending and data storage on S3).

Overwrite Author in Git History

With every commit, git records the name of the author as well as the committer along with their respective email addresses. These will be public once you push your project to GitHub. So sometimes it may be advisable to change the email addresses of the author and committer for all the past commits in your repository.

This can easily be verified by running git log.

Git keeping track of my email address …

Thankfully it is surprisingly easy to change the email addresses of author and committer in the repository. Simply run the following command in the toplevel of your working tree:

git filter-branch -f --env-filter "GIT_AUTHOR_EMAIL='newemail@site.com' GIT_COMMITTER_EMAIL='newemail@site.com';" HEAD

Finally just do a push.

git push --force

Note that adding --force is important here, since otherwise the changes will be rejected by the remote with the error message:

 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'git@github.com:repo/repo.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Do not do a git pull in that case since that will undo the updating of the author and committer.

If you only want to update the author or committer of some of the commits, you can also use git filter-branch. For instance as follows:

git filter-branch --commit-filter '
      if [ "$GIT_AUTHOR_EMAIL" = "to_update@mail" ];
      then
              GIT_AUTHOR_NAME="New Name";
              GIT_AUTHOR_EMAIL="new@email.com";
              git commit-tree "$@";
      else
              git commit-tree "$@";
      fi' HEAD

Note that it is easy for things to go wrong here with providing the multi-line commit-filter – the easiest way is to put this command into a separate script file.

Next.js with Bootstrap Getting Started

Next.js is an open-source framework for React that aspires to reduce the amount of boilerplate code required for developing React applications. Key features that Next.js provides out of the box are:

  • Routing
  • Code Splitting
  • Server side rendering

I recently developed a small example application with Next.js and came across some minor difficulties when trying to use React Bootstrap within the Next.js application. I will therefore provide a quick guide here how to get Next.js and Bootstrap working together.

Next.js Application with Bootstrap Styling

Thankfully, it is quite easy to get React Bootstrap and Next.js working together once one knows what to do. Essentially this can be accomplished in three steps.

Step 1: Initialise project

We first create a new Next.js project and install the required dependencies:

yarn init
yarn add react bootstrap next @zeit/next-css react-bootstrap react-dom

Then add the scripts to build and deploy Next.js to package.json:

  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start"
  },

Step 2: Customise Next.js Configuration

In order for Next.js to be able to load the Bootstrap CSS for all pages, we need to create a next.config.js file in our project root and provide the following configuration:

const withCSS = require('@zeit/next-css')

module.exports = withCSS({
  cssLoaderOptions: {
    url: false
  }
});

Step 3: Load Bootstrap CSS for All Pages

Next, we need to create the file pages/_app.js. This allows us to define some logic to run for every page that Next.js renders; for all client-side, server-side and static rendering.

We only need to ensure that the Bootstrap CSS is loaded:

// ensure all pages have Bootstrap CSS
import 'bootstrap/dist/css/bootstrap.min.css';

function MyApp({ Component, pageProps }) {

  return
    <Component {...pageProps} />;

}

export default MyApp;

And that’s it! We can now start developing pages and components using Bootstrap styled React components:

import Container from 'react-bootstrap/Container';
import Row from 'react-bootstrap/Row';
import Col from 'react-bootstrap/Col';

function Landing() {
  return <Container>
    <Row>
      <Col>
        <h1>Next.js React Bootstrap</h1>
      </Col>
    </Row>
  </Container>
}

export default Landing;

I have also put together a small project on GitHub that makes use of Next.js and React Bootstrap:

next-js-react-bootstrap

If you are looking for something a bit more comprehensive, please have a look at the following template as well. This includes scripts for deployment into AWS, ESLint/TypeScript configuration and is regularly updated:

Goldstack Next.js + Bootstrap Template

Strapi 2020 Quick Review

I have recently reviewed KeystoneJS for a little project I am planning. I found it overall quite good but lacking in a few aspects, particularly in the way migrations are handled (or not handled). After some research, it seems that Strapi could be a possible alternative to KeystoneJS and so I decided to give this solution a quick review as well.

As I’ve done for KeystoneJS, I will first go through a little example and then conclude with my thoughts.

Getting Started

I started the project by simply applying yarn create:

yarn create strapi-app strapi --quickstart

Since Strapi uses Sqlite for local development, the Strapi server is ready to go after running this command (no database connection configuration is required). I then logged into the administration console and created an admin user and password.

I then went into the Content Type Builder and created a two types/tables. Quote that holds an author and a quote and Tag that holds tag names.

Content Type Builder

Creating these was really straightforward and simple. In the background, Strapi created matching definitions for these in the project directory api/:

api/quote/models/quote.settings.json

{
  "connection": "default",
  "collectionName": "quotes",
  "info": {
    "name": "Quote"
  },
  "options": {
    "increments": true,
    "timestamps": true
  },
  "attributes": {
    "Author": {
      "type": "string"
    },
    "tags": {
      "collection": "tag"
    },
    "Quote": {
      "type": "string"
    }
  }
}

api/tag/models/tag.settings.json

{
  "connection": "default",
  "collectionName": "tags",
  "info": {
    "name": "Tag"
  },
  "options": {
    "increments": true,
    "timestamps": true
  },
  "attributes": {
    "Name": {
      "type": "string",
      "required": true,
      "unique": true
    }
  }
}

The schema defined in these JSON files is dynamically translated into operations modifying the schema of the database Strapi is connected to. Upon deployment to a production system, Strapi will create correct schemas in the attached production database; e.g. for MongoDB or Postgres (see Running Strapi in production and version control sync, Create db schema from models).

I then installed the Graphql plugin. For me, this did not work through the admin web interface (it would just get stuck).

Strapi getting stuck after trying to install GraphQL plugin

I needed to run yarn again after this to fix the project. However, installing using the strapi cli worked without problems:

yarn strapi install graphql

Next I went to the Roles & Permissions plugin to configure public access to the endpoints generated from models:

Setting Permissions in Strapi

It must be noted here that permission settings are not reflected in the source code of the Strapi project. Therefore they are not versioned and cannot easily be deployed to testing and production environments (see #672 Permissions flow between environments)

After the permissions have been set, it is very easy to query the GraphQL API:

GraphQL query against API

I finally developed a little Next.js application that queries the GraphQL API exposed by Strapi. This as simple as hooking up an Apollo Client with the endpoint exposed by Strapi.

const client = new ApolloClient({
  uri: 'http://localhost:1337/graphql',
});

Which then makes it very easy to write dynamic pages with React:

import { useQuery } from '@apollo/react-hooks';
import { gql } from "apollo-boost";

const QUOTES = gql`
  {
    quotes {
      id
      Author
    }
  }
`;

const QuoteList: any = () =&gt; {
  const { loading, error, data } = useQuery(QUOTES);
  if (error) return "Error loading quotes";
  if (loading) return "Loading ...";

  const { quotes } = data;

  return <ul>{
    quotes.map(({ Author, id }) =&gt; {
      return <li>{Author}</li>;
    })
  }
  </ul>
};

export default QuoteList;
Next.js app powered by Stapi Backend

All code I’ve developed for this example is available on GitHub:

https://github.com/mxro/strapi-playground

Quick Review

Based on my experiences building the simple example above and studies of the documentation, my initial impressions of Strapi are:

  • I was very impressed with the speed of development using Strapi. I especially liked the Content Type Builder to quickly design the schema for the data.
  • Strapi provides both a very powerful Restful and GraphQL API for the defined data.
  • In contrast to KeystoneJS, database migrations are handled seamlessly.
  • Strapi feels still a bit rough around the edges, for instance some plugins lack proper descriptions and it crashed on me when trying to install the GraphQL plugin. I probably wouldn’t feel comfortable rolling it out for a mission critical production system.
  • For some reason, permissions are not migrated between environments, they are only stored in the database of the local system. I believe this can make deploying Strapi quite painful.

Overall, I think Strapi is a great technology, and so far it appears the best fit for the small project I am planning. I am especially impressed by the ‘no code’ approach to define the data models.

See also:

5 Things I love about Strapi, a Node.js headless CMS

KeystoneJS 5 Quick Review

I have recently started on a little project to organise the quotes that I have collected in years of reading (see kindle-citation-extractor). I originally got my quotes into Airtable but I quickly hit the limit for the free tier.

I figured that it would be great if I could develop a simple database with a simple user interface. Ideally I would not want to implement the basic CRUD views and so I had a look around for tools that can generate simple UIs for databases. My initial search revealed Keystone and Strapi.

I really liked the looks of KeystoneJS (Version 5) since it appears simple and clean. In this article, I will first document my experiences with the Getting Started example for KeystoneJS and conclude with my first impressions and comparison to similar solutions.

Getting Started

After some browsing around, I decided to follow the getting started guide from the Keystone documentation.

I am particularly interested in running Keystone with Postgres, so to get my local example running, I quickly spun up a Postgres server using Docker:

docker run --name keystone-pg -e POSTGRES_PASSWORD=password -d -v db:/var/lib/postgresql/data -p 5432:5432 postgres

(db-start.sh)

Then I configured the keystone project as per instructions:

yarn create keystone-app  keystone-playground

Provided answers for the prompts:

Prompts for Keystone Project initialisation

Then I connected to the Postgres instance in Docker and created a keystone table:

Create keystone database

And finally run the example:

DATABASE_URL=postgres://postgres:password@localhost:5432/keystone &amp;&amp; yarn dev

Unfortunately, loading the AdminUI then resulted in the following error:

> GraphQL error: select count(*) from "public"."Todo" as "t0" where true – relation "public.Todo" does not exist

There appears to be an open issue for this already: Trouble running starter

I was able to fix this issue by modifying index.js as follows:

...
const keystone = new Keystone({
  name: PROJECT_NAME,
  adapter: new Adapter({
    dropDatabase: true,
    knexOptions: {
      client: 'postgres',
      connection: process.env.DATABASE_URL,
    }
  }),
});
...

Adding the dropDatabase option here seems to force Keystone to create the data in the database upon startup.

Keystone example

The interface on localhost:3000 is also up and running:

Keystone 5 Example To Do list App

Quick Review

Based on looking around the documentation and my experiences with the sample app, my observations for Keystone JS 5 are as follows:

  • KeystoneJS 5 appears very modern, with excellent capabilities for GraphQL
  • Based on my experiences with the Getting Started example, it seems that the documentation for KeystoneJS leaves some things to be desired.
  • I like how lightweight KeystoneJS feels. It runs fast and the code to configure it seems very straightforward and simple.
  • A few lines of declarative code can yield impressive outcomes, such as a fully featured GraphQL API and a nice admin interface.
  • Seems like it is possible to deploy Keystone in Serverless environments, see Serverless deployment using Now.
  • KeystoneJS does not manage migrations when the data model is changed (see this comment). This requires to create any additional lists and fields manually in the database. Here an example how this can be accomplished using Knex migrations.

Potential alternatives for KeystoneJS are:

  • Strapi: Very similar to Keystone but based on a REST API first (GraphQL available as a plugin). Allows creating and editing table schema using the Admin UI. Overall it is more of a CMS that KeystoneJS.
  • Prisma: Prisma is closer to traditional ORM tools than KeystoneJS. The recently released Prisma Admin is similar to the Admin interface of KeystoneJs. Prisma offers a client library whereas KeystoneJS depends on clients interfacing with the data through the GraphQL API.

Overall I still believe that KeystoneJS is a viable technology for my use case. My biggest concern is around migrations; I believe it may be quite difficult to orchestrate this easily across development, test and production system. I will probably continue to poke around a bit more in the KeystoneJS examples and documentation and possibly try out one of the alternatives.

I have uploaded my project resulting from following the Getting Started guide to GitHub. I think it can be quite useful for complementing the existing Getting Started documentation, particularly when wanting to get started using Postgres:

keystone-playground

Knex and Typescript Starter Project

SQL is a very expressive and powerful language. Unfortunately, it has often been difficult to interact with database using SQL from object-oriented languages due to a mismatch of the data structures in the database versus the structures in the application programming language. One common solution to this problem where Object-relational mapping frameworks, which often come with their own issues.

I was most delighted when I started working with Knex, a simple yet versatile framework for connecting and working with relational databases in JavaScript. Using it feels like working with an ORM but it only provides a very thin abstraction layer on top of SQL; this helps avoid many of the pitfalls potentially introduced by ORMs while still providing us with most of their conveniences.

As it turns out, Knex has excellent TypeScript support and I think building applications relying on a database using Knex and TypeScript is an excellent starting point.

I have put together a small project on GitHub that sets up the basics of getting started with Knex and TypeScript. This project specifically focuses on the setting up Knex and TypeScript and no other framework, for instance Express is included.

You can go ahead and clone the project from here:

https://github.com/mxro/knex-typescript-starter-project

After running yarn the following scripts can be run:

  • yarn test: Which will set up Jest in watch mode.
  • yarn build: Which will transpile TypeScript to ES6.
  • yarn watch: Which will run index.ts after every change (and compiles any changes using tsc)

The scripts for defining the database schema are placed in the folder migrations. Here the only currently defined migration:

import Knex from "knex";
import { Migration } from "./../migrationUtil";

export const migrations: Migration[] = [
    {
        name: "000_define_quotes",
        async up(knex: Knex) {
            await knex.schema.createTable("quotes",
                (table) => {
                    table.bigIncrements("id").unsigned().primary();
                    table.uuid("document_id").notNullable();
                    table.uuid("user_id").notNullable();
                    table.timestamp("created", { useTz: true });
                    table.text("quote").notNullable();
                    table.string("author", 512).notNullable();
                    table.string("book", 1024).notNullable();
                    table.text("raw_source").notNullable();
                    table.dateTime("date_collected", { useTz: true });
                    table.string("location", 1024).notNullable();
                    table.string("link", 1024).notNullable();
                    table.index(["document_id"], "document_id_index");
                });

            await knex.schema.createTable("tags",
                (table) => {
                    table.bigIncrements("id").unsigned().primary();
                    table.uuid("tag_id").notNullable();
                    table.timestamp("created", { useTz: true });
                    table.string("name", 512).notNullable();
                    table.uuid("document_id").notNullable();
                });
        },
        /* eslint-disable-next-line  @typescript-eslint/no-empty-function */
        async down(kenx: Knex) {
        },
    }
];

This migration is defined as one of the migrations for this application.

import { migrations as mig001 } from "./migrations/001_define_quotes";
import { runMigration, Migration } from "./migrationUtil";
import Knex from "knex";

export async function runMigrations({ knex }: { knex: Knex }): Promise<void> {

  const migrations: Migration[] = [].concat(mig001);

  await runMigration({ migrations, knex });

}

The migrations are run upon application start up or before tests are run. See the test in migrations.test.ts:

import { runMigrations } from "../src/migrations";
import Knex from "Knex";

describe("Test migrations.", () => {

  it("Should run migrations without error.", async () => {
    const knex = Knex({
      client: "sqlite3",
      connection: { filename: ":memory:" },
      pool: { min: 1, max: 1 },
    });
    await runMigrations({ knex });
    await knex.destroy();
  });

});

Note that this way of running migrations differs a bit from the default way suggested on the Knex website, namely to define migrations in individual files and then run migrations through the Knex CLI. I find this suggested default way a bit cumbersome and I think defining the migrations as native part of the application allows for more flexibility; specifically making it easier to test the application and allowing to develop the application in a more modular way, by allowing us to define migrations per module rather than for the application as a whole (as long as foreign keys are used sparingly).

This is just a very simple starter project. There are other starter projects for TypeScript, such as TypeScript-Node

Textures and Lighting with React and Three.js

In my previous three posts, I have developed a simple WebGL application using react-three-fiber and three.js. In this post, I am adding texture loading and proper lighting to the application.

For reference, here the links to the previous versions of the app:

  • Version 1: Just being able to drag a shape on the screen
  • Version 2: Dragging and dropping shapes using physics
  • Version 3: Being able to move the camera

Here the version developed for this post:

threejs-react-textures-light

Source Code

You can click to add objects, click and drag them as well as move the camera using WASD keys and mouse wheel to zoom.

Loading Textures

Textures can be loaded easily in react-three-fiber using the useResource hook.

All that is required to place the texture in the public/ folder of the react application, load the texture and then link it to the material by setting the map property.

    const [texture] = useLoader(TextureLoader, 'textures/grasslight-big.jpg');

    if (texture) {
        texture.wrapS = texture.wrapT = THREE.RepeatWrapping;
        texture.repeat.set(1500, 1500);
        texture.anisotropy = 16;
    }

    return (
        <mesh ref={ref} receiveShadow position={position}
            onClick={onPlaneClick}>
            <planeBufferGeometry attach="geometry" args={[10000, 10000]} />
            {texture &&
                <meshPhongMaterial attach="material" map={texture} />
            }

        </mesh>
    )

I found that textures are often quite large in size; larger than 1 MB. This significantly extends loading times. Thus I have added a simple loading screen. Unfortunately to be able to display the text ‘loading’ I had to create a TextGeometry which in turn required a font to be loaded (I prepared the Roboto font using facetype.js. This font by itself is more than 300 kb, so even loading the loading screen takes a bit of time.

Lighting

The goal of this application is to have a simple, very large plane on which any number of objects may be added. The issue I encountered with this was that to get shadows working with a DirectionalLight turned out to be very tricky. In the end, I used a combination of an AmbientLight with a SpotLight.

        <ambientLight intensity={0.9} />

        <primitive object={lightTarget} position={lightTargetPosition} />
        <spotLight
            castShadow
            intensity={0.25}
            position={lightPosition}
            angle={Math.PI / 3}
            penumbra={1}
            shadow-mapSize={new Vector2(2048 * 5, 2048 * 5)}
            target={lightTarget}
        />

Since the SpotLight would not be able to cover the whole of the plane (as said, it is meant to be very large) and provide accurate shadows, I opted for moving the SpotLight when a user moves the camera.

    const lightTargetYDelta = 120;
    const lightTargetXDelta = 80;
    const [lightPosition, setLightPosition] = useState([-lightTargetXDelta, -lightTargetYDelta, 200]);
    const [lightTargetPosition, setLightTargetPosition] = useState([0, 0, 0]);
    const onCameraMoved = (delta) => {
        const newLightPosition = delta.map((e, idx) => lightPosition[idx] + e);
        setLightPosition(newLightPosition);
        const newLightTargetPosition = [newLightPosition[0] + lightTargetXDelta, newLightPosition[1] + lightTargetYDelta, 0];
        setLightTargetPosition(newLightTargetPosition);
    };

This required both updating the position of the light setLightPosition as well as moving the light target setLightTargetPosition.

Modularity

Since the amount of code for this example increased quite a bit over the past three iterations, I broke up the application into multiple modules, with most React components now sitting in their own file.

I think this really shows the advantage of using React with Three.js, since it is easy for each component to manage its own state.

For the next iteration, I will most likely be looking at how I can remove the textures or use much smaller textures. I would like the application to be able to load as quickly as possible, and textures clearly do not seem a great option for this.

Camera Movement with Three.js

I have recently been working on small example application using three.js and react-three-fiber. In the first two iterations, I first developed a simple draggable shape floating in space and then supported multiple shapes that can be moved on a physical plane. In this post, I am going to be extending the example to support camera movements. Here links to the previous two iterations and the one developed in this post:

Prototypes

Iteration 3: Camera Movements (this post)

Source Code: threejs-test

Published App: three-js-camera.surge.sh

You can move the camera using WASD and zoom in and out using the mouse scroll wheel. You can create new objects by clicking on any empty spot on the plane.

Iteration 2: Movable objects on Plane

Blog Post: Create and Drag Shapes with Three.js, React and Cannon.js

Published App: react-three-fiber-draggable-v2

Iteration 1: Draggable Shape in Space

Blog Post: Creating a Draggable Shape with React Three Fiber

Published App: react-three-fiber-draggable.surge.sh

In the following I will describe the two ways in which camera movement is supported:

Camera Movements

Using the Keyboard

The easiest way to move the camera using the keyboard would be to change the position of the camera on key press. That is what I first tried and it turned out to be a very insufficient solution. Instead, I stored for how long each key is pressed in an object and then calculate camera movement for every frame.

const keyPressed = {
}

function App() {
    ...
    const handleKeyDown = (e) => {
        if (!keyPressed[e.key]) {
            keyPressed[e.key] = new Date().getTime();
        }
    };

    const handleKeyUp = (e) => {
        delete keyPressed[e.key];
    };
    ...
}

In react-three-fiber, the useful useFrame hook is available where we then calculate the camera movement:

    useFrame((_, delta) => {
        // move camera according to key pressed
        Object.entries(keyPressed).forEach((e) => {
            const [key, start] = e;
            const duration = new Date().getTime() - start;

            // increase momentum if key pressed longer
            let momentum = Math.sqrt(duration + 200) * 0.01 + 0.05;

            // adjust for actual time passed
            momentum = momentum * delta / 0.016;

            // increase momentum if camera higher
            momentum = momentum + camera.position.z * 0.02;

            switch (key) {
                case 'w': camera.translateY(momentum); break;
                case 's': camera.translateY(-momentum); break;
                case 'd': camera.translateX(momentum); break;
                case 'a': camera.translateX(-momentum); break;
                default:
            }
        });
    });

We first calculate the duration how long a key is pressed and then use this to determine the momentum. Finally we use this momentum to update the position of the camera.

Using the Mouse Wheel

We use the mouse wheel to zoom in and out. For this, the position of the camera needs to change along the z axis.

    const mouseWheel = (e) => {
        let delta = e.wheelDelta;
        delta = delta / 240;
        delta = -delta;
        if (delta <= 0) {
            delta -= camera.position.z * 0.1;
        } else {
            delta += camera.position.z * 0.1;
        }
        if (camera.position.z + delta > 1 && camera.position.z + delta < 200) {
            camera.translateZ(delta);
        }
    };

Here we simply determine the direction in which the wheel is scrolled and adjust the position of the camera to be higher or lower accordingly. There is a certain range in which the camera is permitted. If it gets to close to the ground or too far away from it, its position is not changed anymore even if the mouse wheel is turned.

I further had to change the way the drag movement was implemented. This originally worked only using the position of the mouse on the screen, since the screen never moved.

In case of an dynamic camera, we need a bit more calculation to be done which I encapsulated in the get3DPosition method.

const get3DPosition = ({ screenX, screenY, camera }) => {
    var vector = new THREE.Vector3(screenX, screenY, 0.5);
    vector.unproject(camera);
    var dir = vector.sub(camera.position).normalize();
    var distance = - camera.position.z / dir.z;
    var pos = camera.position.clone().add(dir.multiplyScalar(distance));
    return [pos.x, pos.y, 0];
};

The major limitation I want to work on next is the lightning. There is currently a SpotLight light source that only covers a small part of the plane and objects which are not within the cone of this spotlight are not rendered in an aesthetically pleasing fashion.

The full source code is available on GitHub.

Create and Drag Shapes with Three.js, React and Cannon.js

Following up from my article published a few days ago, I have now extended and improved the simple WebGL application that I originally developed using Three.js and react-three-fiber.

Version 1 of the application allowed dragging a simple shape around on the screen:

App: https://react-three-fiber-draggable.surge.sh/

Source Code: https://github.com/mxro/threejs-test/tree/master/test1

Version 2 combines this basic premise with the cannon.js physics engine. Multiple objects can now be created and they drop down onto a solid plane, on which they can be moved.

App: https://react-three-fiber-draggable-v2.surge.sh/

Source Code: https://github.com/mxro/threejs-test/tree/master/test2

Simply click the canvas to add new shapes that then can be dragged around the plane.

The most important logic for this solution is in the DraggableDodecahedron component:

function DraggableDodecahedron({ position: initialPosition }) {
    const { size, viewport } = useThree();
    const [position, setPosition] = useState(initialPosition);
    const [quaternion, setQuaternion] = useState([0, 0, 0, 0]);
    const aspect = size.width / viewport.width;

    const { ref, body } = useCannon({ bodyProps: { mass: 100000 } }, body => {
        body.addShape(new CANNON.Box(new CANNON.Vec3(1, 1, 1)))
        body.position.set(...position);
    }, []);

    const bind = useDrag(({ offset: [,], xy: [x, y], first, last }) => {
        if (first) {
            body.mass = 0;
            body.updateMassProperties();
        } else if (last) {
            body.mass = 10000;
            body.updateMassProperties();
        }
        body.position.set((x - size.width / 2) / aspect, -(y - size.height / 2) / aspect, -0.7);
    }, { pointerEvents: true });

    useFrame(() => {
        // Sync cannon body position with three js
        const deltaX = Math.abs(body.position.x - position[0]);
        const deltaY = Math.abs(body.position.y - position[1]);
        const deltaZ = Math.abs(body.position.z - position[2]);
        if (deltaX > 0.001 || deltaY > 0.001 || deltaZ > 0.001) {
            setPosition(body.position.clone().toArray());
        }
        const bodyQuaternion = body.quaternion.toArray();
        const quaternionDelta = bodyQuaternion.map((n, idx) => Math.abs(n - quaternion[idx]))
            .reduce((acc, curr) => acc + curr);
        if (quaternionDelta > 0.01) {
            setQuaternion(body.quaternion.toArray());
        }
    });
    return (
        <mesh ref={ref} castShadow position={position} quaternion={quaternion} {...bind()}
            onClick={e => {
                e.stopPropagation();
            }}
        >

            <dodecahedronBufferGeometry attach="geometry" />
            <meshLambertMaterial attach="material" color="yellow" />

        </mesh>
    )
}

Most notably here are three React hooks:

With the first hook we create a Cannon body that is set to the same dimension and position as the shape.

     const { ref, body } = useCannon({ bodyProps: { mass: 100000 } }, body => {
        body.addShape(new CANNON.Box(new CANNON.Vec3(1, 1, 1)))
        body.position.set(...position);
    }, []);

In the second hook, we use react-use-gesture to react to drag events. We temporarily set the mass of the body/shape to be moved to 0 and reset it to the original mass once the drag gesture is complete. Finally we also set the position of the cannon.js body to the position that the drag gesture current indicates.

      const bind = useDrag(({ offset: [,], xy: [x, y], first, last }) => {
        if (first) {
            body.mass = 0;
            body.updateMassProperties();
        } else if (last) {
            body.mass = 10000;
            body.updateMassProperties();
        }
        body.position.set((x - size.width / 2) / aspect, -(y - size.height / 2) / aspect, -0.7);
    }, { pointerEvents: true });

The third hook, useFrame(), runs a callback function before every frame is rendered (this hook is supplied by react-three-fiber). This is used here to synchronise the position of the body in cannon.js with the three.js shape. Since cannon.js updates positions with a very fine granularity, we first assert that a body has changed its position or orientation by a significant margin. Only if this is the case, we update the shape. This helps React to avoid unnecessary updates of the ‘virtual dom’.

    useFrame(() => {
        // Sync cannon body position with three js
        const deltaX = Math.abs(body.position.x - position[0]);
        const deltaY = Math.abs(body.position.y - position[1]);
        const deltaZ = Math.abs(body.position.z - position[2]);
        if (deltaX > 0.001 || deltaY > 0.001 || deltaZ > 0.001) {
            setPosition(body.position.clone().toArray());
        }
        const bodyQuaternion = body.quaternion.toArray();
        const quaternionDelta = bodyQuaternion.map((n, idx) => Math.abs(n - quaternion[idx]))
            .reduce((acc, curr) => acc + curr);
        if (quaternionDelta > 0.01) {
            setQuaternion(body.quaternion.toArray());
        }
    });

Apart from these hooks there is a simple click handler that stops click events from propagating. This is to prevent the event handler defined for the plane to trigger (this triggers adding a new shape to the scene).

Next I will be adding camera movement to this example.