Setting up Continuous Deployment with Lerna and Buildkite

Buildkite is a great tool for running multi-step build and deployment pipelines. Lerna is a tool for managing multiple JavaScript packages within one git repository.

Given that both Lerna and Buildkite are quite popular, it is surprising how difficult it is to set up a basic build and deployment with these tools.

It is very easy to configure a deployment pipeline in Buildkite that will run every time a new commit has been made to a git branch. However, using Lerna, we want to build only those packages in a repository that have actually changed, rather than building all packages in the repository.

Lerna provides some build in tooling for this, chiefly the ls command which will provide us a list of all the packages defined in the monorepo. Using the –since filter with this command, we can easily determine all packages that have changed since the last commit as follows:

lerna ls --json --since HEAD^

Where ^HEAD is the git reference to the commit proceeding HEAD. The --json flag provides us with an output that is a bit easier to parse, for instance using jq.

However, in a CD environment, we usually build not only when a new merge to master has occurred but also when changes to a branch have been submitted. In that instance, we are not interested in the changes which occurred since the last commit but all the changes that have been made in the branch in comparison to the current master branch.

In this instance, the lerna ls command with a --since filter can help us when comparing the current branch with master.

lerna ls --json --since refs/heads/master

Internally, Lerna would run a command such as the following:

git --no-pager diff --name-only refs/heads/master..refs/remotes/origin/$BUILDKITE_BRANCH

This diff goes both ways, so when a file is changed in a package in master only or the branch only, it will cause the package to be listed among the packages to be build. We are only interested in those packages however that are changed in the branch. This can be somewhat assured by running a git pull before the lerna ls command:

git pull --no-edit origin master
lerna ls --json --since refs/heads/master

Unfortunately Buildkite by default does something of a ‘lazy clone’ of the repository. It will only ensure that the branch that is currently being built is checked out with the latest commit and other branches, including master, might be cached from previous builds and are on an old commit. This will prevent the above approach for building branches from working. Thankfully there is an environment variable in Buildkite we can use to force it to get the latest commit for all branches: BUILDKITE_CLEAN_CHECKOUT=true.

Having this list of packages that have changed, we can then trigger pipelines specific to building the changed packages. This can be accomplished using a trigger step.

- trigger: "[name of pipeline for package x]"
  label: ":rocket: Trigger: Build for package x"
  async: false
  build:
    message: "${BUILDKITE_MESSAGE}"
    commit: "${BUILDKITE_COMMIT}"
    branch: "${BUILDKITE_BRANCH}"

Another way to go about deploying with Lerna might be using lerna publish where we could push npm packages to an npm registry and then trigger builds from there. I haven’t tested this way and I think this would require a private npm registry, which the way outlined in this article would not.

If anyone has a more elegant way to go about orchestrating the builds of packages in a Lerna repository, please let everyone know in the comments.

A gentle introduction to vim

vim is a great tool for editing any form of text, be it a blog post, text or programming code. Unfortunately there is a bit of a learning curve when first getting started with vim. In this article I want to provide an overview of some of the key features of vim and how to get started using this editor.

If you are using Linux, it is very easy to use vim. Just type vi in the terminal and vim will be open. In windows, I find the easiest way to use is to install cmder. Once it is installed, just start cmder and type vi just as one would in a Linux system.

Initially when you start up vim you will be in a new empty file and what is called normal mode. In normal mode, it is not possible to enter new text. To do so, we need to switch to insert mode, which is easily done by typing i. You can now write text to your heart’s content. Finally pressing Esc will get us back into normal mode.

To save our work, we must go into command mode, which is achieved from normal mode by prefixing commands with :. :w test.txt followed by enter will save the text we have entered and :q will quit the vim editor.

Switching between these modes may seem a bit cumbersome at the start but splitting up the task of working with text into these different modes allows for very efficient and effective workflows.

In the following I will first describe how each of the individual modes works and then show how we can use operators and text objects to unlock the full potential of vim.

Modes

Normal Mode

Normal mode is the default mode that vim will be in once its started and can usually be enabled by hitting the Esc key. It is the mode in which it is easiest to enable the other modes and also to navigate within a document. Some of the most useful commands for normal mode are the following:

  • j, k: Go one line up or down
  • h, l: Move one character left or right
  • i: Enable insert mode
  • v: Enable visual mode for selecting by character
  • V: Enable visual mode for selecting lines
  • :: Enable command line mode

Insert Mode

Insert mode is closest to how a normal editor works. It is possible to add and delete text. This also renders most keys on the keyboard unavailable for composing commands. So most importantly, we will need to remember the keyboard shortcut of only Esc to get back to normal mode.

Visual Mode

Visual mode is used for selecting text. The most important commands are:

  • j, k: Select line above or below
  • h, l: Select character to the right or left
  • d: Delete selected characters
  • y: Cut selected lines and place in register (press p in normal mode for pasting)
  • >, <: Indent lines to the left or right

Command Line Mode

Command line mode is for system level tasks such as saving or loading files and, importantly, for closing vim. Some important commands (as entered when in normal mode).

  • :wq: Save currently open file and quit
  • :q!: Quit vim and discard any unsaved changes

Operators and Text Objects

Since keys on the keyboard are only used for writing text in insert mode, the keys of the keyboard become available for other operations in the remaining modes. That is great for making complex operations quickly accessible using the keys that are the easiest to type. Since it is not required to use function keys such as Ctrl and Alt, it also becomes very easy to string different commands together. For instance, the following command will delete a word:

daw

Such more complex commands in vim are composed of operators, text objects and motions. In the example above d is an operator for deleting whereas aw denotes the text object of a word including its trailing space. So given that c is the operator for changing we could also write:

caw

This would enable us to change the word at our current cursor position rather than deleting it. Given that p references a paragraph text object, we could also write:

dap

This would delete a whole paragraph; with a paragraph being any block of text proceeded by an empty line.

Here a quick reference of some operators:

  • d: Delete
  • c: Change
  • y: Copy/yank into register

And some text objects:

  • aw: A complete word
  • as: A complete sentence
  • ap: A paragraph
  • i{: Everything within curly braces
  • i": Everything within quotation marks
  • a": The content of the quotation marks plus the quotation marks themselves.
  • /word: Everything up until and including ‘word’

There are usually two flavours of text objects. Those prefixed with a and those with i. The a prefix denotes that we are including whitespace, for instance the training whitespace after a word, whereas the i prefix does not include the whitespace. Therefore, when deleting a superfluous word, we would usually use daw whereas when we want to replace the word (and preserve the whitespace around the word), we would use ciw.

Motions

Motions can be used for two purposes. Without a proceeding operator, a motion can be used to navigate within the text when in normal mode. with a proceeding operator, the operator will be applied by whatever is captured by the motion.

For instance, entering $ in normal mode will get us to the end of a line. If we enter d$, we will delete all text until the end of the line.

Here a list of a few of the motions available in vim:

  • 0, $: Go to the beginning or end of a line
  • j, k: Go one line up or down
  • h, l: Move one character left or right
  • fc: Find the next occurrence of character c
  • Fc: Find the previous occurrence of character c
  • w, b: Go to the next/previous word
  • G: Go to the last line of the document
  • gg: Go to the first line of the document
  • (, ): Sentence backward / forward
  • {, }: Paragraph backward / forward

References and Further Resources

Testing Apollo Client/Server Applications

Following up on the GraphQL, Node.JS and React Monorepo Starter Kit and GraphQL Apollo Starter Kit (Lerna, Node.js), I have now created an extended example which includes facilities to run unit and integration tests using Jest.

The code can be found on GitHub:

apollo-client-server-tests

The following tests are included:

React Component Test

This tests asserts a react component is rendered correctly. Backend data from GraphQL is supplied via a mock [packages/client-components/src/Books/Books.test.js]

import React from 'react';
import Books from './Books';

import renderer from 'react-test-renderer'
import { MockedProvider } from 'react-apollo/test-utils';

import GET_BOOK_TITLES from './graphql/queries/booktitles';

import wait from 'waait';

const mocks = [
  {
    request: {
      query: GET_BOOK_TITLES
    },
    result: {
      data: {
        books: [
          {
            title: 'Harry Potter and the Chamber of Secrets',
            author: 'J.K. Rowling',
          },
          {
            title: 'Jurassic Park',
            author: 'Michael Crichton',
          }
        ]
      },
    },
  },
];

it('Renders one book', async () => {

  const component = renderer.create(<MockedProvider mocks={mocks} addTypename={false}>
    <Books />
  </MockedProvider>);
  expect(component.toJSON()).toEqual('Loading...');

  // to wait for event loop to complete - after which component should be loaded
  await wait(0);

  const pre = component.root.findByType('pre');
  expect(pre.children).toContain('Harry Potter and the Chamber of Secrets');

});

GraphQL Schema Test

Based on the article Extensive GraphQL Testing in 3 minutes, this test verifies the GraphQL schema is defined correctly for running the relevant queries [packages/server-books/src/schema/index.test.js].

import {
    makeExecutableSchema,
    addMockFunctionsToSchema,
    mockServer
} from 'graphql-tools';

import { graphql } from 'graphql';

import booksSchema from './index';

const titleTestCase = {
    id: 'Query Title',
    query: `
      query {
        books {
            title
        }
      }
    `,
    variables: {},
    context: {},
    expected: { data: { books: [{ title: 'Title'} , { title: 'Title' }] } }
};

const cases = [titleTestCase];

describe('Schema', () => {
    const typeDefs = booksSchema;
    const mockSchema = makeExecutableSchema({ typeDefs });

    addMockFunctionsToSchema({
        schema: mockSchema,
        mocks: {
            Boolean: () => false,
            ID: () => '1',
            Int: () => 1,
            Float: () => 1.1,
            String: () => 'Title',
        }
    });

    test('Has valid type definitions', async () => {
        expect(async () => {
            const MockServer = mockServer(typeDefs);

            await MockServer.query(`{ __schema { types { name } } }`);
        }).not.toThrow();
    });

    cases.forEach(obj => {
        const { id, query, variables, context: ctx, expected } = obj;

        test(`Testing Query: ${id}`, async () => {
            return await expect(
                graphql(mockSchema, query, null, { ctx }, variables)
            ).resolves.toEqual(expected);
        });
    });

});

GraphQL Schema and Resolver Test

Extending the previous test as suggested by the article Effective Testing a GraphQL Server, this test affirms that GraphQL schema and resolvers are working correctly [packages/server-books/tests/Books.test.js].

import { makeExecutableSchema } from 'graphql-tools'
import { graphql } from 'graphql'
import resolvers from '../src/resolvers'
import typeDefs from '../src/schema'

const titleTestCase = {
    id: 'Query Title',
    query: `
      query {
        books {
            title
        }
      }
    `,
    variables: {},
    context: {},
    expected: { data: { books: [{ title: 'Harry Potter and the Chamber of Secrets' }, { title: 'Jurassic Park' }] } }
};

describe('Test Cases', () => {

    const cases = [titleTestCase]
    const schema = makeExecutableSchema({ typeDefs: typeDefs, resolvers: { Query: resolvers } })

    cases.forEach(obj => {
        const { id, query, variables, context, expected } = obj

        test(`query: ${id}`, async () => {
            const result = await graphql(schema, query, null, context, variables)
            return expect(result).toEqual(expected)
        })
    })
})

Conclusion

As with the previous two articles on getting started with Apollo etc. the code developed again aims to be as minimalistic as possible. It shows how Apollo client/server code may be tested in three different ways. These are quite exhaustive, even if the presented tests are simplistic.

The only test missing is an integration tests which will test the React component linked to a live Apollo server back-end. I am not sure if it is possible to run an ’embedded’ Apollo server in the browser. Running such a server for testing the React component would also be a good addition.

GraphQL, Node.JS and React Monorepo Starter Kit

Following the GraphQL Apollo Starter Kit (Lerna, Node.js), I wanted to dig deeper into developing a monorepo for a GraphQL/React client-server application.

Unfortunately, things are not as easy as I thought at first. Chiefly the create-react-app template does not appear to work very well with monorepos and local dependencies to other packages.

That’s why I put together a small, simple starter template for developing modular client-server applications using React, GraphQL and Node.js. Here is the code on GitHub:

nodejs-react-monorepo-starter-kit

Some things to note:

  • There are four packages in the project
    • client-main: The React client, based on create-react-app
    • client-components: Contains a definition of the component app. Used by client-main
    • server-main: The Node.js server definition
    • server-books: Contains schema and resolver for GraphQL backend. Used by server-main.
  • Each package defines it’s own package.json and can be built independent of the other packages.
  • The main entry point for the dependent packages (client-components and server-books) is set to dist/index.js. This way, packages which use them, can use the transpiled version created by babel and don’t need to worry about specific JS features used in the dependent packages.

Like GraphQL Apollo Starter Kit (Lerna, Node.js) this starter kit is meant to be very basic to allow easy exploration of the source code.

GraphQL Apollo Starter Kit (Lerna, Node.js)

In many ways developing in Node.js is very fast and lightweight. However it can also be bewildering at times. Mostly so since for everything there seems to be more than one established way of doing things. Moreover, the right way to do something can change within 3 to 6 months, so best practices documented in books and blogs quickly become obsolete.

Some days ago I wanted to create a simple sample project that shows how to develop a simple client/server application using Apollo and React. I thought this would be a simple undertaking. I imagined I would begin with a simple tutorial or ‘starter kit’ and quickly be up and running.

I found that things were not as easy as I imagined, since there are plenty of examples to spin up a simple react app chiefly using create-react-app. I also found the awesome apollo-universal-starter-kit. However, the former lacks insight of how to link this to a Node.js back-end server and the latter is a bit complex and opinionated in some ways as to which frameworks to use.

This motivated me to develop a small, simple ‘starter kit’ for Apollo Client and Server in GraphQL with the following features:

  • Uses ES 2018 ‘vanilla’ JavaScript
  • Uses Lerna to define project with two packages (client and server)
  • Uses Node.js / Express as web server
  • Uses Babel to allow using ES 6 style modules for Node.js
  • Provides easy ways to start a development server and spin up a production instance

Here a link to the project on GitHub:

graphql-apollo-starter-kit

The README provides the instructions to run and build the project.

A few things of note about the implementation:

This project is aimed at developers new to Node.js/React/Apollo. It is not implemented in the most elegant way possible but in a way which makes it easy to understand what is going on by browsing through the code. Be welcome to check out the project and modify it!

Deadlines and Estimates in Agile

Time is a key resource in software development and we need to consider it in two fundamental ways:

(1) Total amount of effort measured by time goes into delivering software.

(2) Deadlines by which the software needs to be completed.

The project management triangle provides the insight that these two elements of time are intertwined. If attempt to complete a project quicker (reduce TIME), we need to exert more effort to do so (increase COST).

Project-triangle-en

But chances are that if you are developing software, you are not too bothered with project management theory but instead are using an agile methodology to organise your work.

However, to some degree deadlines and estimates cannot be avoided. If the costs resulting from this effort outweigh the benefits derived from the software, it is probably a good idea not to develop in the first place. Moreover, often external factors dictate a specific date by which a piece of work needs to be completed.

This article is a discussion of how deadlines and estimates fit within the world of agile. First I will briefly discuss in how far these are integral parts of agile methodologies. Finally I will include with a few recommendations of how best to accommodate the requirements of deadlines and estimate in a post project management world.

While there are many flavours in agile, I will focus my discussions here on Scrum and Kanban, given that these are often considered the two most popular implementation of the agile manifesto.

Deadlines

No one likes deadlines in software development since they are often either missed or lead to inferior software being produced as a deadline approaches.

Scrum is organised around sprints, with each having a start and an end date. The end date effectively functions as a deadline by which a key deliverable (increment) should have been completed. For Kanban, in contrast, deadlines are not part of core framework of how work is organised. It is possible in Kanban to assign due dates to individual tasks but this is not aligned with the primary objective of Kanban – which is to increase to continuous flow of tasks worked on by the team.

Thus Scrum cannot work without deadlines while Kanban might not work very well with them.

Estimates

No one likes estimates in software development since they are often inaccurate.

In agile, it is generally emphasised that estimates are not such a big deal. One way this is done is by assigning funny dimensions to estimates instead of duration in hours or costs in dollars (such as story points or t-shirt sizes). However, estimates are still important and a key part of agile methodologies. In Scrum, for instance, estimates are important for determining the product backlog (which features to implement and which order) and for sprint backlog planning (can all selected tasks be completed in the sprint). In Kanban, estimates are not as essentials as in Scrum. That is, it is possible to practice Kanban without estimates but not Scrum. However, it is still often sensible to provide estimates in Kanban in order to determine the order of work and to measure velocity.

How To Do It Right

We have seen that deadlines and estimates are still important factors to consider when doing agile software development. Granted, if one would like to place less emphasis on these, choosing Kanban over Scrum provides some reprieve. However, often external factors make it impossible to go without deadlines and estimates; and, indeed, there might be some benefits to these in certain circumstances.

Given the many bad experiences we have had with deadlines and estimates, one might be inclined to avoid these whenever possible. I disagree with this. Software is not developed in isolation; it is developed in the context of delivering value to someone and without estimates it is often difficult to assess whether value will actually be delivered. Moreover, often external deadlines cannot be avoided in software development projects – since these are mandated by business needs or other factors.

Thus I think we should overcome our antipathy to estimates and deadlines and, rather than trying to avoid them, we should ask ourselves how we can build them into our processes in the best way.

In this spirit, I provide a number of ideas of how we can implement deadlines and estimates within an agile environment in a way which is most beneficial for all stakeholders involved in the following:

Don’t Reward and Punish

The single most risk I see in deadlines and estimates is that they have the potential to negatively affect the culture of a team or organisation. Deadlines and estimates can be used as a way to exert power from the top to the bottom and reward and punish based on these. This, I think, is very dangerous. If teams must fear that they will be punished if a deadline is not met or an estimate is inaccurate, they will loose confidence and satisfaction in their work. This is simply because they would be rewarded and punished based on a measure which is not completely under their control. For instance, estimating effort for software development tasks is fiendishly difficult if not impossible to do, thus perfectly possible to get wrong. Furthermore, work might often prevented from being completed on time by parties external to the team.

Few and Big

Generally there is a cost involved with every estimate and deadline. The cost of deadlines is additional stress on the team as a whole and team members. The cost of estimates lies in creating and maintaining them but also, just as for deadlines, in the additional stress placed on individuals and teams.

Thus I think it is best to work with big and few estimates rather than with many small ones. For instance, rather than breaking a project up into many small steps, leave it as one deadline and one aggregate estimate (that is visible) – then let the team and individuals break it up into as many smaller tasks (with or without deadlines) as they choose.

Enshrine Team Ownership

In agile, it is usually left to the team to determine their own estimates. However this is an area where management is prone to interfere in the teams turf. For instance by questioning estimates or using them for determining the performance of teams. I think it is critical that teams don’t just get to set their estimates but are constantly reminded that these are their estimates and will not be used against them or without their approval.

Deadlines should likewise be determined in collaboration with the teams rather than being imposed on them. There should be a consensus when a deadline should be set rather than it being delivered top-down.

Track Proactively

Given we work with few and big deadlines and estimates, it is imperative that we are actively keeping track of weather deadlines and estimates are stay within acceptable boundaries. This is to assure that if things do go off the rails, we can react early in a way least disruptive to the work of the team. This most be done with great care though; specifically in a way that asserts the team that enquiries are solely to assist the team rather than to ‘control’ or to dote out punishments.

Use for Feedback Loops

While I believe, as said, that deadlines and estimates should not be used for performance tracking, I do think they can be very useful for building meaningful feedback loops. For instance, a team may track over time how accurate their estimates are or how well the team is tracking to achieve a big deadline from small, internal deadlines. These feedback loops should be kept free from criticism (or praise) – they are just means to keep track and improve ourselves.

Standardise

While team ownership of deadlines and estimates is very important, it is also important to try to standardise things were it doesn’t hurt. For instance, using one measure for estimates across the organisation, shouldn’t interfere with team autonomy too much (such as choosing T shirt sizes or estimates in hours).

Make It Easy

Lastly, I think it should be made easy for teams to work with deadlines and estimates. Ideally there should be easy to use tools which can be used across the organisation.  This aids communication across teams and makes it easier to move from one team to another. Moreover, data for feedback looks might be gathered across teams; always with the caveat that this should never be linked to performance management.

 

 

Mastering Modular JavaScript

Today I was having a look around for best practices for defining JavaScript modules. In that search, I came across the book Mastering Modular JavaScript. This book offers a good selection of best practices for JS module development. Also, all chapters are freely available on GitHub:

For a more basic introduction to modules, see the chapter JavaScript Modules from the book Practical Modern JavaScript.

Everything new in JavaScript since ES6

It is no secret that things in the tech world change rather rapidly. It’s difficult to keep track of everything at the same time. For instance I have been working with JavaScript quite extensively some years ago but recently have been more involved with other tech stacks. Thus I have only followed the developments in the JavaScript world sporadically and was quite surprised by how many things have changed since the days of JavaScript: The Good Parts.

Since before ES6 things have not changed much for a long time, I imagine I am not the only one who could benefit from a little refresher of all the things that have changed since ES6. Thus I have compiled some of the changes I think are most important for ordinary development work. The idea is to provide a quick overview rather than explain every feature in detail – assuming that more information on any of the changes is readily available on the web.

This is not a complete list of everything that has changed. For instance, I included promises but omitted changes made to the way regular expressions work in ECMAScript 2018; since we are likely to come across promises many times per day whereas the changes to regular expressions only affect us in particular edge cases.

ECMAScript 6 / ECMAScript 2015

Variable Scoping

  • let x = 1;: To define block scoped variables

Arrow Functions

  • x => x + 1: Concise closure syntax
  • x => { return x + 1; }: Concise closure syntax
  • this: within lambdas refers to enclosing object (rather than to lambda function itself)

Promises

Promises for wrapping asynchronous code.


let p = new Promise((resolve, reject) => {

   resolve("hello");

});

p.then((msg) => console.log(msg)); 

Executing asynchronous operations in parallel

let parallelOperation = Promise.all([p1, p2]);
parallelOperation.then((data) => {let [res1, res2] = data; } );

Default Parameters and Spread Operator

  • function (x = 1, y = 2): Default values for function parameters
  • function (x, y, ...arr) {}: Capturing all remaining arguments in array for variadic functions
  • var newarr = [ 1, 2, ...oldarr]: ‘Spreading’ of elements from an array as literal elements
  • multiply(1, 2, ...arr): Spreading of elements from an array as individual function parameters

Multiline Strings and Templates

  • `My String⏎NewLine`: Multi-line string literals
  • `Hello ${person.name}`: Intuitive string interpolation
  • const proc = sh`kill -9 ${pid}`;: Tagged template literals for parsing custom languages. The example would result in calling the function sh with the parameters (['kill -9 '], pid)

Object Properties

  • let obj = { x, y }: Property shorthand for defining let object = { x: x, y: y }
  • obj = { func1 (x, y) { } }: Methods allowed as object properties

Deconstructor Assignment

  • var [ x, y, z ] = list: Deconstructing arrays into individual variables by assignment.
  • var [ x=0, y=0 ]: Default values for deconstructing arrays.
  • function( [ x, y ] ): Deconstructing arrays in function calls.
  • var { x, y, z } = getPoint(): Deconstructing objects into individual variables by assignment.
  • var { name: name, address: { street: street }, age: age} = getData(): Deconstructing objects into individual variables by assignment, including nested properties.
  • var p = { x=0, y=0 }: Default values for deconstructing objects.
  • function( { x, y } ): Deconstructing objects in function calls.

Modularity

  • export function add(x,y) { return x + y; }: Exporting functions
  • export var universe = 42;: Exporting variables
  • import { add, universe } from 'lib/module';: Importing functions and variables
  • import * from 'lib/module': Wildcard import
  • export default (x, y) => x + y;: Defining default export
  • import add from 'lib/add': Importing default export
  • import add, { universe } from 'lib/add': Importing default export and additional exports
  • export * from 'lib/module';: Reexporting from other modules

Classes

class keyword for constructing simple classes.

class Point {

  constructor (x, y) {
     this.x = x;
     this.y = y;
  }

  move (deltax, deltay) {
     new Point(this.x + deltax, this.y + deltay);
  }

}

extends keyword for extending classes:


class Car extends Vehicle {

  constructor (name) {
     super(name);
  }

static keyword for static methods


class Math {

  static add(x, y) {
    return x + y;
  }

}

get and set keywords for decorated property access.


class Rectangle {

  get area() { return this.x * this.y }

}

...

new Rectangle(2, 2).area === 4;

Iteration Through Object Values

  • for (let value of arr) { }: for … of loop for going iterating through values of objects.
  • Also note that objects can define their own iterators and generators

Data Structures

  • new Set(): For sets
  • new Map(): For maps
  • new WeakSet(): For sets whose items will be garbage collected when required
  • new WeakMap(): For sets whose items will be garbage collected when required

Symbols

  • Symbol(): For creating an object with a unique identity.
  • Symbol("note"): For creating a unique object with a descriptor.
  • Note: Symbol("node") !== Symbol("node")

ECMAScript 2016

  • **: Exponentiation operator
  • Array.prototypes.includes: Like indexOf but with true/false result and support for NaN

ECMAScript 2017

async/await for more expressive asynchronous operations

async function add1(x) {
  return x + 1;
}

async function add2(x) {
  let y = await add1(x);
  return await add1(y);
}

add2(5).then(console.log);

ECMAScript 2018

Rest/Spread Operators for Object Properties

Collect all not deconstructed properties from an object in another object:


var person = { firstName: "Paul", lastName: "Hendricks", password: "secret"};
var {password, ...sanitisedPerson } = person;
// sanitisedPerson = {firstName: "Paul", lastName: "Hendricks"}

Spread object properties

let details = { firstName: "Paul", lastName: "Hendricks" };

let user = { ...details, password: "secret" };

Finally for Promises

finally callback is guaranteed to be executed if promise succeeds or fails.


async function sayHello() {
console.log("hello");
}
sayHello().then(() => console.log("success") )
.catch((e) => console.log(e))
.finally(() => console.log("runs always")

for await Loop

Special for loops that resolve promises before every iteration.


const promises = [
  new Promise(resolve => resolve(1) ),
  new Promise(resolve => resolve(2) )
];

async function runAll() {
  for await (p of promises) {
    console.log(p);
  }
}

runAll();

References

Image credits: Flickr

Designing Micro Services the Right Way

For a few years now, micro services have been all the rage when it comes to the architecture of large applications. Personally I have always been a bit puzzled about what was so new a great about micro services in comparison to what came before them: Service Oriented Architecture (SOA). Indeed, SOA itself is often portrayed as a frightful antipattern from our past to be mentioned in the same breath as CORBA.

To me, the move from CORBA et all to SOA to Micro Services has not been one of disrupting innovation but one of continuous learning; chiefly in relation to the technologies we employ. It just makes a world of difference setting up a big old monolithic application from the past or an express server in Node.JS (which is also a ‘monolith’ in its own right but just a smaller one – hopefully).

The core problem we are trying to solve has not changed: distributed computing. Unfortunately, one of the first things we learnt about distributed computing seems to have been given less attention recently: that it is best avoided wherever possible. Why? Because it introduces great complexity into an application and can result in many development and operational problems (see YouTube: 10 Tips for failing badly at Microservices by David Schmitz).

One of the most problematic areas is data or persisted state. If the same piece of data needs to be used by multiple services, things become very complicated since it is often required to keep data in sync between multiple places (see YouTube: Managing Data in Microservices by Randy Shoup).

Recently I came across a presentation which I think outlined a very nice approach for dealing with micro services – one that relied heavily on code generation, enforcing common standards and automated testing. Furthermore in the presented architecture one language was used primarily, which I think is a very good approach. I highly recommend viewing this presentation for anyone interested in a way to deal with the complexity of micro services:

YouTube: Design Microservice Architectures the Right Way by Michael Bryzek

What I personally took away from this:

  • Focus on testability. Allow for fast unit and integration tests and even testing with production data. Only code that is easy to test and heavily tested allows for fast and bold development. This organisation for instance automatically updates all their dependencies once per week – automatically, since they have full confidence that their tests will pick up any issues.
  • Utilise code generation. The sad truth of micro services is that we will have to duplicate things, such as commonly used entities – especially if multiple programming languages are involved. Code generation provides an elegant way to deal with this unfortunate situation.
  • Enforce common standards. Although micro services are intended to reduce complexity by dividing up a complex system into small manageable chunks, they can actually result in increased overall complexity, especially if many different technologies are employed. In that case, enforcing strict common standards can help in keeping things simple for developers and ops.
  • Embrace events. Triggering services into action by using events rather than direct API calls can help in making a distributed system more predictable and easier to debug.

I think this presentation provides an excellent overview of best practices for micro services and I couldn’t think of anything to criticise or add. I think it represents the best way of building micro services I am aware of as of now.

I do think, however, there is one important additional issue to consider, and that is that a micro service built according to the best principles and standards will still be a liability if it wasn’t necessary to build a micro service to begin with. This is not so much the question if we should micro services or not (in any organisation of a certain size they are an imperative) but how many.

One of the key drivers of success for micro services within a larger system is to get the boundaries of the services right (see bounded context) and I think we should aim to make micro services as large as possible so we have as few of them as possible; taking into consideration the restrictions of team size, data and complexity:

  • Team: It might sound like heresy but I do think that one ‘physical’ micro service could be maintained by up to three to five teams (and not just one team per micro service). That of course would be the upper maximum, there is nothing wrong with having just one team per micro service. It really depends on what service you are building.
  • Data: And some more heresy: I think that for data it is often better to scale up rather than scale out. Why? Data is all about state and being able to keep state within the physical confines of one systems leads to much improved performance and reduced complexity. Thus we should think about the database management system we will be using for our service and what is the maximum we can scale it up to. Then take 20% of that and ask yourself if your data will stay within that limit. If not, it might be prudent to break the micro service apart or maybe change the DBMS.
  • Complexity: The main drivers of complexity in software are code size, inter-dependencies and heterogeneity. If our micro service would contain large amounts of code with many intricate inter-dependencies that tackles many different problems in different ways, it may be advisable to think about breaking the service up.

As mentioned, distributed systems are inherently more complex that non-distributed ones. Therefore, if we have larger micro services, our system becomes less distributed overall and we hopefully have less accidental complexity to deal with.

Thus, to sum things up, we must be aware of the dangers of micro services and deploy tooling strategically as outlined in the presentation as well as be mindful of how we can build our system in a way that we avoid the complexities of distributed systems as much as possible.