Sometimes it feels that it takes more time setting up the infrastructure of an application that actually writing the application code.
The following infographic from gruntwork nicely illustrates this by showing the breath of tasks one needs to undertake to get a production-grade infrastructure running:
Following up on the GraphQL, Node.JS and React Monorepo Starter Kit and GraphQL Apollo Starter Kit (Lerna, Node.js), I have now created an extended example which includes facilities to run unit and integration tests using Jest.
The code can be found on GitHub:
The following tests are included:
This tests asserts a react component is rendered correctly. Backend data from GraphQL is supplied via a mock [packages/client-components/src/Books/Books.test.js]
import React from 'react';
import Books from './Books';
import renderer from 'react-test-renderer'
import { MockedProvider } from 'react-apollo/test-utils';
import GET_BOOK_TITLES from './graphql/queries/booktitles';
import wait from 'waait';
const mocks = [
{
request: {
query: GET_BOOK_TITLES
},
result: {
data: {
books: [
{
title: 'Harry Potter and the Chamber of Secrets',
author: 'J.K. Rowling',
},
{
title: 'Jurassic Park',
author: 'Michael Crichton',
}
]
},
},
},
];
it('Renders one book', async () => {
const component = renderer.create(<MockedProvider mocks={mocks} addTypename={false}>
<Books />
</MockedProvider>);
expect(component.toJSON()).toEqual('Loading...');
// to wait for event loop to complete - after which component should be loaded
await wait(0);
const pre = component.root.findByType('pre');
expect(pre.children).toContain('Harry Potter and the Chamber of Secrets');
});
Based on the article Extensive GraphQL Testing in 3 minutes, this test verifies the GraphQL schema is defined correctly for running the relevant queries [packages/server-books/src/schema/index.test.js].
import {
makeExecutableSchema,
addMockFunctionsToSchema,
mockServer
} from 'graphql-tools';
import { graphql } from 'graphql';
import booksSchema from './index';
const titleTestCase = {
id: 'Query Title',
query: `
query {
books {
title
}
}
`,
variables: {},
context: {},
expected: { data: { books: [{ title: 'Title'} , { title: 'Title' }] } }
};
const cases = [titleTestCase];
describe('Schema', () => {
const typeDefs = booksSchema;
const mockSchema = makeExecutableSchema({ typeDefs });
addMockFunctionsToSchema({
schema: mockSchema,
mocks: {
Boolean: () => false,
ID: () => '1',
Int: () => 1,
Float: () => 1.1,
String: () => 'Title',
}
});
test('Has valid type definitions', async () => {
expect(async () => {
const MockServer = mockServer(typeDefs);
await MockServer.query(`{ __schema { types { name } } }`);
}).not.toThrow();
});
cases.forEach(obj => {
const { id, query, variables, context: ctx, expected } = obj;
test(`Testing Query: ${id}`, async () => {
return await expect(
graphql(mockSchema, query, null, { ctx }, variables)
).resolves.toEqual(expected);
});
});
});
Extending the previous test as suggested by the article Effective Testing a GraphQL Server, this test affirms that GraphQL schema and resolvers are working correctly [packages/server-books/tests/Books.test.js].
import { makeExecutableSchema } from 'graphql-tools'
import { graphql } from 'graphql'
import resolvers from '../src/resolvers'
import typeDefs from '../src/schema'
const titleTestCase = {
id: 'Query Title',
query: `
query {
books {
title
}
}
`,
variables: {},
context: {},
expected: { data: { books: [{ title: 'Harry Potter and the Chamber of Secrets' }, { title: 'Jurassic Park' }] } }
};
describe('Test Cases', () => {
const cases = [titleTestCase]
const schema = makeExecutableSchema({ typeDefs: typeDefs, resolvers: { Query: resolvers } })
cases.forEach(obj => {
const { id, query, variables, context, expected } = obj
test(`query: ${id}`, async () => {
const result = await graphql(schema, query, null, context, variables)
return expect(result).toEqual(expected)
})
})
})
As with the previous two articles on getting started with Apollo etc. the code developed again aims to be as minimalistic as possible. It shows how Apollo client/server code may be tested in three different ways. These are quite exhaustive, even if the presented tests are simplistic.
The only test missing is an integration tests which will test the React component linked to a live Apollo server back-end. I am not sure if it is possible to run an ’embedded’ Apollo server in the browser. Running such a server for testing the React component would also be a good addition.
Following the GraphQL Apollo Starter Kit (Lerna, Node.js), I wanted to dig deeper into developing a monorepo for a GraphQL/React client-server application.
Unfortunately, things are not as easy as I thought at first. Chiefly the create-react-app template does not appear to work very well with monorepos and local dependencies to other packages.
That’s why I put together a small, simple starter template for developing modular client-server applications using React, GraphQL and Node.js. Here is the code on GitHub:
Some things to note:
package.json and can be built independent of the other packages.dist/index.js. This way, packages which use them, can use the transpiled version created by babel and don’t need to worry about specific JS features used in the dependent packages.Like GraphQL Apollo Starter Kit (Lerna, Node.js) this starter kit is meant to be very basic to allow easy exploration of the source code.
In many ways developing in Node.js is very fast and lightweight. However it can also be bewildering at times. Mostly so since for everything there seems to be more than one established way of doing things. Moreover, the right way to do something can change within 3 to 6 months, so best practices documented in books and blogs quickly become obsolete.
Some days ago I wanted to create a simple sample project that shows how to develop a simple client/server application using Apollo and React. I thought this would be a simple undertaking. I imagined I would begin with a simple tutorial or ‘starter kit’ and quickly be up and running.
I found that things were not as easy as I imagined, since there are plenty of examples to spin up a simple react app chiefly using create-react-app. I also found the awesome apollo-universal-starter-kit. However, the former lacks insight of how to link this to a Node.js back-end server and the latter is a bit complex and opinionated in some ways as to which frameworks to use.
This motivated me to develop a small, simple ‘starter kit’ for Apollo Client and Server in GraphQL with the following features:
Here a link to the project on GitHub:
The README provides the instructions to run and build the project.
A few things of note about the implementation:
This project is aimed at developers new to Node.js/React/Apollo. It is not implemented in the most elegant way possible but in a way which makes it easy to understand what is going on by browsing through the code. Be welcome to check out the project and modify it!
Time is a key resource in software development and we need to consider it in two fundamental ways:
(1) Total amount of effort measured by time goes into delivering software.
(2) Deadlines by which the software needs to be completed.
The project management triangle provides the insight that these two elements of time are intertwined. If attempt to complete a project quicker (reduce TIME), we need to exert more effort to do so (increase COST).
But chances are that if you are developing software, you are not too bothered with project management theory but instead are using an agile methodology to organise your work.
However, to some degree deadlines and estimates cannot be avoided. If the costs resulting from this effort outweigh the benefits derived from the software, it is probably a good idea not to develop in the first place. Moreover, often external factors dictate a specific date by which a piece of work needs to be completed.
This article is a discussion of how deadlines and estimates fit within the world of agile. First I will briefly discuss in how far these are integral parts of agile methodologies. Finally I will include with a few recommendations of how best to accommodate the requirements of deadlines and estimate in a post project management world.
While there are many flavours in agile, I will focus my discussions here on Scrum and Kanban, given that these are often considered the two most popular implementation of the agile manifesto.
No one likes deadlines in software development since they are often either missed or lead to inferior software being produced as a deadline approaches.
Scrum is organised around sprints, with each having a start and an end date. The end date effectively functions as a deadline by which a key deliverable (increment) should have been completed. For Kanban, in contrast, deadlines are not part of core framework of how work is organised. It is possible in Kanban to assign due dates to individual tasks but this is not aligned with the primary objective of Kanban – which is to increase to continuous flow of tasks worked on by the team.
Thus Scrum cannot work without deadlines while Kanban might not work very well with them.
No one likes estimates in software development since they are often inaccurate.
In agile, it is generally emphasised that estimates are not such a big deal. One way this is done is by assigning funny dimensions to estimates instead of duration in hours or costs in dollars (such as story points or t-shirt sizes). However, estimates are still important and a key part of agile methodologies. In Scrum, for instance, estimates are important for determining the product backlog (which features to implement and which order) and for sprint backlog planning (can all selected tasks be completed in the sprint). In Kanban, estimates are not as essentials as in Scrum. That is, it is possible to practice Kanban without estimates but not Scrum. However, it is still often sensible to provide estimates in Kanban in order to determine the order of work and to measure velocity.
We have seen that deadlines and estimates are still important factors to consider when doing agile software development. Granted, if one would like to place less emphasis on these, choosing Kanban over Scrum provides some reprieve. However, often external factors make it impossible to go without deadlines and estimates; and, indeed, there might be some benefits to these in certain circumstances.
Given the many bad experiences we have had with deadlines and estimates, one might be inclined to avoid these whenever possible. I disagree with this. Software is not developed in isolation; it is developed in the context of delivering value to someone and without estimates it is often difficult to assess whether value will actually be delivered. Moreover, often external deadlines cannot be avoided in software development projects – since these are mandated by business needs or other factors.
Thus I think we should overcome our antipathy to estimates and deadlines and, rather than trying to avoid them, we should ask ourselves how we can build them into our processes in the best way.
In this spirit, I provide a number of ideas of how we can implement deadlines and estimates within an agile environment in a way which is most beneficial for all stakeholders involved in the following:
The single most risk I see in deadlines and estimates is that they have the potential to negatively affect the culture of a team or organisation. Deadlines and estimates can be used as a way to exert power from the top to the bottom and reward and punish based on these. This, I think, is very dangerous. If teams must fear that they will be punished if a deadline is not met or an estimate is inaccurate, they will loose confidence and satisfaction in their work. This is simply because they would be rewarded and punished based on a measure which is not completely under their control. For instance, estimating effort for software development tasks is fiendishly difficult if not impossible to do, thus perfectly possible to get wrong. Furthermore, work might often prevented from being completed on time by parties external to the team.
Generally there is a cost involved with every estimate and deadline. The cost of deadlines is additional stress on the team as a whole and team members. The cost of estimates lies in creating and maintaining them but also, just as for deadlines, in the additional stress placed on individuals and teams.
Thus I think it is best to work with big and few estimates rather than with many small ones. For instance, rather than breaking a project up into many small steps, leave it as one deadline and one aggregate estimate (that is visible) – then let the team and individuals break it up into as many smaller tasks (with or without deadlines) as they choose.
In agile, it is usually left to the team to determine their own estimates. However this is an area where management is prone to interfere in the teams turf. For instance by questioning estimates or using them for determining the performance of teams. I think it is critical that teams don’t just get to set their estimates but are constantly reminded that these are their estimates and will not be used against them or without their approval.
Deadlines should likewise be determined in collaboration with the teams rather than being imposed on them. There should be a consensus when a deadline should be set rather than it being delivered top-down.
Given we work with few and big deadlines and estimates, it is imperative that we are actively keeping track of weather deadlines and estimates are stay within acceptable boundaries. This is to assure that if things do go off the rails, we can react early in a way least disruptive to the work of the team. This most be done with great care though; specifically in a way that asserts the team that enquiries are solely to assist the team rather than to ‘control’ or to dote out punishments.
While I believe, as said, that deadlines and estimates should not be used for performance tracking, I do think they can be very useful for building meaningful feedback loops. For instance, a team may track over time how accurate their estimates are or how well the team is tracking to achieve a big deadline from small, internal deadlines. These feedback loops should be kept free from criticism (or praise) – they are just means to keep track and improve ourselves.
While team ownership of deadlines and estimates is very important, it is also important to try to standardise things were it doesn’t hurt. For instance, using one measure for estimates across the organisation, shouldn’t interfere with team autonomy too much (such as choosing T shirt sizes or estimates in hours).
Lastly, I think it should be made easy for teams to work with deadlines and estimates. Ideally there should be easy to use tools which can be used across the organisation. This aids communication across teams and makes it easier to move from one team to another. Moreover, data for feedback looks might be gathered across teams; always with the caveat that this should never be linked to performance management.
Today I was having a look around for best practices for defining JavaScript modules. In that search, I came across the book Mastering Modular JavaScript. This book offers a good selection of best practices for JS module development. Also, all chapters are freely available on GitHub:
For a more basic introduction to modules, see the chapter JavaScript Modules from the book Practical Modern JavaScript.
It is no secret that things in the tech world change rather rapidly. It’s difficult to keep track of everything at the same time. For instance I have been working with JavaScript quite extensively some years ago but recently have been more involved with other tech stacks. Thus I have only followed the developments in the JavaScript world sporadically and was quite surprised by how many things have changed since the days of JavaScript: The Good Parts.
Since before ES6 things have not changed much for a long time, I imagine I am not the only one who could benefit from a little refresher of all the things that have changed since ES6. Thus I have compiled some of the changes I think are most important for ordinary development work. The idea is to provide a quick overview rather than explain every feature in detail – assuming that more information on any of the changes is readily available on the web.
This is not a complete list of everything that has changed. For instance, I included promises but omitted changes made to the way regular expressions work in ECMAScript 2018; since we are likely to come across promises many times per day whereas the changes to regular expressions only affect us in particular edge cases.
let x = 1;: To define block scoped variablesx => x + 1: Concise closure syntaxx => { return x + 1; }: Concise closure syntaxthis: within lambdas refers to enclosing object (rather than to lambda function itself)Promises for wrapping asynchronous code.
let p = new Promise((resolve, reject) => {
resolve("hello");
});
p.then((msg) => console.log(msg));
Executing asynchronous operations in parallel
let parallelOperation = Promise.all([p1, p2]);
parallelOperation.then((data) => {let [res1, res2] = data; } );
function (x = 1, y = 2): Default values for function parametersfunction (x, y, ...arr) {}: Capturing all remaining arguments in array for variadic functionsvar newarr = [ 1, 2, ...oldarr]: ‘Spreading’ of elements from an array as literal elementsmultiply(1, 2, ...arr): Spreading of elements from an array as individual function parameters`My String⏎NewLine`: Multi-line string literals`Hello ${person.name}`: Intuitive string interpolationconst proc = sh`kill -9 ${pid}`;: Tagged template literals for parsing custom languages. The example would result in calling the function sh with the parameters (['kill -9 '], pid)let obj = { x, y }: Property shorthand for defining let object = { x: x, y: y }obj = { func1 (x, y) { } }: Methods allowed as object propertiesvar [ x, y, z ] = list: Deconstructing arrays into individual variables by assignment.var [ x=0, y=0 ]: Default values for deconstructing arrays.function( [ x, y ] ): Deconstructing arrays in function calls.var { x, y, z } = getPoint(): Deconstructing objects into individual variables by assignment.var { name: name, address: { street: street }, age: age} = getData(): Deconstructing objects into individual variables by assignment, including nested properties.var p = { x=0, y=0 }: Default values for deconstructing objects.function( { x, y } ): Deconstructing objects in function calls.export function add(x,y) { return x + y; }: Exporting functionsexport var universe = 42;: Exporting variablesimport { add, universe } from 'lib/module';: Importing functions and variablesimport * from 'lib/module': Wildcard importexport default (x, y) => x + y;: Defining default exportimport add from 'lib/add': Importing default exportimport add, { universe } from 'lib/add': Importing default export and additional exportsexport * from 'lib/module';: Reexporting from other modulesclass keyword for constructing simple classes.
class Point {
constructor (x, y) {
this.x = x;
this.y = y;
}
move (deltax, deltay) {
new Point(this.x + deltax, this.y + deltay);
}
}
extends keyword for extending classes:
class Car extends Vehicle {
constructor (name) {
super(name);
}
static keyword for static methods
class Math {
static add(x, y) {
return x + y;
}
}
get and set keywords for decorated property access.
class Rectangle {
get area() { return this.x * this.y }
}
...
new Rectangle(2, 2).area === 4;
for (let value of arr) { }: for … of loop for going iterating through values of objects.new Set(): For setsnew Map(): For mapsnew WeakSet(): For sets whose items will be garbage collected when requirednew WeakMap(): For sets whose items will be garbage collected when requiredSymbol(): For creating an object with a unique identity.Symbol("note"): For creating a unique object with a descriptor.Symbol("node") !== Symbol("node")**: Exponentiation operatorArray.prototypes.includes: Like indexOf but with true/false result and support for NaNasync/await for more expressive asynchronous operations
async function add1(x) {
return x + 1;
}
async function add2(x) {
let y = await add1(x);
return await add1(y);
}
add2(5).then(console.log);
Collect all not deconstructed properties from an object in another object:
var person = { firstName: "Paul", lastName: "Hendricks", password: "secret"};
var {password, ...sanitisedPerson } = person;
// sanitisedPerson = {firstName: "Paul", lastName: "Hendricks"}
Spread object properties
let details = { firstName: "Paul", lastName: "Hendricks" };
let user = { ...details, password: "secret" };
finally callback is guaranteed to be executed if promise succeeds or fails.
async function sayHello() {
console.log("hello");
}
sayHello().then(() => console.log("success") )
.catch((e) => console.log(e))
.finally(() => console.log("runs always")
Special for loops that resolve promises before every iteration.
const promises = [
new Promise(resolve => resolve(1) ),
new Promise(resolve => resolve(2) )
];
async function runAll() {
for await (p of promises) {
console.log(p);
}
}
runAll();
Image credits: Flickr
For a few years now, micro services have been all the rage when it comes to the architecture of large applications. Personally I have always been a bit puzzled about what was so new a great about micro services in comparison to what came before them: Service Oriented Architecture (SOA). Indeed, SOA itself is often portrayed as a frightful antipattern from our past to be mentioned in the same breath as CORBA.
To me, the move from CORBA et all to SOA to Micro Services has not been one of disrupting innovation but one of continuous learning; chiefly in relation to the technologies we employ. It just makes a world of difference setting up a big old monolithic application from the past or an express server in Node.JS (which is also a ‘monolith’ in its own right but just a smaller one – hopefully).
The core problem we are trying to solve has not changed: distributed computing. Unfortunately, one of the first things we learnt about distributed computing seems to have been given less attention recently: that it is best avoided wherever possible. Why? Because it introduces great complexity into an application and can result in many development and operational problems (see YouTube: 10 Tips for failing badly at Microservices by David Schmitz).
One of the most problematic areas is data or persisted state. If the same piece of data needs to be used by multiple services, things become very complicated since it is often required to keep data in sync between multiple places (see YouTube: Managing Data in Microservices by Randy Shoup).
Recently I came across a presentation which I think outlined a very nice approach for dealing with micro services – one that relied heavily on code generation, enforcing common standards and automated testing. Furthermore in the presented architecture one language was used primarily, which I think is a very good approach. I highly recommend viewing this presentation for anyone interested in a way to deal with the complexity of micro services:
YouTube: Design Microservice Architectures the Right Way by Michael Bryzek
What I personally took away from this:
I think this presentation provides an excellent overview of best practices for micro services and I couldn’t think of anything to criticise or add. I think it represents the best way of building micro services I am aware of as of now.
I do think, however, there is one important additional issue to consider, and that is that a micro service built according to the best principles and standards will still be a liability if it wasn’t necessary to build a micro service to begin with. This is not so much the question if we should micro services or not (in any organisation of a certain size they are an imperative) but how many.
One of the key drivers of success for micro services within a larger system is to get the boundaries of the services right (see bounded context) and I think we should aim to make micro services as large as possible so we have as few of them as possible; taking into consideration the restrictions of team size, data and complexity:
As mentioned, distributed systems are inherently more complex that non-distributed ones. Therefore, if we have larger micro services, our system becomes less distributed overall and we hopefully have less accidental complexity to deal with.
Thus, to sum things up, we must be aware of the dangers of micro services and deploy tooling strategically as outlined in the presentation as well as be mindful of how we can build our system in a way that we avoid the complexities of distributed systems as much as possible.
Sometimes it is necessary to move the location of a git repository; be it from one GitHub repo to another or moving a repo from GitHub to Bitbucket. This can be surprisingly tricky since one needs to make sure to include all branches, tags, etc. when copying the data.
Thankfully git magic allows doing this fairly easily. Just run the following commands:
git clone --mirror <old-repo-url> cd <repo-name> git remote add new-origin <new-repo-url> git push new-origin --mirror
That should be it!
Note that if you are copying a GitHub repo you might get lovely messages such as the following. That should be fine and nothing to worry about.
! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref) ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref) ! [remote rejected] refs/pull/100/head -> refs/pull/100/head (deny updating a hidden ref) ! [remote rejected] refs/pull/101/head -> refs/pull/101/head (deny updating a hidden ref) ! [remote rejected] refs/pull/102/head -> refs/pull/102/head (deny updating a hidden ref)
For many files, it is good to keep track of when they were originally created. In theory, every file created on any mainstream operating system should have its ‘creation date’ and ‘last date modified’ recorded as part of its metadata. This mechanisms, however, is often unreliable, especially when using file synchronisation tools such as Dropbox or Google Drive.
Thus it is often handy to prefix file names with the date they were created, such as:
2018 09 30 Letter.pdf
There are some nice tools available for this purpose, for instance Bulk Rename Utility. However, I often find these a bit too complex for what I need.
I have thus developed a little tool – Date Namer – , which does just the one thing I require: To prefix file names with the current date.
Be welcome to download this tool from here:
Upcoming release will be published on this page: Releases. Further, all the source code for this tool is available on GitHub.
For those interested in the implementation details: For this project, I tried using Electron. This allows developing a Desktop application using Node.js. I found this overall quite easy to use. Internally this application will run an instance of Chromium to render the application. The running application takes thus around 50 MB of RAM. I think this is not too bad for this use case. The app performance is very good.
