aws

The Ultimate Guide to TypeScript Monorepos

Max Rohde

Max Rohde

— 11 min read
The Ultimate Guide to TypeScript Monorepos

I've written a couple of posts about how to set up JavaScript and TypeScript Monorepos over the past three years (#1, #2, #3, #4, #5, #6, #7), and I kind of thought I had it all figured out - but I didn't.

It turned out that for various reasons it is fiendishly difficult to develop a JavaScript/TypeScript project that is broken up into multiple independent modules. To make this easier, I even created a little website, Goldstack, that generates modular starter projects.

However, I have always been somewhat unsatisfied with my solutions - with them often involving clumsy workarounds and issues that would prevent them to scale up to larger projects. Now I believe I have finally arrived at a solution that has minimal remaining workarounds and works well for smaller and larger projects.

This solution includes:

In this guide, I will briefly go through the challenges and solutions for each one of these.

tl;dr

If you just want to get started with an already fully configured TypeScript monorepo for your convenience, consider using one of the open-source templates on https://goldstack.party/.

Why Monorepo

Before we go into the implementation, I briefly want to give a few situations when a monorepo may be a good choice for setting up a project:

  • For Fullstack Applications: When developing frontend and backend code in the same repository, it becomes easier to create end-to-end integration tests as well as allows defining and using types across the frontend and backend. For more sophisticated use cases, it can also be useful to be able to reuse the same logic on frontend and backend, for instance for validation.
  • For Large Applications: Being able to divide these larger applications into multiple packages increases modularity and can help reduce complexity. Complexity is reduced chiefly by enforcing a hierarchical dependency pattern between modules (npm dependencies cannot be circular) - as opposed to the every file can import any other file free-for-all of a normal JavaScript project.
  • For Serverless Applications: While traditional applications can be bundled up and deployed in one big package that contains all application logic, serverless applications are often deployed as many independent components, for instance as serverless functions. This deployment pattern lends itself well to monorepos, since each independently deployed component can live in its own package while still making it easy to share code between components.

Yarn 2 Workspaces

Yarn 2 workspaces provide a convenient way to manage the packages and dependencies in large JavaScript projects. Yarn workspaces enable to create projects such as the following:

packages/
  localPackageA/
    package.json
    ...
  localPackageB/
    package.json
    ...

Yarn enables to run a simple yarn add [localPackageName] that will add one local package as the dependency of another.

In addition to this, Yarn 2 ('Berry') gets rid of the dreaded node_modules folder that is conventionally used in Node.js to save dependencies locally. Instead, every dependency used by any of the local packages is stored as a zip file in a special .yarn/cache folder.

Dependencies cached by Yarn as zip files.

This is especially useful in a monorepo, since it is likely that multiple local packages use the same dependencies. By declaring these in one central folder, dependencies do not need to be downloaded multiple times.

Unfortunately a few challenges remain in using Yarn 2 workspaces. Chiefly, using this approach will conflict with any packages that depend on reading files directly from their node_modules folder. But there are also issues with ESM modules that are not yet supported in Yarn 2. Note there is a workaround for this by defining a different node linker.

TypeScript Project References

TypeScript project references have chiefly been developed to help address the problem of long compilation times in large TypeScript projects. They allow breaking up a large project into multiple smaller modules that can each be compiled individually. This also allows for developing more modular code.

Essentially, instead of having one tsconfig.json file in our project, we will have multiple ones, one for each module. To use project references, we need to provide a number of configuration parameters for TypeScript.

  • The composite option needs to be enabled. This allows TypeScript to compile only the modules that have changed.
  • The declaration option should be enabled to provide type information across module boundaries.
  • The declarationMap option also should be enabled. This will allow code navigation between projects.
  • Enabling the incremental option will help speed up compilation times by caching compilation results.
  • outDir should be defined in the tsconfig.json of every module, so that the compiler output will be stored for each module seperarely.

In addition, we need to add a references property to our tsconfig.json that defines all modules within the project that this module depends on.

With that, the tsconfig.json of a module in the project may look as follows:

{
    "compilerOptions": {
        "composite": true,
        "declaration": true,
        "declarationMap": true,
        "incremental": true,
        "outDir": "./dist",
    }
    "references": [
        {
            "path": "../path-to-module"
        },
    ]
}

It is also very useful to define a tsconfig.json in the root of your project that defines a reference to all modules in the project. This makes it easy to compile all modules through one command.

Note that when the composite flag is enabled, running the TypeScript compiler should include the --build parameter:

tsc --build

This default setup generally works very well. However, for larger projects, code editors like VSCode may run into performance problems. If that is the case, also enable the option disableSourceOfProjectReferenceRedirect which will prevent the code editor from constantly recompiling dependent modules. Note though that when enabling this option you will need to ensure that TypeScript files are recompiled when they are changed (e.g. by running the TypeScript compiler in watch mode).

The main issue remaining with respect to TypeScript project references is that these need to manually maintained. When using Yarn workspaces, it is easy to infer what the local references should be, however, TypeScript does not do so by default. For this, I wrote a little tool that keeps the TypeScript project references in sync with Yarn workspace dependencies: Update TypeScript Project References for Yarn Workspaces – magically!

ESLint and Prettier

Prettier is a great tool for maintaining consistent formatting in a project. Prettier works quite well for a monorepo. One can simply define a .prettierrc file in the root of the monorepo and run Prettier using that configuration file. It will automatically apply to all packages in the monorepo.

ESLint provides sophisticated analysis of JavaScript or TypeScript sourcecode. Thankfully it can be configured as easy as Prettier for a monorepo. We can define an .eslintrc.json file in the project root and that will apply to all files in the Monorepo.

When installing the Prettier and ESLint extensions for VSCode, formatting and linting will also work within VSCode for any files in the monorepo. Only tweak required to make this work is to configure the Prettier plugin for ESLint (see example .eslintrc.json). Otherwise Prettier and ESLint will get in each other's way and make for a poor editing experience. To make this work, the following two settings will also need to be configured in a .vscode/settings.json configuration (see settings.json):

{
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  },
  "files.eol": "\n",
  "editor.formatOnSave": false
}

Generally Prettier and ESLint work very well within a monorepo. Only potential issue is that running Prettier or ESLint over larger monorepos may take a long time, if there are many files. In that case, one can configure Prettier and ESLint to only run for specific packages in a monorepo, by adding script definitions in package.json of a local package that can reference the Prettier and ESLint configuration in the root of the project.

Jest

Jest is a great tool for running Unit tests within a JavaScript or TypeScript project. Unfortunately, running tests in Jest is often more difficult than one wishes it to be due to the somewhat fragmented nature of the JavaScript ecosystem. For instance, when using TypeScript and/or React, we need to ensure that source files are transpiled into JavaScript before running tests. When using Yarn workspaces, we also need to ensure that Jest is able to resolve local dependencies.

Thankfully using TypeScript and TypeScript project references makes the intricate problem of using Jest easier since we can make use of the excellent ts-jest Jest transformer. All we need to do it point ts-jest to the respective tsconfig.json file for each package (see example jest.config.js). Since we have configured TypeScript to be composite and incremental, we do not need to recompile TypeScript for dependencies of a package we want to test, which significantly reduces the runtime for unit tests. ts-jest will also ensure that any error message will reference the line numbers in the source TypeScript files.

Running one test in 2s - not too bad considering how bad things can get with Jest if it is not configured correctly

Webpack and ESBuild

To use bundling tools for your deployments is critical in a monorepo. Since without efficient bundling, we would need to deploy all code in the repository, even if individual deployments are composed of only some of the source files.

Similar to Jest, it is very easy to use Webpack in a monorepo configured to use TypeScript project references. We can simply use the ts-loader loader, and everything should work automatically.

Likewise it is easy to use esbuild. esbuild supports TypeScript by default and will automatically resolve all local references since we have TypeScript project references configured. Only additional configuration we need to provide is to use the plugin [@yarnpkg/esbuild-plugin-pnp](https://github.com/yarnpkg/berry/tree/master/packages/esbuild-plugin-pnp) so that esbuild can resolve external dependencies from the local Yarn cache. Find below an example script (build.ts) to bundle code for a AWS lambda:

import { build } from 'esbuild';
import { pnpPlugin } from '@yarnpkg/esbuild-plugin-pnp';

build({
  plugins: [pnpPlugin()],
  bundle: true,
  entryPoints: ['src/lambda.ts'],
  external: ['aws-sdk'],
  minify: true,
  format: 'cjs',
  target: 'node12.0',
  sourcemap: true,
  outfile: 'distLambda/lambda.js',
}).catch((e) => {
  console.log('Build not successful', e.message);
  process.exit(1);
});

React/Next.js

Many JavaScript/TypeScript projects will want to include some from of frontend and in the JavaScript ecosystem we unfortunately often need to jump through some additional hoops to make different frameworks/libraries work with each other.

Next.js is a very powerful framework for React development and it is not too difficult to make this framework work in a TypeScript monorepo. Again, thanks to Next.js native support for both Yarn 2 workspaces and TypeScript project references there is not much we need to configure in this monorepo. We can simply define a tsconfig.json that references all local dependencies and Next.js will pick that up automatically.

We need to do one little tweak to our Next.js configuration to make it work with all our local dependencies. For this, we need to configure the plugin next-transpile-modules.

/* eslint-disable @typescript-eslint/no-var-requires */
/* eslint-disable @typescript-eslint/no-unused-vars */
/* eslint-disable @typescript-eslint/explicit-function-return-type */
const withPlugins = require('next-compose-plugins');

const getLocalPackages = require('./scripts/getLocalPackages');

const localPackages = getLocalPackages.getLocalPackages();
const withTM = require('next-transpile-modules')(localPackages);

const nextConfig = {
  webpack: (config, options) => {
    return config;
  },
  eslint: {
    // ESLint managed on the workspace level
    ignoreDuringBuilds: true,
  },
  images: {
    disableStaticImages: true,
  },
};

const config = withPlugins([[withTM()]], nextConfig);

module.exports = config;

next-transpile-modules requires us to provide it with a list of all local dependencies, e.g. ["@myproject/package1", "@myproject/package2"]. Rather than having to maintain this list manually in the [next.config.js](https://github.com/goldstack/goldstack/blob/master/workspaces/templates/packages/app-nextjs-bootstrap/next.config.js), we can easily write a little script that reads out the package's package.json and determine the local dependencies using the Yarn cli.

yarn workspaces list --json

Please find the complete script for this here: getLocalPackages.js.

AWS Lambda

AWS Lambda is well suited to deploy backend application code from a monorepo. In order to develop code for a Lambda function, there are two things to consider: bundling and local testing.

As mentioned above, it is easy to use esbuild to bundle the code from the monorepo. All we need to provide is the pnp plugin for esbuild. For bundling a lambda, we will also want to make sure that we use cjs as format and Node 12 as compilation target.

Find an example complete esbuild configuration here: build.ts.

There are many ways to develop, deploy and test Node.js lambda functions. In my reference template, I provide an example that uses an Express.js server. That is not necessarily the optimal way to deploy lambdas, chiefly because this results in deploying one lambda function that handles multiple routes. The most 'serverless' way to deploy a backend using functions would be to use different functions for different endpoints.

However, using Express.js makes it very easy to deploy and to develop locally, and so I have chosen this option for an initial implementation but hope to improve on that in the future (see #5, #10). To make local testing work for an Express.js-based lambda, we can use the package ts-node-dev. This will enable starting a server locally and automatically reload it on changes to any files in the monorepo (see package.json).

"scripts": {
   "watch": "PORT=3030 CORS=http://localhost:3000 GOLDSTACK_DEPLOYMENT=local ts-node-dev ./src/local.ts"
},

Infrastructure and Deployment

Most solutions presented so far for the JavaScript/TypeScript monorepo have taken advantage of common JavaScript tools, frameworks and libraries. Unfortunately, I was not able to find a framework that met my requirements for setting up infrastructure and deployment. Very important to me was being able to use Terraform, which I believe provides the most 'standard' way to define infrastructure as code. Almost any kind of infrastructure that can be deployed on any of the popular cloud platforms can be defined in Terraform, and there are plenty of examples and documentation available. Alternatives such as the Serverless framework or AWS SAM in comparison more lean towards being special purpose tools. Pulumi is also a great option but I am not yet convinced that the additional magic it provides on top of basic infrastructure definition (which is based on Terraform) is required over vanilla Terraform.

Given this, I implemented a collection of lightweight scripts that allow standing up infrastructure in AWS using Terraform and perform deployments using the AWS CLI or SDK. For instance for deploying a lambda function, one can simply define a number of Terraform files (e.g. see lambda.tf).

resource "aws_lambda_function" "main" {
  function_name = var.lambda_name

  filename = data.archive_file.empty_lambda.output_path

  handler = "lambda.handler"
  runtime = "nodejs12.x"

  memory_size = 2048
  timeout = 900

  role = aws_iam_role.lambda_exec.arn

  lifecycle {
    ignore_changes = [
       filename,
    ]
  }

  environment {
    variables = {
      GOLDSTACK_DEPLOYMENT = var.name
      CORS = var.cors
    }
  }
}

This is accompanied by scripts written in TypeScript that will deploy the lambda using the AWS CLI (templateLambdaExpressDeploy.ts):

  awsCli({
    credentials: await getAWSUser(params.deployment.awsUser),
    region: params.deployment.awsRegion,
    command: `lambda update-function-code --function-name ${readTerraformStateVariable(
      params.deploymentState,
      'lambda_function_name'
    )} --zip-file fileb://${targetArchive}`,
  });

This allows standing up infrastructure and deploying using simple commands such as (see Infrastructure Commands and Deployment in the Goldstack documentation):

yarn infra up prod
yarn deploy prod

Deployments are configured in goldstack.json configuration files that are transformed into Terraform variables for standing up infrastructure and picked up by deployment scripts as required. Here for instance the goldstack.json file for an AWS Lambda.

{
  "$schema": "./schemas/package.schema.json",
  "name": "lambda-express-template",
  "template": "lambda-express",
  "templateVersion": "0.1.0",
  "configuration": {},
  "deployments": [
    {
      "name": "prod",
      "awsRegion": "us-west-2",
      "awsUser": "goldstack-dev",
      "configuration": {
        "lambdaName": "goldstack-test-lambda-express",
        "apiDomain": "express-api.templates.dev.goldstack.party",
        "hostedZoneDomain": "dev.goldstack.party",
        "cors": "https://app-nextjs-bootstrap.templates.dev.goldstack.party"
      },
      "tfStateKey": "lambda-express-template-prod-8e944cec8ad5910f0d3d.tfstate"
    }
  ]
}

Note that the reference template and templates generated by Goldstack can be used without these tools for infrastructure and deployment. Simply do not use the script and replace them with your preferred way to define infrastructure and deploy.

Next Steps

While I mentioned in the beginning of the article that I am relatively happy with the current state of my reference TypeScript monorepo template, I still think there are a couple of things that can be improved. Chiefly I think that Yarn 2 ('Berry') is still not as mature as I would like it to be. Support for ESM for instance would be awesome, the lack of which caused me some problems in trying to make Svelte work within the monorepo. However, I think it is very worthwhile what the Yarn team attempts to achieve with Yarn 2 and I am happy to support it by trying to make it work in the monorepo template.

Another remaining limitation is the need to run the utils-typescript-references tool manually after changing the dependencies between local packages (to keep workspace dependencies and TypeScript project references in sync). I wonder if it maybe possible to write a Yarn plugin to achieve the same (there is already one for TypeScript).

Otherwise I think most improvements can be made with respect to configuring the infrastructure in the template projects (see issues #3, #5, #10). I am also certain that new versions of Jest, Next.js, TypeScript etc. will break the template before long so there will definitely be some ongoing work to keep this template working.

While the monorepo templates generated on the Goldstack site have already been downloaded hundreds of times, there has so far been not much engagement on GitHub. I assume that is because this is a rather big and complicated project and I have probably been not successful at making it easy to contribute to. I will endavour to make this easier in the future and hope that this will encourage more contributions to the project.

Featured Image Credit: Pete Linforth from Pixabay

Read more