Creating custom Heroku buildpack for Webpack and Ruby on Rails integration

… and check why 5600+ Rails engineers read also this

Creating custom Heroku buildpack for Webpack and Ruby on Rails integration

Heroku and Rails loves each other from long time - and this combo is still widely used both by beginners and experts to build and host their web applications. It’s easy and it’s fast to host an app - and those two factors are very important in early stages of the project’s lifecycle.

In modern web applications backends and frontends are often equally sophisticated - and unfortunately solutions that Sprockets offers by default are suboptimal choice. Using ECMAScript 2015 features, modern modularization tools and keeping track of dependencies are hard to achieve in typical Rails asset pipeline. That’s why modern JavaScript tooling is used more and more often to deliver those features.

In Arkency we use Webpack and Babel.js to manage and compile code written in modern dialects of JavaScript. Apart from configuration and Rails integration problems there is also a problem of deployment and configuring the deploy machinery to wire everything together. In this article I’d like to show you how you can deploy Rails + Webpack combo to Heroku. This is the thing that is expected from us by our clients from time to time.

Assumptions

Of course to deploy Rails together with Heroku we need to have both tools configured and working. In this article the goal is to make Webpack compile the bundle that can be used by Rails to serve this bundle.

In the described configuration Webpack only provides modularization and ES2015 compilation using babel-loader. It doesn’t minify files - the assumption is that Rails’ asset pipeline can do this just fine during precompilation phase.

The assumption is that the stack is configured as follows:

  • The Node.js package.json resides under app/assets within the Rails application root.
  • Source files are stored under app/assets/source.
  • Webpack compiles the bundle into app/assets/javascripts directory and this bundle is required within application manifest (application.js)
  • Webpack, Babel and loaders are installed as devDependencies so they are not installed if Node.js environment is set to production.
  • All JavaScript dependencies of the codebase are installed on the Node.js side as regular dependencies.
  • There is a npm script named build-production that creates the production-ready bundle (by production-ready I mean - deduped one)

So from the Rails point of view there is only another JavaScript file to include - and this JavaScript file is a bundle emitted by Webpack. The rest (serving pages and serving the compiled JavaScript) is done on the Rails side.

That configuration implies that the whole process of bundle compilation can be done in a total separation from Rails - and that will be important later. To understand how to deploy such configuration, you need to understand how Heroku deployment process works - and fortunately it is relatively simple.

Buildpacks

Heroku is using so-called slug compiler to create a version of an app that is suitable for being served in the Heroku dyno infrastructure. This tool is using so-called buildpacks to perform necessary steps to deploy your application. There are many buildpacks for many technologies - like Ruby buildpack for deploying Ruby (and Rails) apps or Node.js buildpack to deploy and serve Node.js-based apps. By default Heroku is guessing which buildpack to use by doing heuristics built into the buildpacks.

In their essence buildpacks are simple bash scripts that are splitted into three parts:

  • detect part checking whether buildpack should be ran to compile this particular application. Heroku uses it to guess which buildpack is used if none is specified explicitly.
  • compile part is responsible for all necessary preparations needed to run the app. In this step dependencies are installed, assets precompiled and necessary caches filled.
  • release part is responsible for running the app after it has been built.

You can see it by yourself - these three bash scripts are always included in buildpacks under bin/ directory.

In the case of described configuration there are two applications in Heroku terms - one is the Ruby app and it is seamlessly prepared for being run on Heroku (no steps needed) and the other is a Node.js app residing under app/assets directory. It would be ideal to use those two buildpacks - Ruby one and Node.js one and call it a day. Unfortnately, it’s not that simple.

The Problem

Node.js buildpack has almost everything that you need - it installs all necessary dependencies, caches them, downloads Node.js and so on. But unfortunately there are things that it can’t do out-of-the-box:

  • Since package.json is not located under the root directory of the app it won’t get detected correctly. Also during compilation build directories will be invalid.
  • Node.js package is unaware of the npm script you’d like to run.
  • By default Node.js buildpack is running in the development Node.js environment so webpack won’t be installed at all.

To address those three issues you’d need to fork and modify heroku Node.js buildpack.

Modifying Node.js Heroku Buildpack

First of all, you need to have Node.js buildpack forked or cloned and published in a repo. It is important because while configuring buildpacks you’d need to provide a Git repository from which buildpack is going to be fetched.

After having source files, it is needed to modify four files:

  • bin/detect to change directory where package.json will be searched.
  • bin/compile to change directories where Node.js will be built and to add your npm runscript to be ran.
  • bin/release to make it a no-op operation.
  • ‘lib/environment.sh’ to change defaults of node.js environment

Let’s start with bin/detect script. It looks like this:

# bin/detect <build-dir>

if [ -f $1/package.json ]; then
  echo "Node.js"
  exit 0
fi

exit 1

Since package.json resides under app/assets of the build directory, you need to change this test:


[ -f $1/package.json ];

To:


[ -f $1/app/assets/package.json ];

After this change everything will work as planned.

The next step would be to change build directory in bin/compile. You need to look after:


BUILD_DIR=${1:-}

And change it to:


BUILD_DIR=$(cd ${1:-}; cd app/assets; pwd)

This way the build directory of your builpack will change to X/app/assets where X is the argument passed to this script. BTW. You can test this script by calling bin/compile <your-app-path> /tmp to see what happens and troubleshoot if any problems arise with using this buildpack. The same with rest of the scripts inside bin/. At the top of the file there is a comment describing the usage of these scripts.

So far, so good. Build directory is set correctly, now you need to run your npm script. After:


header "Building dependencies"
build_dependencies | output "$LOG_FILE"

You need to have:


npm run build-production

So the compilation step is reconfigured correctly. Unfortunately it still doesn’t work. It is because devDependencies are not installed. By default they are not installed if NODE_ENV variable is set to production. It is a default in this buildpack and it needs to be changed.

That’s why lib/environment.sh needs to be modified. You’re interested in the create_default_env() procedure:


create_default_env() {
  export NPM_CONFIG_PRODUCTION=${NPM_CONFIG_PRODUCTION:-true}
  export NPM_CONFIG_LOGLEVEL=${NPM_CONFIG_LOGLEVEL:-error}
  export NODE_MODULES_CACHE=${NODE_MODULES_CACHE:-true}
  export NODE_ENV=${NODE_ENV:-production}
}

It needs to be changed to:

create_default_env() {
  export NPM_CONFIG_PRODUCTION=${NPM_CONFIG_PRODUCTION:-false}
  export NPM_CONFIG_LOGLEVEL=${NPM_CONFIG_LOGLEVEL:-error}
  export NODE_MODULES_CACHE=${NODE_MODULES_CACHE:-true}
  export NODE_ENV=${NODE_ENV:-development}
}

You can also omit this step and set corresponding environment variables using Heroku Toolbelt. Since this Node.js buildpack is used for development tasks I find it sane to change defaults - it is not default way to use Node.js buildpack after all.

The last part is about disabling any ‘releasing’ behaviour in this buildpack. Modify bin/release to be:


exit 0

That’s it. Commit and push your changes - the buildpack is ready to use!

Wiring buildpacks together

Since you want to use multiple buildpacks you need to explicitly tell heroku which buildpacks will be used. By default Heroku guesses it and sets only one buildpack. Here two are used.

To configure buildpacks used by an app you need to use Heroku Toolbelt. There is handy command called heroku buildpacks:add that will be used here.

First of all, you need to add Ruby buildpack. heroku buildpacks:add accepts URL of the repository as an argument. So the command you need to issue in your app directory is:

heroku buildpacks:add https://github.com/heroku/heroku-buildpack-ruby

Then, you need to have your repository URL. You need to add it as the first buildpack that is executed. There is a --index 1 option which does exactly this - setting the buildpack as first to be executed. So the next command you need to issue is:

heroku buildpacks:add <YOUR-REPO-URL> --index 1

Your repo must be in HTTP format, not the SSH one (so not git@github.com:heroku/heroku-buildpack-ruby.git for example, but https://github.com/heroku/heroku-buildpack-ruby). It is also need to be accessible by Heroku.

If you make a mistake during typing those values there is a heroku buildpacks:remove command that accepts URL or index to be removed.

How it works now?

This is a lot of knowledge here. So to make a quick recap let’s enumerate how Heroku will behave now after you issue git push:

  1. The slug compiler will invoke.
  2. Then, the first buildpack from the list will be invoked. This is your buildpack. After compile there will be your bundle compiled and placed in app/assets/javascripts directory.
  3. Then, the second buildpack starts. It is a Ruby buildpack. It installs your Rails app and precompiles assets.
  4. Then, release scripts are invoked. Custom buildpack release does nothing and Ruby buildpack runs Rails app.

So your app should be deployed with all compiled Webpack scripts, just as planned.

Summary

As you can see, integration of Rails and Webpack on Heroku can still be done in a relatively easy way. Unfortunately it is not as straightforward as the typical Rails-only process, but it’s still manageable. I think being able to work with modern JavaScript tooling is worth the effort.

You might also like