Reify: Meteor’s evolving JavaScript module compiler

How it works and where it’s going next

On the surface, Meteor supports the same ECMAScript import and export syntax as many other web frameworks, bundlers, and build tools. Like most of those projects, we too use Babel to configure and run a set of syntax transformations that turn modern JavaScript into code that works in all browsers. Babel is a fantastic tool, and Meteor relies heavily on it.

However, the code Meteor generates for import and export declarations differs radically from what other tools produce, thanks to a special compiler called Reify that we developed and first shipped in Meteor 1.3.3, back in June 2016. This unique approach has been tested and proven in tens of thousands of Meteor apps over those thirteen months, and we believe the benefits are worth understanding. Even if the code you end up writing looks the same as anyone else’s ECMAScript, it will compile faster, behave more like natively-supported modules, and be easier to debug.

This post begins by explaining some subtle details of how import and export declarations are supposed to behave, then explains how Babel works within difficult constraints to achieve those requirements with its popular babel-plugin-transform-es2015-modules-commonjs plugin, then shows how Reify improves on that developer experience, and concludes with some exciting news about the future of the Reify compiler.

What’s so tricky about import and export?

If you’re not already an expert on the differences between ECMAScript modules and the module systems that came before, such as the CommonJS system used by Node, I would recommend watching this detailed talk that I gave at EmpireNode 2015, in which I explain how import and export build on the successes of CommonJS require and exports, while also solving some of CommonJS’s thorniest problems.

What I want to highlight in this post is the concept of immutable live bindings, a subtle yet critical feature that enables imported symbols to remain up-to-date with export declarations in other modules. In particular, live bindings allow ECMAScript modules to cope gracefully with circular dependencies. However, as you will see later in this post, simulating live bindings faithfully is one of the most challenging parts of implementing a compiler for import and export declarations.

If live bindings are already familiar to you, feel free to skim or skip the rest of this section. If not, keep reading!

When you write an import declaration such as

import increment, { value } from "./some/module";

it’s important to realize that the imported symbols, value and increment, are not ordinary variables. Instead, you should think of them as reflections of variables that were declared in ./some/module, made visible to the current module. That’s what we mean when we say the import declaration creates bindings for value and increment, as opposed to declaring new variables that happen to hold the same values.

This sharing of declarations between files is a concept that did not exist in JavaScript before the specification of modules, so you’re not alone if it seems strange the first time you hear about it.

Here’s how ./some/module might declare and export value and increment:

// Export `value` by name, hence the curly braces to import it.
export let value = 1234;

// Export `increment` as the default export, hence no braces.
export default function increment(by) {
  value += by;
}

We say the imported bindings are live because the exporting module has the ability to change their values at any time, and those changes will be reflected immediately by bindings in other modules.

Suppose we add the following code after the import declaration:

import increment, { value } from "./some/module";

console.log(value); // 1234
increment(1111);
console.log(value); // 2345

In this example, the importing module is indirectly responsible for changing value by calling increment(1111), but that’s possible only because ./some/module exported the increment function. The implementation of increment is still controlled by ./some/module, so it’s only at the whim of ./some/module that value can be updated in this way.

We say the live bindings are immutable because the importing module is forbidden from assigning anything to them directly. In other words, the following code throws an error much like assigning to a const variable:

import { value } from "./some/module";

value += 1111; // Error!

As we saw above, ./some/module can change the value of value, or even export a function like increment that allows other modules to influence value’s value. So value is not exactly constant, since its value might change over time. The binding simply isn’t mutable (that is, able to be mutated) on the importing side.

So there you have it: import declarations create immutable live bindings, which allows imported symbols to remain up-to-date with export declarations in other modules.

If you found this explanation of live bindings confusing or inadequate, here’s a great post by Axel Rauschmayer on the same topic: http://2ality.com/2015/07/es6-module-exports.html

Meteor ❤ Babel

Before getting into the details of how Babel’s module compiler differs from the Reify compiler used by Meteor, I want to make something perfectly clear: this blog post is not about finding fault with Babel, or even with the way Babel compiles modules by default.

Meteor depends heavily on the Babel toolchain, and uses many Babel plugins that have nothing to do with modules. In fact, if you want to get really pedantic about it, we still use the babel-plugin-transform-es2015-modules-commonjs plugin to clean up import declarations inserted by other Babel plugins, especially babel-plugin-transform-runtime.

Triumphant blog posts that pretend to invalidate (or conveniently ignore) their competitor’s technologies are my least favorite part of the JavaScript ecosystem these days. The maintainers of Babel are absolutely not Meteor’s competitors. I myself am a frequent Babel contributor. If you get a boastful tone from this post, then I did a bad job of writing it, or maybe you’ve been primed to expect competition where there doesn’t have to be any.

I hope that I leave you with a renewed appreciation for the problems that Babel has to solve, as well as a working understanding of how Meteor approaches those problems in its own way.

How Babel compiles import and export

By default, if you’re using babel-preset-es2015, Babel will compile import and export syntax using a plugin called babel-plugin-transform-es2015-modules-commonjs. The goal of this plugin is to translate ECMAScript module syntax into code that uses CommonJS require and exports. Unfortunately for Babel, CommonJS is implemented in a wide variety of different JavaScript environments, each with a slightly different flavor: Node may be the biggest driver of CommonJS usage, but Babel also has to generate code that works in bundles built by Browserify, Webpack, FuseBox, Meteor, et cetera, etc, &c.

To get an intuition for how that translation works, let’s revisit our earlier value and increment example:

export let value = 1234;

export default function increment(by) {
  value += by;
}

After compilation with babel-plugin-transform-es2015-modules-commonjs, this code becomes:

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.default = increment;
var value = exports.value = 1234;
function increment(by) {
  exports.value = value += by;
}

Notice how the local variables value and increment have to be copied over to the exports object, and that relationship must be carefully maintained whenever the variables are reassigned.

To see what happens on the importing side, let’s revisit our earlier example:

import increment, { value } from "./some/module";

console.log(value); // 1234
increment(1111);
console.log(value); // 2345

After compilation, this code becomes:

var _module = require("./some/module");
var _module2 = _interopRequireDefault(_module);

function _interopRequireDefault(obj) {
  return obj && obj.__esModule ? obj : { default: obj };
}

console.log(_module.value); // 1234
(0, _module2.default)(1111);
console.log(_module.value); // 2345

Notice how Babel rewrites references to value and increment as _module.value and _module2.default, which allows those references to remain up-to-date with the current properties of the exports object. This rewriting of references is how the Babel-generated code achieves live bindings, since every time you access _module.value you get the latest value of the property from the exports object of ./some/module.

The downside is that the generated code is harder to debug in the Node REPL or your browser’s dev tools, since the value and increment symbols have become properties of _module and _module2. You can sprinkle calls to console.log(value) in the original code, so that Babel can rewrite both the import declaration and the value references in the same compilation pass, but Babel won’t be able to compile console.log(value) correctly at a later time, because the import declaration is no longer available.

Also, if you use the handy as syntax to rename an imported binding—

import { value as aBetterName } from “./some/module";

you won’t find aBetterName mentioned anywhere in the generated code. Instead, it gets rewritten to _module.value, since that’s what ./some/module decided to name it, and that’s just how the rewriting works. Both of these behaviors are gotchas that you can internalize with some effort, but they trip up almost everyone the first few times.

In addition to making debugging harder, this rewriting strategy requires the compiler plugin to traverse the entire abstract syntax tree, since references to the imported symbols could appear anywhere in the module, which makes babel-plugin-transform-es2015-modules-commonjs relatively expensive, as Babel plugins go.

The big challenge that Babel solves

Because babel-plugin-transform-es2015-modules-commonjs strives to generate code that works everywhere, it can’t make any assumptions that aren’t true in all of the environments it targets. In practice, serving the needs of so many different environments has a number of subtle, sometimes unfortunate consequences. To name just a few:

• The CommonJS module object may not be available at all, may not inherit from a common Module.prototype, or may not provide useful information like module.id or module.parent.
• Module identifier strings like "./some/module" may be replaced by numbers (as in Browserify and Webpack), so require(id) and require.resolve(id) won’t work at runtime if the value of id was unknown at build time.
• Babel can’t assume that all modules have been compiled by Babel, so cooperation between the compiler and the runtime is usually out of the question. The exports.__esModule property is one rare exception, born of absolute necessity.
• CommonJS modules that reassign module.exports require shims like the _interopRequireDefault helper we saw above.
• A module that uses export declarations might be imported by other modules using require, so the exports object must always be kept up-to-date with exported local variables.

These constraints restrict the design space within which Babel must operate, so it is hardly obvious that babel-plugin-transform-es2015-modules-commonjs could or should work any other way.

If the complexity of this situation makes you think that JavaScript should have a native module system already, you are not alone! These multi-environment acrobatics are exactly what happens when there is no preexisting specification to rely upon. However, that makes what Babel is doing all the more important, because this careful synthesis is the only way to get all those implementations to adopt the new standard.

How Reify compiles import and export

It’s a testament to the flexibility of Babel that Meteor can simply replace the babel-plugin-transform-es2015-modules-commonjs plugin with our own custom plugin called babel-plugin-transform-es2015-modules-reify. If you ignore the details of how this plugin works, then that’s all there is to it. Hooray for abstractions!

Compared to Babel and babel-plugin-transform-es2015-modules-commonjs, Reify’s job is relatively easy, because Reify can make stronger assumptions about the runtime module system that it’s targeting. Specifically, Reify was designed with Node and Meteor in mind. It can be made to work in other environments, but the decision to target only those implementations of CommonJS has been liberating, to say the least.

The value of these assumptions should become clearer with an example. Let’s start on the importing side. After compilation with Reify,

import increment, { value } from "./some/module";

console.log(value); // 1234
increment(1111);
console.log(value); // 2345

becomes

let increment, value;

module.watch(require("./some/module"), {
  default(v) { increment = v; },
  value(v) { value = v; }
});

console.log(value); // 1234
increment(1111);
console.log(value); // 2345

The module.watch API uses require to import the module, then registers a set of callback functions to be called whenever an export of the given name changes value. Most exports change value only once, when first initialized by the exporting module. However, the first initialization might happen some time after the require call returns, in cases of circular imports.

The module.watch method is defined on a shared Module.prototype object, which exists in Node (conveniently) and Meteor (deliberately), but not in most other CommonJS environments (unfortunately).

That, in a nutshell, is how Reify achieves live bindings: by calling callback functions to update local variables whenever new values are exported.

On the exporting side,

export let value = 1234;

export default function increment(by) {
  value += by;
}

becomes

module.export({
  value: () => value,
  default: () => increment
});

let value = 1234;

function increment(by) {
  module.runSetters(value += by);
}

Here we see two more Module.prototype methods at work: Module.prototype.export and Module.prototype.runSetters.

The module.export API (not to be confused with the CommonJS module.exports object!) registers callback functions that tell the module system how to find the current value of every export. Whenever a module finishes evaluating, or an exported variable gets updated, the generated code calls module.runSetters, which retrieves the latest values of each export, then calls any callbacks registered with module.watch in other modules, reporting any new (or changed) exported values.

Making sure that module.runSetters gets called whenever a module finishes evaluating, even if that module was not compiled by Reify (e.g. built-in Node modules like fs and http), requires some cooperation from the runtime module system. I haven’t found a way to make that work using other bundling tools like Webpack or Browserify, though I have some ideas. For now, it’s definitely easiest if you’re using Node or Meteor.

These three APIs (module.watch, module.export, and module.runSetters) have proven extremely versatile. For example, re-exporting symbols from another module is an easy job for module.watch:

export { a, b as c } from "module";

becomes

module.watch(require("module"), {
  a(v) { exports.a = v; },
  b(v) { exports.c = v; }
});

Reify can even detect changes to exported variables that occur as a result of eval, by simply wrapping eval(...) expressions with module.runSetters:

export let value = 0;

function runCommand(command) {
  return eval(command);
}

runCommand("value = 1234");

becomes

module.export({
  value: () => value
});

let value = 0;

function runCommand(command) {
  return module.runSetters(eval(command));
}

runCommand("value = 1234");

If you’re curious how Reify handles all the different kinds of import and export syntax, your best resource is the README.md.

What’s next

Meteor Development Group, as a company and as a group of people, deeply values collaborative working relationships with other companies and people. Working with others broadens our impact as a small startup, and makes our jobs that much more fun.

For the past several months, Reify has benefitted enormously from the involvement of John-David Dalton, of lodash fame. His questions, ideas, commits, and legendary zeal for performance optimization have made Reify significantly smaller, faster, more standards-compliant, and easier to use outside of Meteor.

In fact, that’s exactly what he was hoping to do. Using the technologies described in this post, JDD has built a module loader for Node 4+ that allows any npm package to use ECMAScript module syntax without an explicit compilation step. Yes, you read that correctly—the Reify compiler is fast enough to run completely on-the-fly, even after npm packages are installed, rather than needing to run in a build step before publishing. Of course the compiled code is cached on disk, and you can even publish your cache to npm if you like, but that’s not strictly necessary.

Why is this so important? It’s important because it means npm packages can finally begin publishing ECMAScript modules directly to npm, without also having to publish a compiled version, and anyone using Node 4 or later will be able to consume those packages without running a build tool. Tools that understand module syntax, such as Rollup, will be able to consume this module code directly, without having to look for alternate entry point modules. If this plan works as well as we hope, the next version of lodash, v5, will be published purely as ECMAScript modules.

John-David Dalton’s @std/esm package differs slightly from Reify in its commitment to standards compliance. Whereas Reify is essentially a way of using ECMAScript module syntax within CommonJS modules, the @std/esm loader goes the extra mile to hide CommonJS variables like require, exports, and module, and Node variables like __dirname and __filename by default. Also, in keeping with the plans of the Node team, you’ll have to give your modules an .mjs file extension, though that behavior can be toggled with an option in package.json. Several other experimental behaviors are configurable, too.

Final thanks

Special thanks are due to Bradley Farias, a member of the Node team who is currently working on the native implementation of ECMAScript modules, for providing lots of technical feedback on this project, which has been vital to keeping the plans for @std/esm aligned with those of the Node team.

Finally, we owe a huge debt to the Meteor community for using the Reify compiler in their applications over the past year, and providing extensive feedback in the form of GitHub issues. Your testing and validation are what allowed this project to grow beyond the immediate Meteor ecosystem, and I expect you will continue to benefit from what we learn out there.

Reify: Meteor’s evolving JavaScript module compiler was originally published in Meteor Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.