Bundling local dependencies for browser jobs

In Node.js, job.requires(['./local-file']) bundles that file and everything it transitively require()s automatically – see the “Local Modules” section of the basic Node.js tutorial for how that actually works under the hood. That automation is Node-specific: it needs real filesystem and build-tooling access a browser doesn’t have, so a browser job author has no automatic equivalent.

That doesn’t mean a browser job is limited to one file’s worth of code, though – it just means the bundling step that Node does for you invisibly on every deploy is one you run yourself, once, as an ordinary part of your own build. webpack (or Rollup, esbuild, Browserify – any bundler that can target CommonJS) does exactly the job.

Bundle your modules

Using the same charInfo.js/vowels.js pair from the basic tutorial – charInfo.js classifies a character as a vowel, consonant, or punctuation, and itself require()s vowels.js – a minimal webpack config bundles both into one file:

// webpack.config.js
const path = require('path');

module.exports = {
  mode: 'production',
  entry: './charInfo.js',
  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: 'charInfo-bundle.js',
    library: { type: 'commonjs2' },
  },
};

library: { type: 'commonjs2' } is the part that matters: it makes the bundle’s top level do module.exports = <charInfo.js's own exports>, the same shape a single hand-written CommonJS file would have. Run npx webpack, and dist/charInfo-bundle.js is one self-contained file – vowels.js’s contents are in there too, however many local modules or transitive dependencies were pulled in along the way.

Ship it as a job argument

From here, the bundle is just one more single-file module – fetch it as text, eval it, use whatever it exports, the identical pattern the FFTW-WASM web tutorial uses for a compiled module:

async function getCharInfoBundleSource() {
  return fetch('./charInfo-bundle.js').then((r) => r.text());
}

async function workFunction(letter, bundleSourceArg) {
  progress();

  const moduleShim = { exports: {} };
  new Function('module', 'exports', bundleSourceArg)(
    moduleShim,
    moduleShim.exports,
  );
  const charInfo = moduleShim.exports;

  return { upper: letter.toUpperCase(), kind: charInfo(letter) };
}

const bundleSource = await getCharInfoBundleSource();
const job = compute.for(inputSet, workFunction, [bundleSource]);

That’s the whole thing – charInfo('e') returns 'vowel' inside the worker exactly as it would locally, with vowels.js’s contents folded in and never referenced by name anywhere in the browser code.

Note

A bundler pulls in real npm-installed packages the same way it pulls in your own local files – this isn’t limited to hand-written modules. Node’s automatic job.requires() bundling only walks your local require() graph, so a browser job author doing this explicitly can actually end up with more reach than the Node shortcut, not less.

If it’s worth publishing

A bundle built this way is also already most of the way to being a published DCP package: wrap that one output file in bravojs’s module.declare(...) format and it’s ready for job.requires() to pull in on any platform, no job argument needed at all. This is the same “bundle first, wrap once” move described in Building WASM modules for DCP jobs for dependency-heavy libraries – that guide was written with a wasm module in mind, but the technique doesn’t care whether what’s being bundled is wasm-related. If a bundle like this would benefit other DCP job authors, email dan@dcp.dev.