Publishing a DCP package

This guide is for the Distributive team only, at this time. If you have a package you'd like published to the DCP package manager, contact dan@distributive.network.

Unlock Incorrect password.

Publish a module once to the DCP package manager, and any job – yours or a colleague’s – pulls it in with one line, job.requires(['pkgname/file.js']), instead of re-shipping the module’s code on every dispatch. This page covers the mechanics of publishing itself; see Building WASM modules for DCP jobs for producing a clean module to publish in the first place, and Bundling local dependencies for browser jobs for the job-argument alternative this page’s last section compares against.

When to publish vs. ship as a job argument

	Publish (job.requires)	Ship as a job argument
Setup	Wrap in module.declare(...), write package.dcp, publish once	None – fetch/bundle, pass as an argument
Per-dispatch cost	Nothing extra sent – the sandbox pulls the package, cached per worker	Full module resent on every single dispatch
Reuse across jobs/authors	job.requires(['pkgname/file.js']) – one line, any job, any author	Copy-paste the fetch/bundle/argument-passing code
Best for	Shared/reused modules, larger binaries, multi-team use	One-off/demo jobs, small modules, rapid iteration

Publishing is the right call for anything meant to be reused – a compiled library, a shared utility, anything more than one job or one author will need. For a one-off script or a self-contained demo, shipping as a job argument is simpler and has nothing to publish or version.

Why this is harder than local job.requires('./local-file')

In Node.js, job.requires(['./local-file']) on an unpublished local path tolerates plain CommonJS (module.exports = ...) because DCP silently runs a real webpack build at deploy time to bundle it – this is Node-only machinery (spawns webpack in a child_process), which is why it doesn’t work from a browser job author at all, and why local job.requires() was never actually a lighter-weight mechanism than publishing – it’s publishing, fully automated, on every single deploy.

Publishing doesn’t build anything – it’s a raw file upload. A file destined for publish must already be pre-wrapped in bravojs’s own module format before you ever run the publish command:

module.declare([], function (require, exports, module) {
  // your module's code, using `require`/`exports`/`module` exactly like
  // ordinary CommonJS -- module.declare's factory args ARE require/
  // exports/module, just supplied differently than Node's ambient globals
  exports.someFunction = function () {
    /* ... */
  };
});

The package.dcp manifest

{
  "name": "yourpackagename",
  "version": "1.0.0",
  "files": {
    "local/path/to/file.js": "file.js"
  }
}

• name – lowercase, must not collide with an existing published package.

• version – real semver. A hyphen suffix (1.0.0-1) is a semver pre-release of 1.0.0, lower precedence than the plain version, not higher – republishing under one fails with VERSIONCONFLICT: Module version conflicts with existing version(s). Must deploy with a higher version number., even though that exact string was never published before. Use a real increment (1.0.1, 1.1.0) instead.

• files – keys are the real local files that get uploaded; each value’s basename is what becomes require()-able afterward.

Publishing

node ~/DCP/node_modules/dcp-util/bin/publish package /absolute/path/to/package.dcp

• The manifest path resolves against the shell’s current working directory, not the manifest’s own location – always pass an absolute path.

• No --apiKey needed for this specific command (unlike real job dispatch, which does need one) – dcp-util’s publish never calls identity.set().

• Success looks like published package: yourpackagename@1.2.3 – but don’t trust that version string as confirmation of what actually got uploaded. It echoes an arbitrary key from the response’s versions map (alphabetically first in every case observed here), not necessarily the version you just published. Seeing an old version number in the success message right after publishing a new one isn’t a failure signal by itself – verify by dispatching against the package instead (next section).

Consuming a published package

job.requires(['yourpackagename/file.js']); // packagename/filename.js -- WITH extension

// inside the work function:
const mod = require('file.js'); // bare filename, no ./ prefix

Note both differences from the local-path form: job.requires takes packagename/filename.js (with extension) instead of an extension-less relative path, and require() inside the work function uses the bare filename instead of ./filename.

Verify what you actually published

Publishing succeeding only confirms the manifest/upload was accepted, not that the module works correctly on a real worker. Always dispatch a small, real, cheap job against the published package (not your local unpublished build) as a last step:

async function testWork() {
  const mod = require('file.js');
  return mod.someFunction();
}

const job = compute.for([0], testWork, []);
job.requires(['yourpackagename/file.js']);
const [result] = await job.exec();

Gotchas specific to packages with more than one or two files

Everything above works cleanly for a single-file (or tightly flattened two-file) package – fftw3wasm-v3, cfitsio4wasm, and sofia2wasm all fit this shape, and none of them ever hit what follows. A genuinely multi-file package – many separate module.declare()-wrapped files that require() each other – surfaces two real bugs that stay invisible until you have enough files to trigger them.

job.requires() only automatically discovers ~2 levels of declared dependencies

List only the entry point in job.requires() – reasonable, since the entry’s own module.declare([...deps], ...) array already lists every file it needs – and files beyond the first couple fail on a real worker with Module './SomeFile.js' is not available., even though the entry point itself resolved fine and the identical code runs correctly locally (--local/unpublished job.requires never reveals this, since Node’s webpack bundling papers over it).

Isolated by bisection on a minimal package with only the file count varying: 1 dependency works, 2 works, 3 fails – uniformly, on every entry, regardless of flat vs. nested directory layout. The fix: list every file the package needs directly in job.requires(), not just the entry point:

// Not enough beyond ~2 files:
job.requires(['yourpackagename/entry.js']);

// Required once the package has more than a couple files:
job.requires([
  'yourpackagename/entry.js',
  'yourpackagename/dep1.js',
  'yourpackagename/dep2.js',
  // ...every other file in package.dcp's `files` map, each pkgname/basename.js
]);

module.exports = X (whole-object reassignment) silently breaks

Raw Emscripten UMD output (and some other bundler/build output) ends with a pattern like:

if (typeof exports === 'object' && typeof module === 'object') {
  module.exports = SomeModule;
  module.exports.default = SomeModule;
}

That reassignment breaks inside a module.declare() factory on a real worker – Cannot set properties of undefined (setting 'default') – even though the identical file loads fine under plain Node. Only property assignment on the given exports object is safe:

// Breaks on a real worker:
module.exports = SomeModule;

// Works everywhere:
exports.default = SomeModule;

If you’re flattening third-party UMD/Emscripten output into a bundle rather than hand-writing every file, wrap it in a local, disposable module object first so the reassignment lands there instead of touching the real sandbox-provided module:

module.declare([], function (require, exports, module) {
  var bundleExports = (function () {
    var module = { exports: {} }; // local shim -- NOT the outer module
    var exports = module.exports;
    /* ...raw UMD/Emscripten content, verbatim, unchanged... */
    return module.exports;
  })();
  for (var k in bundleExports) exports[k] = bundleExports[k];
});

Bundler scripts go stale silently

If a package is generated by a build script (flattening/wrapping source into the published form), re-run that script – and republish – every time the underlying source changes, even for edits that look unrelated to publishing. There’s no error or warning when a published package drifts out of sync with its own source; it just sits there, quietly out of date, until someone happens to rebuild and notices the diff.

Large single-file packages (100 MB+)

No size ceiling found up to 129 MB, confirmed by publishing four real model-weight packages (single-file, base64-encoded ONNX weights, callrag-* in CallRAG’s ~/DCP/transformers/scripts/build-model-packages.js):

Size	Publish time
7.6 MB	2.4 s
44.2 MB	11.0 s
100.4 MB	25.8 s
129.4 MB	34.7 s

Roughly linear, no failures, no special handling needed. This is a dramatically more reliable transport than shipping the same data as a job argument at this size – a ~281 MB job-argument dispatch (four of these same models, pre-publishing) reliably hit DCPError: connection closed during the deploy/upload stage, and an isolated ~300 MB job-argument reproduction crashed with a hard V8 out-of-memory error inside dcp-client’s own JSON.stringify, before ever reaching the network. Publishing sidesteps both: each model uploads once, and every dispatch after that just pulls the cached package with no per-dispatch serialization of the large payload at all.

Naming multiple same-shaped packages that might get job.requires()’d together: give each package’s published file a distinct basename, not a generic one like model.js. require() inside a work function takes just the bare filename with no package prefix (require('file.js'), not require('pkgname/file.js')), so the module namespace is flat by basename across a whole job’s job.requires() list, not scoped per-package. Four packages all publishing a file called model.js would collide the moment one job needed more than one of them; whisper-model.js, bge-model.js, etc. don’t.

Re-publishing a new version

Bump version in package.dcp to a real, higher semver value (see the hyphen-suffix gotcha above), then re-run the same publish command with the same name.

A real alternative – shipping a whole library as a job argument

If the multi-file gotchas above prove too painful for a particular library, there’s a proven fallback: fetch the library’s raw source files client-side, ship their text as job arguments, and evaluate them inside the work function via a small hand-rolled CommonJS-like loader (new Function('module', 'exports', 'require', fileSource)) in a manually pre-sorted dependency order – the job-argument approach from the top of this page, applied to a whole library instead of one module. The tradeoff is the usual one: no publish/versioning step, but the full source gets re-sent on every dispatch with no per-worker package caching.