Basic DCP Job Tutorial (Node.js)

This tutorial demonstrates how to create and run a basic DCP Job using Node.js. You will learn how to:

Define an input set of data to be processed in parallel.
Write a worker function that operates on each element.
Use a local module inside the work function, via job.requires().
Configure a Job, including public metadata and compute groups.
Attach event listeners to monitor job progress, results, and errors.
Execute the Job and post-process the results.

The example converts a string of lowercase characters into uppercase letters using DCP, giving you a hands-on introduction to distributed computation.

💡 The complete code is available in the Full Code section at the bottom of this page.

A more advanced tutorial will follow, covering topics such as:

Specifying slice payment offers and alternate payment accounts.
Working with remote datasets.
Deploying jobs across multiple compute groups in federated networks.

This tutorial focuses on the core building blocks so you can get a Job running quickly and safely.

Requirements

Node.js (LTS)
npm i dcp-client
id.keystore and default.keystore located in ~/.dcp (see API keys)

DCP Job

Create a JavaScript file called toUpperCase.js and define an asynchronous main function. The program requires the dcp-client module to initialize access to the Compute API.

Initialize DCP and define `main()`

async function main() {
  /* DCP App ... */
}

require('dcp-client').init().then(main);

By default, .init() is equivalent to:

require('dcp-client').init('https://scheduler.distributed.computer');

This connects to the public DCP scheduler. Additional schedulers may be available in the future, allowing job deployers to select a specific target scheduler.

Using `initSync()`

Alternatively:

require('dcp-client').initSync();
/* JOB App ... */

initSync() is primarily intended for quick prototyping or environments where top-level await isn’t available.

Learn more about initializing DCP

Input Set

The input set is an array of enumerable values (for example, integers, strings, objects, images). Each element in the array is treated as an independent input datum and processed in parallel by DCP workers.

In this example, the input set is an array of characters derived from the string "yelling!".

const inputSet = Array.from('yelling!');

Work Function

The work function defines the computation applied to each element of the input set. DCP serializes this function and distributes it to Workers, where it’s executed independently for each input datum in parallel.

Each Worker receives a single input element and returns a corresponding result. Because execution is isolated per input, the work function should be deterministic and self-contained.

In this example, each Worker receives one character, converts it to uppercase, and also classifies it as a vowel, consonant, or punctuation, using a small local module:

async function workFunction(letter) {
  progress();
  const charInfo = require('./charInfo');
  return { upper: letter.toUpperCase(), kind: charInfo(letter) };
}

letter is one element from the input set.
require('./charInfo') pulls in a local module. See Local Modules under Job Configuration below for what that file contains and how DCP gets it onto the Worker that runs this function.
progress() reports task progress back to the scheduler.
The return value becomes one element in the final results array.

When the job completes, the results are returned in the same order as the original input set.

Learn more about `progress()`

progress()

Job

A Job represents the workload to be executed across the DCP network. Each element of the input set is packaged as a Job slice and processed independently by Workers in parallel. Jobs encapsulate the input, the work function, and optional configuration such as compute groups, public metadata, and event handlers.

const job = compute.for(inputSet, workFunction);

At this point, the job object can be further configured before execution (see Job Configuration below).

Learn more about jobs

Job Configuration

After creating a job with compute.for(...), you can configure its behavior and metadata by setting properties on the job object before calling job.exec().

Public information

You may attach optional, publicly visible metadata to your job. This information can be displayed in dashboards or monitoring tools and helps identify the purpose of the job.

job.public = {
  name: 'toUpperCase',
  description: 'Minimal demonstration of a distributed job',
  link: 'https://distributive.network',
};

name: A short identifier for the job
description: A brief explanation of what the job does
link: Optional reference URL for additional context

This metadata doesn’t affect execution behavior; it’s informational only.

Local Modules

The work function above calls require('./charInfo'). That module, and the one it in turn depends on, need to exist alongside toUpperCase.js:

// vowels.js
module.exports = new Set(['a', 'e', 'i', 'o', 'u']);

// charInfo.js
const vowels = require('./vowels');

module.exports = function charInfo(char) {
  if (!/[a-z]/i.test(char)) return 'punctuation';
  return vowels.has(char.toLowerCase()) ? 'vowel' : 'consonant';
};

A Worker has no access to your local filesystem, so a plain require('./charInfo') would fail once the work function actually runs on one. job.requires() is what makes it resolve:

job.requires(['./charInfo']);

Notice that only ./charInfo is listed, not ./vowels too. job.requires() only needs the modules your work function requires directly – charInfo.js’s own require('./vowels') is a transitive dependency, and DCP walks that dependency graph automatically, the same way a bundler would, shipping everything it finds along the way. You never have to enumerate a module’s own dependencies yourself.

Under the hood, this bundles your local files and publishes them on your behalf to the DCP package manager, under an automatically generated name – the same underlying mechanism used for shared, already-published packages, just automated here so you never see it happen. It runs on every deploy.

Note

Local, path-based job.requires() like this is specific to Node.js – bundling a set of local files needs real filesystem and build-tooling access that a browser doesn’t have. A browser job author ships a module’s contents directly instead, as a job argument. The FFTW-WASM tutorials walk through the same module used both ways, on both platforms.

Compute Groups

DCP jobs can be deployed to one or more Compute Groups. A Compute Group may be public (such as the Global DCP group) or private. To deploy into a private group, you must have the appropriate join credentials.

Security note: Never hard-code joinSecret values in production code. Prompt for them at runtime or load them securely from environment variables. Any hard-coded secrets shown in examples are for demonstration purposes only.

If no Compute Group is specified, the job is deployed to the public Global DCP group by default.

Deploy to a Private Compute Group

job.computeGroups = [{ joinKey: '<key>', joinSecret: '<secret>' }];

Deploy to Multiple Private Compute Groups

job.computeGroups = [
  { joinKey: '<key1>', joinSecret: '<secret1>' },
  { joinKey: '<key2>', joinSecret: '<secret2>' },
  { joinKey: '<key3>', joinSecret: '<secret3>' },
];

Deploy to Both Public and Private Groups

To deploy to a private group and the public Global DCP group:

job.computeGroups = [
  { joinKey: '<key>', joinSecret: '<secret>' },
  { joinKey: 'public' },
];

Each listed group becomes eligible to execute slices of the job.

Event Listeners

DCP Jobs support a variety of optional event listeners that let you monitor execution, track results, and handle errors in real time. You can attach listeners to the job object before calling job.exec(). configure some optional events.

Ready State Change

Fires whenever the job’s ready state changes. States typically include: exec, init, preauth, deploying, listeners, compute-groups, uploading, and deployed.

job.on('readystatechange', (ev) => console.log(`Ready state: ${ev}`));

Accepted

Fires when the job is accepted by the DCP Scheduler. At this point, the job is assigned a unique ID that can be displayed or logged.

job.on('accepted', () =>
  console.log(`  Job id: ${job.id}\n  Awaiting results...`),
);

Result

Fires whenever a Worker returns a result. These events may arrive out of order relative to the input set. The final output from job.exec() is collated in the original input order.

job.on('result', (ev) => console.log(ev));

Error

Fires whenever a Worker encounters an error while processing a job slice. You can use this to debug or handle failures gracefully.

job.on('error', (error) => console.error('  Job error:', error));

Console

Allows capturing console.log calls from Workers. Since Worker code runs remotely, its console output isn’t visible locally unless you listen for this event. Useful for debugging distributed computations.

job.on('console', (con) => console.dir(con, { depth: Infinity }));

No Funds

Fires when the account paying for the job doesn’t have sufficient funds to continue processing. The job is paused until the account is topped up.

job.on('nofunds', (ev) => console.log(ev));

Cancel

Fires when the job is cancelled by the scheduler. In most cases, this is optional, since job.exec() rejects when the job is cancelled.

job.on('cancel', (ev) => console.log(ev));

Job Execution

Execute the job by calling job.exec(). This submits the job to the scheduler and returns a Promise that resolves when all slices have completed.

let results = await job.exec();

The resolved value contains the job’s results.

By default:

The Job is associated with the identity stored in id.keystore located in ~/.dcp.
If no additional arguments are provided, job.exec() will use default.keystore to pay for the Job with Compute Credits at the current marketRate.

Advanced topics such as using alternate payment accounts, customizing slice payment offers, and remote datasets will be covered in the next tutorial.

Learn more about executing jobs

Result Post-Processing

Once execution completes, you can process the returned results as needed. In this example, we pull the uppercase letters back out of each result to reassemble the string, and print each character’s classification alongside it.

let RESULTS = results.map((r) => r.upper).join('');
console.log(RESULTS);
console.log(results.map((r) => `${r.upper}:${r.kind}`).join('  '));

This produces the fully reconstructed uppercase string, followed by each letter’s vowel/consonant/punctuation classification.

Run it

Ensure you have installed dependencies and have your API keys in ~/.dcp.

1. From your terminal, run the script:

node toUpperCase.js

2. Observe the job submission and execution:

Job events (accepted, results, errors) will be logged in real time.
Once complete, the final uppercase string will be printed to the console.

3. Example terminal output:

dandesjardins@Dans-MacBook-Air-2 simplest % node toUpperCase
Ready state: exec
Ready state: init
Ready state: preauth
Ready state: deploying
Ready state: listeners
Ready state: compute-groups
Ready state: uploading
Ready state: deployed
  Job id: j7B1j09DGOsE7YNjX6IKwO
  Awaiting results...
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 1, result: { upper: 'Y', kind: 'consonant' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 2, result: { upper: 'E', kind: 'vowel' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 3, result: { upper: 'L', kind: 'consonant' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 4, result: { upper: 'L', kind: 'consonant' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 5, result: { upper: 'I', kind: 'vowel' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 6, result: { upper: 'N', kind: 'consonant' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 7, result: { upper: 'G', kind: 'consonant' } }
{ job: 'j7B1j09DGOsE7YNjX6IKwO', sliceNumber: 8, result: { upper: '!', kind: 'punctuation' } }
YELLING!
Y:consonant  E:vowel  L:consonant  L:consonant  I:vowel  N:consonant  G:consonant  !:punctuation

This demonstrates how DCP distributes computation across workers, even for small tasks like converting characters to uppercase.

Full Code

The following example combines all the concepts covered in this tutorial: input set definition, work function, local modules, job configuration, compute groups, event handling, execution, and result post-processing. Running this code will convert the string “yelling!” to uppercase using DCP Workers, and classify each letter along the way. It needs three files, all in the same directory: toUpperCase.js, charInfo.js, and vowels.js.

// toUpperCase.js
async function main() {
  const compute = require('dcp/compute');

  /* INPUT SET */
  const inputSet = Array.from('yelling!');

  /* WORK FUNCTION */
  async function workFunction(letter) {
    progress();
    const charInfo = require('./charInfo');
    return { upper: letter.toUpperCase(), kind: charInfo(letter) };
  }

  /* COMPUTE FOR */
  const job = compute.for(inputSet, workFunction);

  /* LOCAL MODULES */
  job.requires(['./charInfo']);

  /* COMPUTE GROUPS */
  job.computeGroups = [
    { joinKey: 'demo', joinSecret: 'dcp' },
    { joinKey: 'public' },
  ];

  /* PUBLIC INFO */
  job.public = {
    name: 'toUpperCase',
    description: 'Minimal demonstration of a distributed job',
    link: 'https://distributive.network',
  };

  /* EVENTS */
  job.on('readystatechange', (ev) => console.log(`Ready state: ${ev}`));
  job.on('accepted', () =>
    console.log(`  Job id: ${job.id}\n  Awaiting results...`),
  );
  job.on('result', (ev) => console.log(ev));
  job.on('error', (error) => console.error('  Job error:', error));
  job.on('nofunds', (ev) => console.log(ev));
  job.on('console', (con) => console.dir(con, { depth: Infinity }));

  /* EXECUTION */
  let results = await job.exec();

  /* RESULT POST-PROCESSING */
  let RESULTS = results.map((r) => r.upper).join('');
  console.log(RESULTS);
  console.log(results.map((r) => `${r.upper}:${r.kind}`).join('  '));
}
require('dcp-client').init('https://scheduler.distributed.computer').then(main);

// charInfo.js
const vowels = require('./vowels');

module.exports = function charInfo(char) {
  if (!/[a-z]/i.test(char)) return 'punctuation';
  return vowels.has(char.toLowerCase()) ? 'vowel' : 'consonant';
};

// vowels.js
module.exports = new Set(['a', 'e', 'i', 'o', 'u']);

When executed, the job is submitted to the DCP scheduler, distributed across workers, and the final uppercase string is printed to the console. You can observe job events such as acceptance, results, and errors in real time.

This complete example provides a foundation you can modify to experiment with different input sets, work functions, and Compute Group configurations.

charInfo/vowels are plain JS, but job.requires() doesn’t care what a local module contains – the FFTW-WASM tutorial applies the exact same mechanism to a compiled WebAssembly module.