Deploying jobs with remote data

A DCP job has four pieces: the input set, the work function, the arguments, and the results. By default, all four travel through the Scheduler. DCP also lets you route any subset of them directly between your own infrastructure and the Workers instead – the Scheduler only ever needs to coordinate the job, not hold its data.

This page builds that up one capability at a time, starting from the default and ending with every piece routed remotely. Each stage shows the same job, in both JavaScript and Python, with one new piece of the “everything through the Scheduler” picture peeled away. Every example on this page was deployed and run for real against a live Scheduler to confirm the numbers actually come out right, not just that the code doesn’t crash. For the security rationale behind all of this (and for concrete use cases – protecting proprietary logic, view-only dashboards, regulated data that can’t leave your network), see Data privacy and security in DCP. For what kinds of values can actually make the trip once they’re not routed remotely, see Serializing job data with KVIN.

HTTP, HTTPS, and choosing where to host your data

Whenever a Worker fetches something directly from your server, two things need to line up: your server has to allow the Worker’s origin via CORS, and the Worker has to be configured to allow your server’s origin. Both HTTP and HTTPS work – the examples on this page default to HTTP for simplicity, with HTTPS shown as a drop-in alternative – but which one actually makes sense depends on two independent things: what kind of Worker is fetching, and whether you control every device involved or you’re serving an audience you don’t.

Note

See Setting up DCP workers for Worker setup in general.

Browser Workers vs Docker/native/screensaver Workers

• Browser Workers (dcp.live, dcp.work, or your own): allow the origin from the page’s console, dcpWorker.originManager.add('http://your-server:port', null, null), or via the allowOrigins URL query parameter. The scheme, host, and port all have to match your server exactly.

  • HTTP data fetched from an HTTPS worker page is blocked as “mixed content” unless you explicitly allow it (in Chrome: site settings -> Insecure content -> Allow).

  • HTTPS with a self-signed certificate isn’t trusted by the browser by default, and a background fetch can’t show you the usual “proceed anyway” click-through – visiting each endpoint URL directly first and accepting the warning there works, but doesn’t scale (see below).

• Docker, native, and screensaver Workers: allow the origin with the --allow-origin command-line flag instead. These run on Node.js, not a browser, so CORS headers don’t apply to them at all – they’re a browser-only mechanism, and one of these Workers will happily fetch from an origin with no Access-Control-Allow-Origin header whatsoever. What does still apply is certificate trust: Node’s HTTP client validates TLS the same way a browser does, so a self-signed HTTPS certificate still needs to be trusted some other way – there’s no interactive click-through available for a headless process.

Serving a fleet you control (on-prem, every device is yours)

Self-signed certificates are a completely legitimate choice here, because you can distribute trust yourself instead of relying on a public CA:

• Docker/native/screensaver Workers: launch with the standard Node.js environment variable NODE_EXTRA_CA_CERTS=/path/to/cert.pem (mounting the cert file into the container if applicable). This adds it to that process’s trusted CA bundle, the same way you’d already distribute any other config to your own fleet.

• Browser Workers: install the certificate into each machine’s OS trust store (on macOS, Keychain Access, marked “Always Trust” for SSL) rather than clicking through the warning in every browser individually – or push it via whatever configuration management you already use for the fleet. Tools like mkcert automate this for a single machine (mkcert -install), but that install is local to the machine it ran on – fine for your own dev box, not a distribution mechanism for a fleet.

Plain HTTP works too, with no certificate management at all, if your network doesn’t need encryption in transit.

Serving an audience you don’t control

Self-signed certificates don’t scale here. There’s no way to get a stranger’s device to trust a certificate you haven’t gotten a publicly trusted CA to vouch for – mkcert, and manually clicking through a warning, are both fundamentally single-machine tools. You need two things at once: a certificate from a publicly trusted CA, and an endpoint that’s actually publicly reachable – a private, NAT’d IP address (like the 192.168.2.12 used throughout this page) isn’t reachable by outside devices at all, regardless of what certificate it has. Note this isn’t a reason self-hosting is off the table – a server you run yourself works exactly as well as any other option here, once both of those boxes are checked. Concretely:

• Your own server, a real domain, and a CA like Let’s Encrypt. The most flexible option – full control over behavior – at the cost of managing DNS and certificate renewal yourself.

• Your own server behind a tunneling service (Cloudflare Tunnel, ngrok, etc.). The fastest path to a public HTTPS URL with a certificate already handled for you – a good fit for demos or short-lived events.

• Object storage (Amazon S3, Google Cloud Storage). Objects are served over the provider’s own publicly trusted certificate with no certificate management on your end at all, and both support real, first-class, bucket-level CORS configuration – set it to allow your Worker origins the same way you’d configure any other server, with no server to actually run.

Warning

Google Drive and Dropbox direct-download links are not a substitute for the object storage option above for browser Workers: neither sets the CORS headers browsers require, so a browser Worker’s fetch is blocked outright. This is a long-standing, widely-reported limitation of both services generally (not specific to DCP) – confirmed via bug reports and threads on Google’s and Dropbox’s own community support forums, not something specific to this page’s testing. Docker/native/screensaver Workers aren’t affected, since CORS is a browser-only restriction – the limitation is specifically for browser Workers.

Quick reference:

	Fleet you control	Audience you don’t control
HTTP	Works everywhere; browser Workers need “allow insecure content” set once per worker page	Same mixed-content restriction hits every visitor’s browser individually – impractical past a handful of people
Self-signed HTTPS	Works – NODE_EXTRA_CA_CERTS (Docker/native) or an OS trust-store install (browser)	Doesn’t scale – nothing installed on your own machine, mkcert included, is trusted by devices you don’t control
Publicly trusted HTTPS	Works, usually unnecessary overhead for a fleet you already control	The only option with zero per-device setup – your own server with a real cert, a tunnel, or object storage

Stage 0: The default – everything through the Scheduler

This is ordinary compute.for/compute_for, covered in Getting started – included here as the baseline the rest of this page peels back, one piece at a time.

JavaScriptPython

async function main() {
  const compute  = require('dcp/compute');
  const identity = require('dcp/identity');

  await identity.set('<your-private-key>');

  // INPUT SET
  const inputSet = [1, 2, 3];

  // WORK FUNCTION
  async function workFunction(input, arg1, arg2) {
    progress();
    return input * arg1 * arg2;
  }

  // WORK FUNCTION ARGUMENTS
  const args = [11, 22];

  // COMPUTE.FOR
  const job = compute.for(inputSet, workFunction, args);

  // COMPUTE GROUPS
  job.computeGroups = [{ joinKey: 'public' }];

  // JOB PUBLIC INFO
  job.public.name = 'remote-data-stage-0';

  // EVENTS
  job.on('accepted', () => console.log(`Job id: ${job.id}\nAwaiting results...`));
  job.on('result', (ev) => console.log(ev));

  // JOB.EXEC
  const results = await job.exec();
  console.log(`Done.\n${results}`);
}

require('dcp-client').init('https://scheduler.distributed.computer').then(main);

Confirmed output for both: results 242, 484, 726 (1*11*22, 2*11*22, 3*11*22).

Stage 1: Remote input set

Use case: your input data already lives somewhere – a hospital’s imaging archive, an internal database export – and shouldn’t be uploaded through the Scheduler just to get to a Worker.

Wrapping the input set in RemoteDataSet has Workers fetch each element directly from the URLs you provide, instead of receiving them from the Scheduler.

JavaScriptPython

async function main() {
  const compute  = require('dcp/compute');
  const identity = require('dcp/identity');

  await identity.set('<your-private-key>');

  // INPUT SET
  const inputSet = new compute.RemoteDataSet([
    'http://192.168.2.12:3000/inp1',
    'http://192.168.2.12:3000/inp2',
    'http://192.168.2.12:3000/inp3',
  ]);

  // WORK FUNCTION
  async function workFunction(input, arg1, arg2) {
    progress();
    return input * arg1 * arg2;
  }

  // WORK FUNCTION ARGUMENTS
  const args = [11, 22];

  // COMPUTE.FOR
  const job = compute.for(inputSet, workFunction, args);

  // COMPUTE GROUPS
  job.computeGroups = [{ joinKey: 'public' }];

  // JOB PUBLIC INFO
  job.public.name = 'remote-data-stage-1';

  // EVENTS
  job.on('accepted', () => console.log(`Job id: ${job.id}\nAwaiting results...`));
  job.on('result', (ev) => console.log(ev));

  // JOB.EXEC
  const results = await job.exec();
  console.log(`Done.\n${results}`);
}

require('dcp-client').init('https://scheduler.distributed.computer').then(main);

Warning

The int(input) conversion above isn’t optional. It was found by actually running this example: without it, the Python version doesn’t error or crash – it silently returns a giant string of repeated digits instead of a number, because Python’s * operator repeats strings rather than coercing them like JavaScript’s does. Confirmed output for both languages, once fixed: 2904, 8228, 5082 (12*11*22, 34*11*22, 21*11*22).

Serving the data. Workers fetch each URL directly, so your server needs to be reachable from wherever your Workers run (not localhost):

JavaScript (Node)Python (Flask)

// Runs over HTTP by default. Set PROTOCOL=https to run over HTTPS
// instead (needs key.pem/cert.pem):
//   openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem \
//     -days 365 -nodes -subj '/CN=192.168.2.12'
const http  = require('http');
const https = require('https');
const fs    = require('fs');

const protocol = process.env.PROTOCOL === 'https' ? 'https' : 'http';
const inputSet = { inp1: 12, inp2: 34, inp3: 21 };

function handler(req, res) {
  // Reflects back whatever Origin asked, so this works for any
  // browser worker page without hardcoding a specific allow-list.
  // Fine for local/LAN test servers; restrict this in production.
  // Only relevant to browser Workers -- Docker/native/screensaver
  // Workers ignore this header entirely (see above).
  if (req.headers.origin)
    res.setHeader('Access-Control-Allow-Origin', req.headers.origin);
  res.setHeader('Content-Type', 'text/plain');

  const key = req.url.slice(1);
  if (inputSet[key] !== undefined) {
    res.writeHead(200);
    res.end(String(inputSet[key]));
  } else {
    res.writeHead(404);
    res.end('not found');
  }
}

const server = protocol === 'https'
  ? https.createServer({ key: fs.readFileSync('key.pem'), cert: fs.readFileSync('cert.pem') }, handler)
  : http.createServer(handler);

server.listen(3000, '0.0.0.0');

Stage 2: + remote results

Use case: neither side of the computation should be visible to DCP at all – input and output both stay inside your own infrastructure or compliance boundary.

job.setResultStorage() has each Worker POST its slice’s result straight to an endpoint you specify, instead of to the Scheduler. See Getting results back below for exactly what this means for job.wait().

JavaScriptPython

async function main() {
  const compute  = require('dcp/compute');
  const identity = require('dcp/identity');

  await identity.set('<your-private-key>');

  // INPUT SET
  const inputSet = new compute.RemoteDataSet([
    'http://192.168.2.12:3000/inp1',
    'http://192.168.2.12:3000/inp2',
    'http://192.168.2.12:3000/inp3',
  ]);

  // WORK FUNCTION
  async function workFunction(input, arg1, arg2) {
    progress();
    return input * arg1 * arg2;
  }

  // WORK FUNCTION ARGUMENTS
  const args = [11, 22];

  // COMPUTE.FOR
  const job = compute.for(inputSet, workFunction, args);

  // COMPUTE GROUPS
  job.computeGroups = [{ joinKey: 'public' }];

  // JOB PUBLIC INFO
  job.public.name = 'remote-data-stage-2';

  // EVENTS
  job.on('accepted', () => console.log(`Job id: ${job.id}\nAwaiting results...`));
  job.on('result', (ev) => console.log(ev));

  // REMOTE RESULT STORAGE
  job.setResultStorage(
    'http://192.168.2.12:3000/results',
    { elementType: 'results' },
  );

  await job.exec();
  console.log('Done.');
}

require('dcp-client').init('https://scheduler.distributed.computer').then(main);

Receiving results. Each result arrives as an application/x-www-form-urlencoded POST with elementType, contentType, element (the slice number), and content (the value) fields:

JavaScript (Node)Python (Flask)

// ...add to the Stage 1 server's handler():

if (req.url === '/results' && req.method === 'POST') {
  let body = '';
  req.on('data', (chunk) => (body += chunk));
  req.on('end', () => {
    const params = new URLSearchParams(body);
    console.log('result received:', {
      elementType: params.get('elementType'),
      contentType: params.get('contentType'),
      element:     params.get('element'),
      content:     params.get('content'),
    });
    res.writeHead(200);
    res.end('ok');
  });
  return;
}

Confirmed: the server receives the real computed values (2904, 8228, 5082) regardless of which language deployed the job.

Stage 3: + remote arguments

Use case: shared reference data or configuration – a lookup table, a set of thresholds – that changes independently of the job and shouldn’t be baked into a redeploy.

RemoteDataSet works the same way for arguments as it does for the input set:

JavaScriptPython

async function main() {
  const compute  = require('dcp/compute');
  const identity = require('dcp/identity');

  await identity.set('<your-private-key>');

  // INPUT SET
  const inputSet = new compute.RemoteDataSet([
    'http://192.168.2.12:3000/inp1',
    'http://192.168.2.12:3000/inp2',
    'http://192.168.2.12:3000/inp3',
  ]);

  // WORK FUNCTION
  async function workFunction(input, arg1, arg2) {
    progress();
    return input * arg1 * arg2;
  }

  // WORK FUNCTION ARGUMENTS
  const args = new compute.RemoteDataSet([
    'http://192.168.2.12:3000/arg1',
    'http://192.168.2.12:3000/arg2',
  ]);

  // COMPUTE.FOR
  const job = compute.for(inputSet, workFunction, args);

  // COMPUTE GROUPS
  job.computeGroups = [{ joinKey: 'public' }];

  // JOB PUBLIC INFO
  job.public.name = 'remote-data-stage-3';

  // EVENTS
  job.on('accepted', () => console.log(`Job id: ${job.id}\nAwaiting results...`));
  job.on('result', (ev) => console.log(ev));

  // REMOTE RESULT STORAGE
  job.setResultStorage(
    'http://192.168.2.12:3000/results',
    { elementType: 'results' },
  );

  await job.exec();
  console.log('Done.');
}

require('dcp-client').init('https://scheduler.distributed.computer').then(main);

The server from Stage 2 already covers this – arg1/arg2 are served the same way as inp1/inp2/inp3, just a different set of keys behind the same lookup. Confirmed output for both languages: server receives 2904, 8228, 5082.

Stage 4: + remote work function

Use case: protecting proprietary logic, or building a catalogue of vetted, pre-approved work functions third parties can run without seeing (or being able to alter) the code. See Data privacy and security in DCP for more on why this matters.

The work function can be a URL, fetched by the Worker at execution time instead of being embedded in the job. At this point, only authentication still touches the Scheduler – it never sees the input set, the work function, the arguments, or the results:

JavaScriptPython

async function main() {
  const compute  = require('dcp/compute');
  const identity = require('dcp/identity');

  await identity.set('<your-private-key>');

  // INPUT SET
  const inputSet = new compute.RemoteDataSet([
    'http://192.168.2.12:3000/inp1',
    'http://192.168.2.12:3000/inp2',
    'http://192.168.2.12:3000/inp3',
  ]);

  // WORK FUNCTION
  const workFunction = new URL('http://192.168.2.12:3000/workFunction');

  // WORK FUNCTION ARGUMENTS
  const args = new compute.RemoteDataSet([
    'http://192.168.2.12:3000/arg1',
    'http://192.168.2.12:3000/arg2',
  ]);

  // COMPUTE.FOR
  const job = compute.for(inputSet, workFunction, args);

  // COMPUTE GROUPS
  job.computeGroups = [{ joinKey: 'public' }];

  // JOB PUBLIC INFO
  job.public.name = 'remote-data-stage-4';

  // EVENTS
  job.on('accepted', () => console.log(`Job id: ${job.id}\nAwaiting results...`));
  job.on('result', (ev) => console.log(ev));

  // REMOTE RESULT STORAGE
  job.setResultStorage(
    'http://192.168.2.12:3000/results',
    { elementType: 'results' },
  );

  await job.exec();
  console.log('Done.');
}

require('dcp-client').init('https://scheduler.distributed.computer').then(main);

Warning

Known issue, confirmed by testing: with the worktime fix above, this job runs without error, but the value that actually reaches the results server is currently broken specifically for jobs deployed via Python – the server receives content: null instead of the real computed value. The byte-identical JS version above works correctly (confirmed: 2904, 8228, 5082). Remote work functions on their own work fine from Python (see Stage 5), and setResultStorage() on their own works fine from Python (see Stage 2/3) – it’s specifically the combination, via bifrost2, that currently loses the value. If you need both together today, deploy from JavaScript.

Serving the work function. It’s fetched as plain text – the same server as before, with one more route serving the function’s source:

JavaScript (Node)Python (Flask)

const workFunction = `
async function workFunction(input, arg1, arg2) {
  progress();
  return input * arg1 * arg2;
}
`;

// add 'workFunction': workFunction to the same key/value lookup
// used for inputs and arguments in the Stage 2/3 server.

Note

Whatever runs on the Worker runs in DCP’s JavaScript sandbox, so a remotely-fetched work function is always JavaScript source, regardless of which language deployed the job – this is the same as the work_function: Union[str, Callable] distinction Python’s compute_for already makes for local work functions.

Stage 5: Mix and match

Use case: most jobs don’t need everything remote – one large dataset that shouldn’t transit the Scheduler, mixed with a couple of small literal values that are perfectly fine to send normally. RemoteDataSet wraps a whole collection; a bare URL works for a single element, dropped directly into an ordinary array.

This stage keeps results flowing through the normal Scheduler channel (no setResultStorage()), so – unlike Stage 4 – it works correctly from both languages, including the remote work function:

JavaScriptPython

async function main() {
  const compute  = require('dcp/compute');
  const identity = require('dcp/identity');

  await identity.set('<your-private-key>');

  // INPUT SET -- mixing local values and remote URLs freely
  const inputSet = [
    new URL('http://192.168.2.12:3000/inp1'),
    654,
    new URL('http://192.168.2.12:3000/inp3'),
  ];

  // WORK FUNCTION
  const workFunction = new URL('http://192.168.2.12:3000/workFunction');

  // WORK FUNCTION ARGUMENTS
  const args = [
    17,
    new URL('http://192.168.2.12:3000/arg2'),
  ];

  // COMPUTE.FOR
  const job = compute.for(inputSet, workFunction, args);

  // COMPUTE GROUPS
  job.computeGroups = [{ joinKey: 'public' }];

  // JOB PUBLIC INFO
  job.public.name = 'remote-data-stage-5';

  // EVENTS
  job.on('accepted', () => console.log(`Job id: ${job.id}\nAwaiting results...`));
  job.on('result', (ev) => console.log(ev));
  job.on('error', (err) => console.error(err));

  // JOB.EXEC
  const results = await job.exec();
  console.log('Done.');
  console.log(results);
}

require('dcp-client').init('https://scheduler.distributed.computer').then(main);

Confirmed output for both languages: 4488, 244596, 7854 (12*17*22, 654*17*22, 21*17*22).

Note

A fetched remote value arrives at the work function as raw text by default – the servers throughout this page serve everything as text/plain, and a numeric value like input * arg1 * arg2 only works in JavaScript because of its loose string-to-number coercion. Python doesn’t coerce that way (see the int() conversions above), so a remote value used for anything beyond simple JS-side arithmetic – structured data, or any Python work function – generally needs to be parsed explicitly (json.loads, etc.) inside the work function, unless the server sends it with a Content-Type KVIN already knows how to decode. See that page for the full picture of what does and doesn’t survive serialization.

RemoteDataPattern is a convenience for the common case of many URLs that all follow the same template, so you don’t have to write them out individually:

const { RemoteDataPattern } = require('dcp/compute');

const pattern = new RemoteDataPattern('http://192.168.2.12:3000/inp{slice}', 3);
const job = compute.for(pattern, workFunction, args);

Getting results back once storage is remote

Once you’re using setResultStorage(), the real computed value never transits DCP’s infrastructure at all – not the worker-to-scheduler leg, not any leg back to you. What actually flows through the normal job.wait()/'result' event path is whatever your own results endpoint’s response to the POST happens to be – in every example on this page, that’s just the literal string "ok", because that’s what these example servers respond with.

This isn’t a limitation to route around – it’s the point. You already have the real data, on your own server; there’s no reason for DCP to also be handed a copy of it, and with setResultStorage() it structurally never is. If your own process also needs the value in-line (not just on the server), design your endpoint to echo back something useful in its response, or read it from wherever you stored it – but don’t expect job.wait() to hand it to you automatically once you’ve routed it elsewhere.

See also

• Data privacy and security in DCP – the security rationale and access-control side of this story.

• Serializing job data with KVIN – what values can and can’t survive the trip once they’re not remote.

• Compute API: Remote Data Objects – the full class reference for RemoteDataSet and RemoteDataPattern.