Tutorial

Developer Guide

Welcome to the first developer release of the Distributed Compute Platform! We're super excited to have you aboard. This document should get you from zero-to-distributed-hero in no time.

Overview

This tutorial will cover how to:

  1. Register for a DCP account
  2. Download and setup your keystores
  3. Setup your development environment
  4. Deploy your first job

Tutorial

1. Register For an Account

First, register for an account on our portal. Once you register and verify your account you'll now have access to DCP!

For a quick tour of the portal:

Worker Tab

The worker tab is where you can provide your computers compute resources to the distributed compute network. After pressing start, your computer will begin computing work on the network. You may even hear your fans rev up as you perform work! In addition, under Credits Earned you can see how many Distrbuted Compute Credits (DCCs) you have earned for your computer's work.

Worker Tab UI

The DCP Task Worker runs in the background — even if you switch to another tab — until you click the STOP button. This process does not materially affect other tasks that are running on your computer (apart from using otherwise idle processor time).

Accounts

The accounts tab displays all of your bank 'keystores' with DCP. You're given a default bank keystore and can create as many as your want! If you wish to view your DCC balance, click the refresh icon to retrieve the latest balance from the database. You can even watch your balance increase as you run a worker.

Accounts Tab UI

2. Download and Setup Your Keystores

Download Bank Keystore

To download your bank keystore navigate to the accounts tab in the portal, click the down arrow and then 'download' under the keystore you wish to download. This bank keystore is the one DCCs are given and taken away from as you work on and deploy jobs on the network.

Name the downloaded .keystore file default.keystore as this is the default name DCP applications search for on your computer when finding your bank keystore.

Download the keystore to a location you'll remember because we're going to need it later!

Download Bank Keystore

Download id Keystores

To download your id keystore navigate to the profile tab on the portal. Click the green '+' icon under identity keys near the bottom of the page. Feel free to give it a name, then click the blue download button. Your id keystore is unique to you and lets the network know who is deploying which job on the network.

Download id Keystore

You'll then be prompted to give it a passphrase which you'll have to enter anytime you launch a job on DCP. If you don't want a passphrase just leave each form blank and click continue.

Once it's downloaded be sure to name the file id.keystore as this is the default name DCP applications search for on your computer when finding your id keystore.

Setup Keystores

Every DCP application you launch looks in a default location for your keystores to launch jobs.

Windows

On windows the default location is in C:\Users\<YOUR_USER_NAME>\.dcp.

If .dcp isn't in this directory, you can just make a new folder and move your id.keystore & default.keystore files into it.

MacOS/Linux

On UNIX machines the default location is in /home/.dcp.

If .dcp isn't in this directory, you can just make a new folder and move your id.keystore & default.keystore files into it.

3. Setup your development environment

The process for launching jobs on DCP depends on the Javascript environment you prefer to use. Right now you can launch a DCP application from NodeJS or simply using vanilla Javascript from a web project! We'll show you how to do both to get you started.

NodeJS

Install the latest version of Node.

If this is a new project, be sure to create a new directory and in a terminal run npm init to create a package.json file.

To install dcp-client as a node dependency, run the following command in your terminal:

npm i dcp-client

You're now ready to write and deploy your first job! Check out our Node exemplar for a good example of launching your first node application on DCP.

Vanilla Web

If you prefer to launch jobs on DCP from the web with html & Javascript, this approach is for you!

To gain access to dcp within your web project all you need to do is add the following script tag to the head of your html document before any other JavaScript code:

<script src="https://scheduler.distributed.computer/dcp-client/dcp-client.js"></script>

This will create a dcp global that can be used to access the DCP modules.

Don't know what that means? No problem! Here's a quick example we made using JSFiddle to help get you started.

See demo

Deploying Jobs

At its core, a job can be thought of as an input set, and a Work function; executing a job yields an output set.

Jobs can be created with the compute.for method. The arguments to this method determine how many 'slices' the job will have and what data each 'slice' should be executed with. A slice is your work function paired with one of your provided inputs.

In Node

Create 10 slices. In this case the first slice will run your function with 1 as an input, then 2 ... all the way to 10.

require('dcp-client').initSync();
const compute = require('dcp/compute');
// dcp-client is ready to be used 

const job = compute.for(1, 10, function(n) {
  let result = n + 100;
  progress('50%');
  result = result * n;
  progress('100%');
  return result;
});

// execute the job
job.exec();

In Vanilla-Web

Create a slice for each color in an array:

const { compute } = dcp;

const job = compute.for(['red', 'green', 'blue'], function(colour) { 
  console.log(colour);
  progress();
  return colour;
});

// execute the job
job.exec();

Note:

The last argument, the work function, is what will be executed on the workers. It must be either a string, or stringifyable via toString().

The work function must call the global function progress() at least one time during its execution, and at least once per minute. This is to ensure that the slice is operating nominally and has not crashed. If the slice does not call progress, or does not call it frequently enough, the job will be cancelled with the error ENOPROGRESS. Progress can be provided with a value that will be used to show how far the slice has completed. This value can be a percentage between 0 and 1, a string like '50%', or undefined to indicate indeterminate progress.

Limitations to consider: Since the Work function must be either a string or stringifyable via toString(), it means that native functions (i.e. Node functions written in C++) cannot be used for Work. Additionally, the function must be completely defined and not a closure, since stringification cannot take the closure environment into account. This means requiring all dependent code to be within the work function, since dependencies that are being required outside will not be available in the workers. A rule of thumb is that if you cannot eval() it, you cannot distribute it.

A word of advice: deploy locally & save your credits ⚠️

​ If you want to test deploying a job but don't want to use up all your credits, we suggest to try deploying your job to your own computer first using a method called localExec() rather than the regular exec() . It's always better safe than sorry to make sure your job works before spending credits distributing it to hundreds of computers! You will find some examples of this method in our documentation so if you have any questions, don't hesitate to ask us in our forum.

Using DCCs

Distributed Compute Credits, or DCC, are used within DCP to facilitate payment for work. The DCP bank maintains a ledger of DCCs associated to accounts. Accounts are represented and accessed by keystore files, which contain the private key used to authorize access to a bank account. Try launching a job and view your DCC balance decrease, or do some work on the worker tab and see yourself earn credits.

By default, when deploying a job from NodeJS it will look for your account keystore in .dcp/default.keystore. When deploying a job from the web, a modal will be displayed to prompt you for your payment keystore.

Note In its current implementation, the value of DCC is not tied to anything. It's a placeholder for testing / experimental purposes. The MVP release will include an implementation of the costing and metering algorithms fundamental to tying DCC balance to actual work done.

Listening for Events

There are a number of useful events available on the job handle, which is itself an Event Emitter. The table below lists the most frequently used events; the rest of which can be found in the Compute API Documentation.

Event Name Emitter Description
accepted job Emitted when the scheduler accepts this job during deploy
status job Emitted when a slice is distributed or finished being computed
result job Emitted when a result is submitted by a worker
console job Emitted when a console method (log, warn, error, debug) is invoked in the work function
uncaughtException job Emitted when a slice throws an error

Examples

  1. Deploying a simple job and checking the results
let job = compute.for(1, 3, function (i) {
  progress();
  return i*10;
});

job.on('accepted', () => console.log("Job accepted", job.id));
job.on('complete', () => console.log("Job complete!"));

let results = await job.exec();
console.log('results:    ', results);
console.log('entries:    ', results.entries());
console.log('fromEntries:', results.fromEntries());
console.log('keys:       ', results.keys());
console.log('values:     ', results.values());
console.log('key(2):     ', results.key(2));
See demo

  1. Listening for console events
let job = compute.for(1, 5, function (i) {
  progress();
  console.log("Hello, World!", i);
  return i;
});

job.on('console', (event) => console.log(event));

await job.exec();
See demo

  1. Listening for exceptions thrown during execution
let job = compute.for(1, 3, function (i) {
  throw new Error("Oops, an error was thrown! Better listen for these events to know when it happens!");
});

job.on('error', (event) => {
  console.error("An exception was thrown by the work function:", event.message);
});
await job.exec();
See demo

Modules

The following modules are available from the DCP Client package:

Global dcp (script tag) CommonJS Module Description Documentation
dcp['client-modal'] n/a Module for creating modals in the browser module:dcp/client-modal
dcp.compute 'dcp/compute' The Compute API, used to deploy and manage work that has been deployed to the network module:dcp/compute
dcp['dcp-build'] 'dcp/dcp-build' The build object is exported for introspection. module:dcp/dcp-build
dcp['dcp-cli'] 'dcp/dcp-cli' Provides a standard set of DCP CLI options and related utility functions via yargs. module:dcp/dcp-cli
dcp['dcp-config'] 'dcp/dcp-config' The active configuration object. module:dcp/dcp-config
dcp['dcp-events'] 'dcp/dcp-events' Provides classes related to cross-platform event emitting. module:dcp/dcp-events
dcp.wallet 'dcp/wallet' The Wallet API, used to access and manipulate DCP addresses and keystores module:dcp/wallet
dcp.worker 'dcp/worker' The Worker API, used to start workers on DCP module:dcp/worker

GPUs

DCP currently supports WebGL and WebGL2.0 for parrallel data processing. This allows for fractal rendering, massively parallelized neural networks, WebGL based render farming, and other GPU heavy tasks.

Since WebGPU promises significantly more functionality than WebGL, DCP is working on supporting the WebGPU Compute API (not for rendering graphics yet) which exposes the new compute pipelines available from the latest GPU backends such as Vulkan, Metal, and DX13. Direct access to modern Backend GPU APIs will make the DCP Distributed GPU Supercomputer even more powerful!

For information on how to set requirements for your jobs see Requirements Object Properties.

What's Next?

You're all set to go and deploy your own jobs on dcp! Please feel free to check out other guides or read through our documentation to explore what dcp has to offer. Happy coding!