RemoteDataPattern

class RemoteDataPattern()

A compact alternative to RemoteDataSet() for many URIs that all share the same shape. Instead of listing every URI individually, describe a pattern containing the literal placeholder {slice} and a sliceCount; RemoteDataPattern expands this into sliceCount URIs, one per slice, substituting {slice} with each slice number. As with RemoteDataSet, each slice’s work function receives the fetched contents of its URI, not the URI itself.

Constructor

new RemoteDataPattern(pattern, sliceCount);

Arguments:

• pattern (string|URL|DcpURL) - A URI containing the literal substring {slice}, for example, 'https://example.com/data/{slice}.json'. If pattern is a URL/DcpURL, its percent-encoded equivalent %7Bslice%7D is also accepted.

• sliceCount (number) - The number of URIs the pattern expands to.

Throws a DCPError if pattern doesn’t contain {slice} (or its percent-encoded form), or if sliceCount is falsy or not a number.

new is optional – RemoteDataPattern(...) and new RemoteDataPattern(...) behave identically.

Note

Slice numbers are substituted starting from 1, not 0 – new RemoteDataPattern('.../{slice}.json', 3) expands to .../1.json, .../2.json, and .../3.json.

Note

A RemoteDataPattern can only be used as the input set of a job (e.g. the first argument to compute.for()); passing one as a job’s arguments throws a DCPError with code EBADARG.

Properties

pattern

RemoteDataPattern.pattern

Type:

string

The pattern this object was constructed with.

sliceCount

RemoteDataPattern.sliceCount

Type:

number

The number of URIs this pattern expands to.

Note

RemoteDataPattern also has a length property, fixed at 2, inherited from its Array-like implementation. It does not reflect sliceCount and should not be used to determine how many URIs the pattern expands to – use sliceCount instead.

Example

const { RemoteDataPattern } = require('dcp/compute');

const pattern = 'http://site.com/{slice}.json';
const dataPattern = new RemoteDataPattern(pattern, 2);
// expands to http://site.com/1.json and http://site.com/2.json

const job = compute.for(dataPattern, function (contents) {
  console.log(contents);
});
await job.exec();