Cloud Compute Core Library

The core library, cloud-compute, provides structure to the interaction between compute providers, event-generators, and registered plugins. 

Jobs

A job is a distinct unit of work. It is an association of a plugin with a specific set of instructions on a compute environment.

Compute Providers

A compute provider is an interface that is used to define how distributed computing is implemented through different vendors and platforms. 

Compute providers must:

• Submit Jobs
• Get Job Status
• Get Job Logs
• Register Plugins
• Unregister Plugins

For each implementation of a compute provider, a distinct set of instructions must exist to support administration of the vendor specific infrastructure on an account. Currently the implementations for compute provider consist of an implementation for AWS Batch, and an implementation for docker local. Additional implementations for Azure, Google Cloud or other compute providers are possible but have not been implemented.

Event Generators

Event generators provide a set of jobs defined by a DAG to a compute provider. Event generators can provide the same DAG to the compute provider with changes to environment variables to create a simulation. An example could be running a Monte Carlo or running a set of forecast ensembles.

Compute

A compute combines an event generator with a compute provider and a directed acyclic graph of jobs to push each job to the compute provider across all generated events.