The plugin software development kit (SDK) is provided to support simpler development of plugins to participate in cloud compute. A cloud compute plugin can be written in any language, and does not need to use the SDK to participate in a compute, however, the SDKs are designed to manage common workflows and standards. The plugin SDK defines data structures and common processes to improve the consistency of plugin development, while maintaining a high degree of flexibility. The SDK is supported in multiple languages with the immediate efforts focused on Go, Java, Python, and DotNet. 


Go SDK https://github.com/USACE/cc-go-sdk

Java SDK https://github.com/USACE/cc-java-sdk

DotNet SDK https://github.com/USACE/cc-dotnet-sdk

Python SDK - https://github.com/USACE/cc-py-sdk


Plugins can be created without the software development kit, however, it will require more effort to build a plugin without the sdk. The developer will need to be able to interpret the environment variables, and if manifests/payloads are used interpret the instructions in the manifest/payloads to perform the appropriate work. The SDKs are developed to simplify that workflow. 

SDK Concepts

The software development kits (SDK)s have some major concepts to support developers in developing plugins. This documentation provides an overview of the concepts, for more detail see the cloud-compute technical documentation.

Plugin Manager

The plugin manager provides the developer the ability to access the information stored in a payload at the time of execution. The plugin manager also facilitates the ability to interact with remote or local storage and read datasources identified in the payload through the IO Manager.

DataSource

A datasource can be a file or a set of files (.prj, .shp, .dbf). A datasource is composed of a name, a set of file paths, and a set of data file paths. The file paths represent the files that make up the data source, the data file paths represent the internal data (e.g a/b/c/d/e/f parts in dss or a table path in HDF). Datasources are stored in DataStores, each datasource has a data source associated with it.

DataStore

A datastore can be a remote datastore, a local datastore, or a database. Examples of datastores are object stores like AWS S3, or file stores like AWS EFS, or a traditional network file store. Datastores store data that are used by plugins for input datasource or as output datasources.

Payload Attributes

A payload attribute is a dictionary of string object that is defined within a payload and used to convey parameters important for the plugin to execute a job. Payload attributes are provided in the base payload and in each Action within the payload.

IO Manager

The IO Manager provides simplified access to datasources stored in a datastore and payload attributes.

IOManager provides the following methods:

  • GetStore
  • GetDataSource
  • CopyFIleToLocal
  • CopyFileToRemote
  • GetAttributeOrDefault
  • GetAttributeOrFail

Actions

An action is composed of a string name, a type, and an IO manager. Each payload can have a list of Actions. Actions are performed in order as defined within the payload. Actions can be utilized to provide a series of tasks for a given plugin container to perform on a set of data. Actions provide enormous flexibility in what a plugin does, and represent a declarative capability to include/exclude, order, or modify parameterization of plugin specific code. For example the plugin for HEC-RAS could have actions for run-unsteady, run-steady, run-geometry-preprocess, generate-stored-map, and perform-qa-qc. The ordered list could look like this:

  • perform-qa-qc
  • run-geometry-preprocess
  • run-unsteady
  • generate-stored-map

Note that actions are optional, not all actions must be used, and actions can be repeated in the list provided via the payload.

Payload

A payload is serialized to a json file that all SDKs can desacralize. The file contains instructions to the plugin regarding attributes, input data sources, and output data sources that will define the overall job assigned to a plugin.

When payloads are loaded by the PluginManager, the SDK updates the payload to dynamically replace information in the payload based on Environment Variables and Payload Attributes. A payload can have a file path, data path, or datasource name with the appropriate information based on the following syntax. Environment variables will be updated if {ENV::<variablename>} is inserted into an updateable element, and the variable name is present in the Environment Variables of the container. Likewise attributes will be updated if {ATTR::<attributename>}  is updated if the PayloadAttributes contain the specified attribute name. If an attribute name or a environment variable is used in substitution but not provided at compute time, the PluginManager will fatally fail.


Example Payload

{
    "stores": [
        {
            "name":      "FFRD",
            "store_type": "S3",
            "profile": "FFRD",
            "params": {
                "root": "/model-library"
            }
        }
    ],
    "payloadAttributes": {
      "modelPrefix": "modelname",
      "modelPath": "model-directory"
	  "outputPath": "simulations"
    },
    "inputs":{
	    {
           "name": "{ATTR::modelPrefix}.json",
           "paths": {
               "default":"{ATTR::modelPath}/{ATTR::modelPrefix}.json"
            },
			"store_name": "FFRD"   
        } 
    },
    "outputs": [
        {
            "name": "output.json",
            "paths": {
                    "default":"{ATTR::outputPath}/{ENV::CC_EVENT_NUMBER}/{ATTR::modelPath}/output.json"
            },
            "store_name": "FFRD"   
        }
    ],
    "actions":[
        {
            "name":        "compute_json",
            "type":        "compute_json",
            "description": "compute something",
            "params": {
                "parameter1":      true,
                "parameter2":      "parametervalue2",
                "parameter3": "2.3"
            }
        } 
    ]   
}
CODE