Referencing a Remote File for Input

Using a compute manifest it is possible to create a reference to a remote file as an input file to a plugin. This is done primarily through two parts, a DataStore, and a DataSource.

DataStore

A DataStore is composed of a name, a store type, a profile name, and a set of payload attributes. The name is used to define which store a DataSource is associated with. The profile name is a string that distinguishes one DataStore from another in the credentials through something such as secrets manager. It is expected that the payload attribute for a store type of S3 contains a "root" key with the value being the root key path to a location on an S3 bucket.

Example:

example store

cc.DataStore{
	Name: "CWMS-BASE",
	ID: <GUID-IGNORED>
	StoreType: cc.S3,
	DsProfile: "CWMS",
	Parameters: cc.PayloadAttributes{
		"root": "/model-library/<repository>,
	},
}

CODE

DataSource

A DataSource defines a specific datasource. it is composed of a name, a set of paths, a set of datapaths and a store name. The set of paths is a list of strings that makes up a reference to a logical unit. This could be a path to a DSS file, HDF file, or three paths one to each of .shp, .shx and .dbf. For files that have internal data, the datapaths make up the internal paths that are pertinant to the datasource. This could represent a specific table in an HDF file, or it could represent the "A/B/C/D/E/F" parts of a DSS record in a DSS file.

Example:

cc.DataSource{
				Name:      "RasModel.p01.tmp.hdf",
				Paths:     []string{"ras/RasModel.p01.tmp.hdf"},
				StoreName: "CWMS-BASE",
			}

CODE

When the plugin SDK plugin manager is used to fetch a dataSource from the IO manager, the store root is concatenated with the path(s) with a "/" seperator to create the key for the object in S3, and the object can be returned as a ByteStream or a ByteBlob depending on which plugin manager endpoint the user uses to fetch the file.