Seed-Generator

In a stochastic compute, it is critical for the plugins to have coordinated seeds for the expression of Natural Variability and Knowledge Uncertainty. Instead of making that central to the distribution of events and jobs, it has been delegated to a plugin to allow for maximum flexibility in application.

This plugin provides a similar seed structure to that of HEC-WAT 1.0, though it is not identical due to different random number generators it is the same in concept.

The seed generator has a model definition (similar to an alternative in HEC-FIA, a simulation in HEC-HMS, or a plan in HEC-RAS) in JSON that controls what plugins to produce seeds for, their initial seeds to control the overall sequence, and the number of events per realization. This is a model file that would be unique to a specific watershed and specify how the plugin should operate for this watershed. Each set of plugin initial seeds would need to be defined for each plugin that needs seeds in that watershed.

{
	"initial_event_seed": 1234,
	"initial_realization_seed": 9876,
	"events_per_realization": 10,
	"plugin_initial_seeds": {
		"fragilitycurveplugin": {
			"event_seed": 234,
			"realization_seed": 987
		}
	}
}

By seeding the overall simulation, and each plugin, the generator can reproduce the exact same sequence of seeds which allows for reproducibility of an event, or a whole simulation. This information is stored in a JSON file called "seedgenerator" and is provided via a datasource to the plugin at runtime.

The plugin entrypoint is the main.go file. It operates as a simple commandline application with no arguments. While arguments can be specified in a plugin's manifest file, this one has none.

func main() {
	fmt.Println("seed generator!")
	pm, err := wat.InitPluginManager()
	if err != nil {
		pm.LogMessage(wat.Message{
			Message: err.Error(),
		})
		return
	}
	payload, err := pm.GetPayload()
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return
	}
	err = computePayload(payload, pm)
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return
	}else{
 	pm.ReportProgress(wat.StatusReport{
		Status:   wat.SUCCEEDED,
		Progress: 100,
	})
}

For simple plugins that internally define their own models (like this one) it is easiest to leverage the plugin manager to get any model files or inputs. This plugin has only one input - the seedgenerator.json file described above, and one output, a set of seeds in a map for each plugin. As you can see on line 3 above the first step in the plugin is to initalize the plugin manager. If the SDK detects any missing necessary environment variables the initialization step will provide an error which can be reported to the logger and then the plugin escapes by returning from main.

If there are no errors in initializing the plugin manager the plugin can request the payload for this job from the plugin manager. The payload will define input resources required for computing (such as the seedgenerator.json file described above).

If the payload is recieved without error, this plugin provides the payload and the plugin manager to the internal compute method for computing the event. If that compute payload completes without error a status of succeeded is sent back to the SDK which then sends it back to the Cloud Compute.

The internal computepayload method for the plugin performs a series of data checks to verify everything necessary for a compute is available and properly formatted.

func computePayload(payload wat.Payload, pm wat.PluginManager) error {
	if len(payload.Outputs) != 1 {
		err := errors.New("more than one output was defined")
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	var modelResourceInfo wat.DataSource
	found := false
	for _, rfd := range payload.Inputs {
		if strings.Contains(rfd.Name, "seedgenerator.json") {
			modelResourceInfo = rfd
			found = true
		}
	}
	if !found {
		err := fmt.Errorf("could not find %s.json", "seedgenerator")
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	modelBytes, err := pm.GetObject(modelResourceInfo)
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	var eventGeneratorModel seedgeneratormodel.Model
	err = json.Unmarshal(modelBytes, &eventGeneratorModel)
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	eventIndex := pm.EventNumber()
	modelResult, err := eventGeneratorModel.Compute(eventIndex)
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	bytes, err := json.Marshal(modelResult)
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	err = pm.PutObject(payload.Outputs[0], bytes)
	if err != nil {
		pm.LogError(wat.Error{
			ErrorLevel: wat.ERROR,
			Error:      err.Error(),
		})
		return err
	}
	return nil
}

On Line 44, an instance of a seedGeneratorModel (developed based on the datasource of the seedgenerator.json file) and the compute of that model instance is called with the input of the event number. The model of the seed generator is completely decoupled from the cloud compute framework and operates only on its own data model and inputs of standard objects (in this case an int). It produces an output, which is then provided to the SDK with the output datasource that defines where the CloudCompute has arranged for outputting the model result so that the result can be stored where other plugins can access it.