System Overview



Python Libraries

  1. CWMS-Python: A wrapper for CDA. A way for the district to read/write into the CWMS database.
  2. DSS-Python: Package to interact with DSS in python.
  3. CWMS-CLI: Command line utility to run cwms functions (i.e. run USGS scripts, save file to BLOB, setup scripts for SHEF imports and USGS ratings script).
  4. SHEF-Parser: Package to interact with SHEF files including parsing SHEF files and storing into CWMS database or DSS.  Can also extract data and store into a SHEF file.
  5. HEC-Python: This package provides many of the capabilities of using Jython to access HEC's Java class libraries. The library is focused on working with time series objects and associated infrastructure:

OpenDCS/CCP

The Open Data Collection System (OpenDCS) is an open-source software is used primarily for collecting, decoding, storing, and distributing environmental time-series data. 

Cumulus

Cumulus serves as the centralized, national gridded data service for CWMS. It collects, archives, and delivers real-time and forecasted meteorological data—primarily precipitation, temperature, and snow—from the National Weather Service and other trusted sources. Users can access and download data on demand via the Cumulus website or through the extract process integrated within HEC-RTS.

A2W

The Access to Water (A2W) is a CWMS Data Dissemination website. It is a publicly accessible, map‑based portal that aggregates water resources data from all USACE districts into a single site. It provides near‑real‑time elevation, precipitation, storage, flow, and stage information USACE reservoir and lock and dam projects.. The site offers simplified charts and graphs so the public and other stakeholders can easily view project status. Users can also navigate to linked local project websites, where available, for more detailed information. 

CWMS Database

The CWMS database is for Corps offices to store time series and relational data used in modeling CWMS watersheds. Oracle was chosen as the database facilitator because it is a widely used database and follows the corporate guidelines for USACE. The CWMS system provides a variety of ways to interact with the CWMS database, including PL/SQL, CDA, and various application programs. The CWMS database schema also stores procedures used to write and receive data to the database.

CWMS Data API

The CWMS Data API (CDA) is a Representational State Transfer (REST) API that enables secure data exchange between computer systems over the internet. CDA serves as the critical interface for transferring Water Management data to and from cloud environment. It allows users to send and receive information through various methods over a secure internet connection. The API supports authentication, granting write access only to authorized users while allowing the public to view publicly available data. 

CWMS-Batch

CWMS‑Batch is a tool that allows districts to run scheduled custom Python or Java scripts using a standard container image that includes the required Python and Java modules. It is triggered by Airflow to run jobs on an hourly or daily basis, and districts can add, update, or remove their scripts through district‑specific GitHub repositories (such as the cwbi‑jobs repos). An API is available to trigger these processes from a workstation or in response to events.

CWMS-Batch-Event/Jobs

An API was added for CWMS-Batch that allows it to be accessed and run from the workstation. District staff can individual scripts in a manual way outside of the Airflow - CWMS-Batch workflow. 

Airflow

Apache Airflow is an open‑source, pure‑Python platform used to programmatically author, schedule, and monitor batch‑oriented workflows through a web‑based UI that provides full visibility into completed and ongoing tasks. Originally developed by Airbnb and now managed by the open‑source community, Airflow was originally intended in CWBI to run all district‑centric scripts (Python workflows, or DAGs) in the cloud environment, similar to how cron is used on the T7 servers, but this responsibility will now be handled by CWMS Batch instead

LDM

The Local Data Manager is an event‑driven data distribution service that connects to NWS River Forecast Centers and USACE district offices to send and receive forecast and other data files for both Cumulus and the CWMS database. It uses the SHEF‑Parser to format incoming data, runs scripts from CWMS‑Batch that are scheduled by Airflow, and in CWBI it retrieves gridded products for Cumulus as well as NWS files for further processing, currently operating on T7 servers.

CWMS Monitoring

Grafana is an open‑source data visualization and analytics platform used to monitor, query, and analyze metrics, logs, and traces from many different sources. In the CWBI environment, Grafana is used to monitor CWMS process and service performance, including uptime and health for A2W, CDA, Cumulus, CWMS‑OpenDCS LDM, Airflow, and the execution of CWMS‑Batch jobs.

Web Data Entry:

CWMS web data entry is the browser-based interface USACE water managers will use to enter, review, and update time‑series and project data directly into the Corps Water Management System.