CWBI Migration Plan Developed in 2021

(updated 9Mar2022 with Modini/McPherson responses to comments by Smith/Dunn)

(Updated 30Apr2022 with Feingold, changed RADAR for CDA)

Executive Summary

This document describes a plan to modernize components of the Corps Water Management System (CWMS) and Water Management Enterprise System (WMES) and migrate to a modern cloud-based architecture.  The plan includes discussion of the benefits of migration, the concept of a minimally viable product, water management software components, what work will look like in the cloud, migration strategy including consideration of always meeting the water management mission, discussion of pilot projects, and a highly-uncertain estimate of schedule and costs associated with software development in support of cloud migration.   Schedule and costs associated with District implementation and capacity-building within the WM community remain to be estimated.

Modernizing these systems allows using the benefits of cloud computing.  This is a large project that will continue for several years.   The scope of the cloud-based modernization exceeds any improvements since the original CWMS design sessions of the late 1990s.   The immediate migration goal focuses on the software and services required to move work away from the dedicated Solaris T7 systems housed in each District.  This includes: changing from the pattern of databases for each district to a single centrally managed database, updating software to use web-based communication, moving district specific data processes from district owned hardware to the cloud, replacing software that only works on what will be obsolete hardware, creating new web services to organize the flow of data, and re-training employees to use new software and workflows.   Although this document focuses heavily on achieving sufficient cloud-compatibility to retire existing hardware, note that ongoing modernization efforts also include more fundamental re-design and re-imagining that will lead to powerful cloud-native capabilities.

Introduction

CWMS and WMES comprise interrelated software applications that work together over the network, using a client-server architecture, to support the water management community.  The server-side components of CWMS currently reside on in-house hardware that is rapidly approaching end-of-life (for convenience this document will simply refer to the whole collection as "CWMS").  The migration plan addresses the server side of the migration and provides the water management community an opportunity to evolve CWMS to leverage a more modern cloud-based computing architecture.  The ability to purchase a replacement server will not available and USACE Senior management has mandated CWMS move to the cloud.  Given the mandate, CWMS will migrate to a central cloud-based deployment, thus allowing the ability to leverage the built-in reliability features of the cloud, as well as streamline workflows, increasing efficiency and cost effectiveness.  

In order to fully leverage the cloud, CWMS needs to move away from the legacy server-based model.  Cloud resources are only more efficient when applications are designed to fully leverage the capabilities of scale that the cloud provides.  In this document, the "cloud" refers to the CWBI (Civil Works Business Intelligence) AWS cloud infrastructure. It is recognized that cloud vendors may change over time and care is being taken to avoid over-fitting to the vendor, while still leveraging services that make CWMS cloud-forward.

The CWMS modeling components are not addressed with detail in this migration plan.  Cloud-based model execution is an important aspect of WM software modernization, and design investigations are underway, but the immediate stage of migration planning assumes forecast modeling to continue on user’s Windows desktops.  

Since CWMS was not originally designed for the cloud, each component of CWMS will each be evaluated, then redesigned or retrofitted in these phases:

• Migration
• Modernization
• Maintenance and enhancements into the Future

The key components to be worked on:

• Cloud hosted databases and other services will be accessed through a modern web service Application Programming Interface (API)s.   
• CWMS applications that process, manipulate, and display data will be modified or re-written to work with the APIs.    
• The core server data processing programs, that are now on Solaris computers, will be replaced with cloud based equivalents.

What are the benefits?  How will it be better?

Having the database in the cloud allows USACE to leverage cloud features that take care of hardware, backups, reliability-failover, and scalability.  COOP will take on a different meaning as we shift data, computations, and reports away from racks in District offices to cloud-hosted services, and re-assess potential failure modes.

Some benefits of centrally hosted software are:

• Consolidate separate District-level data and processes into centrally-hosted and community managed systems.
• Software versioning of cloud components is not the end users responsibility.
• National support for data and APIs.
• District expertise, and custom processing will be shared with the greater visibility of the community system.
• Scalability and computing power opportunities
• opportunities across USACE portfolio of other systems.

The cloud environment will allow retiring individual office servers that are "end-of-life".  This needs to be done quickly because the SPARC T7 hardware is not recommended by any major software company.  Linux with commodity hardware is the preferred dominate choice for cloud computing. There will be less maintenance headaches because the hardware is handled by the cloud hosting company.  Figure 1 shows a conceptual picture of the CWBI cloud. 

Figure 1. CWBI Cloud overview

The migration phase will allow the existing CWMS oracle database to run in the CWBI cloud with minimal changes to Java client programs.   Longer-term moving to another database is desirable; and the Cumulus project demonstrated that the database PostgreSQL works well in the cloud.  However, in the short-term using the existing database allows re-using, without the need to rewrite, 430,000 lines of SQL code inside Oracle supporting multiple features.  The CWMS web API named CWMS DATA API (CDA, formerly known as RADAR) is taking advantage of existing Java database access code and because of this CDA is written in Java.

The current client software communicates directly with Oracle, so adding an API between the database and the client apps is a critical modernization step.  Client software is being modified to allow connecting to the web service APIs on the cloud boundary, while preserving the ability to connect to Oracle on T7s.  This will allow testing incremental changes in the cloud but have the ability to use the T7's during the migration before retiring their T7s.  Migrating to an API allows subsequent changes to the database, as dramatic as changing the database itself, or using two databases to share the work, all while having minimal effect on the users or client software.

The data processes performed by each office on their respective T7’s have a wide variation of needs, with the most common needs addressed first as described in the section: Components of Water Management Software.

More of the migration details, such as keeping the T7s running in order to ensure continuity of water management operations, are described in the Migration Strategy section of this document.

What is the minimally viable product?

The Minimally Viable product (MVP) will allow USACE staff to perform the water management mission without in-house servers.  This mission includes the ability to send data to A2W (national web site/service for CWMS data), and run CWMS models to create forecast results.   Other tasks such as hosting local district data processes will be investigated but not all solved during creation of the MVP.

Moving CWMS to the cloud is a large software project.  The first goal is to create a secure working minimal product as soon as possible, with the most expedient path away from the end-of-life hardware.  A DevSecOps approach will be used allowing experiments with both retrofitting existing software and, abandoning old technology that prevents a clear path to a working product.  The path will be informed by the following in progress efforts on these components, that have some relationship to CWBI or migrating CWMS to the cloud:

• CWMS RDS Database/CDA running in the CWBI production Cloud
• CWMS desktop software updated to use CDA web service
• Access to Water (legacy and modernized)
• Meteorological gridded data services, snow, precipitation, etc.  - Cumulus
• Cloud based calculations and reporting
• Data acquisition for all required data sources
• Uses CWBI Authentication

Components of Water Management Software

Individual offices will not manage a server, instead they will manage and work with data through services.  Some legacy programs running on an office specific server will have a cloud equivalent. Others programs will be retired.   For example if an office performs satellite message decoding with a legacy Fortran program, that will not be supported in the cloud.  Instead that block of work will require migration to a news tools such as OpenDCS. 

The ability to rely on “enterprise-supported” data sources will be a significant convenience to District offices.   Enterprise data sources will initially focus on those with the widest coverage and heaviest usage   i.e. national and certain regional data sources.   Districts will still need to handle data sources with more local scope.    HEC/CRREL will facilitate migration of work relying on existing enterprise-supported tools, such as traditional CWMS data streams and CWMSVue scripts, by providing new generalized helper tools and training.    For example, District programs that connect directly to Oracle must be converted to use the web API’s of CDA or A2W.

The trend of IT organizations ensuring security and restricting what is allowed on servers and local windows computers will be more pronounced in a cloud environment.  The benefits range from IT department being satisfied to simplified ability to have Continuity of Operations Plans (COOP).

Table: Databases, Servers, Client software, data flow

How do I work in CWBI?   

As CWMS is modified to use cloud services USACE employees will be learning new workflows, while maintaining their existing CWMS systems.  Software Developers and System Administrators will have a different workflows to get their job done.  Modelers and others that will be using cloud based data will need to re-configure their existing tools, or learn new tools.  For example, CWMSVue is being modified to support web based data access, through a web service API, instead of communicating directly with an Oracle database.  Other needs such as web reporting could be accomplished with live web based apps reading data from the same API.   Since the biggest workflow change will be for Developers and Administrators this section will focus on that.  Operational support will be available through community based PDTs.

The Users, which use the data include modelers, public users downloading data, employees making configuration changes (e.g. changing a DECODES script).  Developers are those who create and improve the services using programing languages and system configurations.  Developers, may work on products for all CWMS Users or for a single District.  Developers may be doing work on the baseline USACE wide components (CWMS, CDA, Cumulus, etc.) or a local staff Developer may create and maintain the automated retrieval of data for a specific need from a local cooperator.  A given member of the Water Management community may be both a Developer and User.

Traditionally in the WM community develops on running production systems, with little to no isolation between development and the live system.  Developers won't login to a server in the CWBI environment.  Instead the CWBI environment creates a much more structure approach to software development.  Programmers will develop using the exact environment that will be used in production - this is possible using container technology called Docker. 

There are distinct CWBI environments corresponding to the development phases:  Local development , development cloud environment, test could environment, and production cloud environment.  The source code will move through these environments using git repositories for source code.  More people can see and share code and services because, by default, it will be developed in the open on Github.  Each environment will be configured using a "configuration as code" approach which allows declarative definition of infrastructure specifications such as number of servers, memory requirements, database setup, and access control permissions. Doing so will increase the reliability, scalability, and reproducibility of the system.

Migration Strategy 

The migration of each T7 database and services will be done in incremental steps, where each step has checkpoints with questions like:  "Am I getting the same data and core functionality"?   During some migration steps the same data and processes will be available in two places (one official and one replicated), so the current authoritative data source will need be tracked and documented for each office.

There will not be a singular moment where CWBI is declared "production-ready" for all Water Management requirements and offices move off their T7's at the same time.   WM should anticipate that the functionality on CWBI will be developed and refined concurrently with individual offices migrating certain aspects of their operational requirements to the new environments.   The complexity and scale of the effort will vary considerably across the field offices.   A key aspect of CWBI migration will be to provide flexibility so that offices can manage their required adaptations without interruptions to mission readiness.   

Figure 2.  February 2022 - status of LRL migration

For instance, Figure 2 considers the present situation for the Louisville District (LRL).     Louisville relies on its T7 server to host most of its automated water management processes, and desktop computers to perform interactive work.   Gage data is acquired by  satellite or via USGS API.  A remote data entry system is maintained to gather data from projects, gridded data is acquired from Cumulus, non-gridded data is acquired from the Ohio River Forecast Center (OHRFC via Local Data Manager (LDM), and lock data is retrieved through web queries to the Lock Performance Management System.   Automated processes on the District T7 handle the data acquisition and associated validation and transformation calculations.  The results are stored in the District's CWMS database and forwarded to the existing national CWMS database and Access To Water.   LRL uses the CWMS desktop clients for correcting data problems, running forecasts, and posting results.   Other processes generate content for the district public webserver and various reporting requirements.   Note that LRL has already dropped its local processing of gridded data in favor of Cumulus.

Figure 3.  Incremental migration to cloud for LRL (later in 2022)

As functionality on CWBI expands and matures, LRL will migrate parts of its operations to new approaches as shown in figure 3.  As of February 2022,  LRL is starting to work with the national USGS data available from CWBI for CAVI forecasts.  As the district becomes familiar with getting gage data from CWBI, they will adapt their CAVI workflow and automated processes to use the new data source instead of the existing T7-based data acquisition mechanisms.   At some point LRL, will abandon the old DECODES and DOMSAT mechanisms in favor of the CWBI-based data source.   Data acquisition from National Weather Service (NWS) sources is also underway in CWBI, and the same pattern will apply so that the existing OHRFC data acquisition mechanisms can be discontinued.   Concurrently, other work underway will make more data available in CWBI.   The existing CWMS national database is being replicated in CWBI, together with the accompanying CDA web service.   The legacy A2W is being moved to CWBI, while an expanded and modernized version of A2W is under construction.   As the data becomes available in CWBI, LRL can begin re-inventing the automated processes currently hosted on its T7 according into cloud-compatible (or cloud native!) operations to accomplish portions of its operational computations and reports.  CAVI-based forecast modeling will remain on their desktop computers but extracts and read-only CWMSVue work can shift from connections to the District Oracle database towards CWBI-based web services.   However, saving changes with the Data Validation Editor or posting CAVI forecast results must continue relying on the T7-based database to ensure meeting mission requirements.    

Figure 4.  Minimally Viable Product (MVP) for LRL - T7 servers are retired

As the capabilities develop on CWBI, the load on the LRL T7 will diminish.     CWBI will support data acquisition from remaining sources.   Edits and posts can be performed against the CWMS database in CWBI when write-access to the CWMS database is available through CDA.  At some point LRL will decide that the CWBI-based processes are sufficient and retire its T7.

In-Progress Pilot Projects

Three offices are participating in pilot efforts to determine how to perform operational water management work using CWBI-based resources and processes.   Savanah District (SAS) and Wilmington District (SAW) do not have their own server hardware and are willing to assist with CWBI development and migration similar to their motivations when working with CPC-based servers and databases.    LRL is very motivated to learn how to perform its work in the cloud and offers a slightly larger scale and complexity of requirements. 

The existing read-only national CWMS database is running in the CWBI development environment with real time updates from every district T7.   Moving that to CWBI production with enabled write access via CDA is the most urgent goal.  This is urgent because writing data through a web server allows developers and users to incrementally move data sources, calculations, and other functionality in small projects.

Pilot Project Updates:

• Cumulus use is increasing and is now used by several offices
• CDA is running in the CWBI development area.
• Two offices (LRL and SAS) have demonstrated using CWBI data feeds in pilot projects
  • Reading gridded data from CWBI/Cumulus
  • using A2W API to retrieve USGS time series data.
• ORACLE is running in CWBI development area using Amazon RDS database technology (this is the existing read-only National CWMS DB).

Estimate of Migration Costs

With respect to costs, there will be cost savings for not purchasing hardware on a regular cycle, and staffing the associated coordination with G6 and IT service contractors. Short-term training and labor costs for migrating to CWBI while maintaining the T7 servers could go up. In the long-term, labor costs may decrease or stay the same.  The kind of work that system administrators will perform will be different.  There may be more or less work and expense on the cloud compared to the current WM experience, but the costs of continuing to rely on the legacy hardware and software are likely to escalate – if allowed to continue at all. As this migration progresses more will be known about these costs which will include input from the field.

An estimate of migration costs were developed based on the phases of the project and items that comprised each phase and are presented in table 2. Project management costs were also included. Costs were also broken out by fiscal year. The costs in the table are assumed to be borne by CWMS and WMES AIS funds. CRREL will provide in-kind assistance for CWBI related activities such as CWBI technical advice, configuring CWBI environments, coordinating with the AWS provider, onboarding of CWMS in the CWBI production environment, and other similar tasks.  

It is estimated that the total costs to migrate, modernize, and have one year of maintenance and enhancements is approximately $10,478,000. This is for the period of FY2022 – FY2026.  The breakout by fiscal year for 2022 – 2026 is $1,062,000, $1,816,000, $1,730,000, $1,260,000, $860,000, and $3,750,000 respectively. These are shown in table 3.   Please understand that these figures are highly uncertain.    Much technical planning remains in an exploratory phase of finding out what we don’t know.  

In table 2, the costs highlighted in yellow represent the budget necessary to achieve a Minimally Viable Product total. The total is $4,078,000. This would be achieved by the end of FY2025 and the breakout by fiscal year for 2022 – 2025 is $1,062,000, $1,216,000, $1,130,000, and $660,000 respectively. These are shown in table 3. 

Table 2. Estimate of Migration Costs

Table3 – Breakout of Total Project and MVP Costs by Fiscal Year