District Cloud Migration Phases

This roadmap outlines a generalized approach to cloud migration intended to provide a consistent framework across all offices. While it serves as a guiding structure for planning and execution, the specific activities, timelines, and sequencing within each phase may vary based on the unique technical and operational requirements of each individual office. 

Tracking Dashboard

Progress can be tracked using this PowerBI Dashboard.

Phase 1: Modernization (CWBI-DEV)

District staff are responsible for completing Steps 1 through 3 within Phase 1, including the prerequisites Tasks 1 through 5 in this checklist.

Step 1: Convert/Test All Data Processes to Use Cloud Ready Technologies on T7

  • Task 1.1 – Converting all custom scripts running from the server to use cwms-python/CDA/OpenDCS using the script decision flow chart as a guide.
  • Task 1.2 – Use the new getUSGS scripts in cwms-cli to get timeseries data/ratings from the USGS on T7 server (if applicable) (USGS timeseries Tutorial, USGS Ratings Tutorial). 
  • Task 1.3 – Run the critimport function to import critical information into the database for processing SHEF files on local T7 server (SHEF Import Tutorial). 
  • Task 1.4 – Test the python shef-parser package to process SHEF files on local T7 server (SHEF Import Tutorial).
  • Task 1.5 - Test the python shef-parser package to create any SHEF files for export. (SHEF Export Tutorial).
  • Task 1.6 - Verify you are running the latest OpenDCS database schema and update OpenDCS and relevant external processes to write directly to the database.  

Step 2: Implement Custom Data Processes in CWBI-DEV and Prepare for Migration

  • Task 2.1 – Load database objects (Locations, Timeseries IDs) into CWBI-DEV database using cwms-cli (Data Workshop Session 2).
  • Task 2.2 – Run all custom scripts (data processing and reporting) in the CWBI-DEV environment batch process and fully test that they are working as intended (Data Workshop Session 2).
  • Task 2.3 – Compile a list of all data products using LDM (receiving or sending) including file names or regex expressions for grabbing the correct data. 
  • Task 2.4 – Identify any custom database schema functions/processes that will be needed in CWBI. 

Step 3: Test Models in HEC-RTS/CWMS-Vue using CWBI-TEST

  • Task 3.1 – Setup test users in CWBI-TEST (User Setup Tutorial). 
  • Task 3.2 - Test connecting to CWBI-TEST CDA using CWMS-Vue and test a model forecast run using HEC-RTS. 
    • CWBI-TEST currently has all district data being streamed using GoldenGate so you should be able to test operations.

Phase 2: Functional Testing (CWBI-TEST)

Step 4 - Migrate All Processes to CWBI-TEST

HEC, with support from the District, will complete all tasks in Step 4, which may include:

  • Migrating any required custom database Schema functions into the CWMS Database Schema
  • Adding all LDM data products to CWBI-TEST
  • Configuring offices for USGS data processes in CWBI-TEST
  • Migrating CWMS-Batch scripts to CWBI-TEST branch
  • Importing OpenDCS configurations

Step 5 - Fully Test All Data Processes and Models in CWBI-TEST

The District is responsible for completing all tasks in Step 5. HEC will work with District staff to determine the appropriate duration for testing. This includes:

  • Confirming all data processes are running successfully
  • Running HEC-RTS Forecasts using processes set up in CWBI-TEST

Step 6 - Fix Any Issues Found During Step 5

HEC, with support from the District, will complete all tasks in Step 6.

Phase 3: Final Implementation (CWBI-PROD)

Step 7 - Train Staff and Prepare for Final Implementation

The District is responsible for training internal staff on cloud workflows and conducting final testing.

Step 8 - Migrate All Data Processes to CWBI-PROD and Verify All Components Are Working

HEC, with support from the District, will complete all tasks in Step 8, which may include:

  • Adding all LDM data products to CWBI-PROD
  • Configuring offices for USGS data processes in CWBI-PROD
  • Migrating CWMS-Batch scripts to CWBI-PROD branch
  • Importing OpenDCS configurations

Step 9 - Turn off Goldengate and Migrate Historical Data to National Database

HEC is responsible for completing all tasks in Step 9, which may include:

  • Turning off Goldengate connection from T7 to the CWBI-TEST database
  • Importing time series data for the past 2 years into the CWBI-PROD database
  • Migrating all historical data into the CWBI-PROD database

Phase 4: Post Migration

Step 10 - Decommission T7 Server

District staff is responsible for completing Step 10.