District Preparation for Cloud Migration

Last updated  

As CWMS is preparing to move all of the CWMS data to the cloud, many Districts can begin to prepare your data and your Database Landscape to make this transition easier.  This page contains a list of things that Districts can start to do today to make this transition smoother when it is your turn to migrate to the cloud. This list is subject to change as pilot Districts make your move during the first half of 2024. 

Database clean up and system changes

  • Begin using the CWMS Data API (CDA) now.  
      • Use it in extract lists in the CAVI (CWMS 3.3.1)
      • Use it with websites
      • Use it for reporting
    • By using CDA in as many processes as possible, Districts can identify shortcomings of hosting the data in the cloud.  You can begin to identify what will have to change in your scripts and your websites to make use of the data as well.  
      • Any issues found with CDA should be identified on GitHub, please include any and all scripts/descriptions/logs/and screen shots available when posting issues. 
    • CDA will be the Primary Entry and Egress point of data into and out of the Cloud.
  • Ensure that all Processing times are in UTC or GMT time zones.
    • To display other time zone options please utilize the CAVI, CWMS-Vue, or scripts. 
  • Review your data in your District's database, and remove any data that is no longer needed; add data that you are missing.
    • Clean up your database names to follow the proper naming convention
    • Any data located in DSS files, or other sources, that are not also in your database should be migrated to the database.
    • Ensure all the data (including Historic data) you want/need is located inside your database.
  • Clean up any Database processes that are operational on your database.
    • Review any specialty code and make it the most efficient it can be.
    • Become familiar and start working with Git to establish version control for scripting.
      • See GitHub for webinars and additional information.
    • If you have any specialize SQL - investigate if it can be moved to CDA.
      • For assistance with this step, contact Mike Neilson and Eric Novotny. 
  • If your District uses Team Forecasting, migrate to using a Network Drive as the location of that Watershed.

Period of Record (POR) for Access to Water Timeseries

  • All time series that are being reported to Access to Water should contain the entire period of record for that location and parameter
  • Merge all data into a single POR timeseries
  • Assign that time series to Access to Water as the official period of record time series for that location and parameter
  • The time series assigned should follow standard naming conventions.

Server side scripts/data acquisition/computations

  • If your District is using OpenDCS to save data in a data stream or other data format, convert it to save directly into the database. 
  • setup cwms-python and CDA on your server
  • Inventory all of your server-side scripts that are running in Cron 
  • Ask yourself the following questions: (a flow chart of this is available Here: Script Update Flow Chart)
    • Is this a computation?
      • Does it make sense to do in CCP?
        • convert to CCP: If existing computations/Algorithms in CCP do not meet your needs work with OPENDCS team to create a new computation/Algorithms
        • Add issue to openDCS github https://github.com/opendcs/opendcs/issues.
        • Please Note: "CCP" is the CWMS implementation element that exists within OpenDCS. So the GitHub you will be directed to is OpenDCS for issues and requests. 
      • If using CCP does not make sense for your needs then convert script to python 3 using cwms-python
    • Is this script related to your website
      • Work with the web team to convert script to new web framework
    • Is this script acquiring data and storing into CWMS database?
      • Can this be easily done in openDCS
        • Convert to openDCS
    • Any remaining script convert to python 3.9 or later using using cwms-python instead of DBAPI.
    • For exceptional situations where there is no other option, a container of some kind will be provide for use.
  • setup the new getUSGS_CDA.py script to run on your server to replace the legacy getUSGS script. instructions on setup are given here
  • For those using R or any other scripting language, begin using CDA to see how it will affect your District's scripts.
    • If the script cannot run in its current language with CDA , recommend converting it to Python 3.9 or higher. 
  • Begin to research and prepare to work in "Docker" if your district will require custom codes.
    • All custom codes and scripts will be required to be built in a docker container prior to deployment on the cloud.
  • Begin Knowledge Sharing with other Districts.
    • Many Districts have the same or similar processes.  Share how your District solved challenges, and ask others how they did it.
  • If unfamiliar with JavaScript and Python begin learning the basics.


Web Migration/Reports

  • Start working with the National Web Development Team to modernize your District's website to use a JavaScript framework.
  • Change all plots over to a JavaScript renderer (ex Plotly).
    • GFX2D plots will no longer be supported.
  • Begin updating custom plots, reports, and weblinks to point to CDA as the data source. 
  • Attempt to use the CDA instance on your District's T7 to save reports as a CLOB.