IMPORTANT: CWMS 3.2.1 and HEC-ResSim 3.5 are now using Jython 2.7 and 2.7.2 respectively. Scripts running on your clients, server, or as part of your ResSim model may require changes.   See the sections below and the ResSim section in Modeling Software Upgrade Notes for CWMS 3.2.1 for details on what to expect and what to do.

There are two important issues to note regarding scripts running under CWMS 3.2.1, on the clients and servers.

  • Most significantly for CWMS 3.2.1 and HEC-ResSim 3.5 - when importing Java classes into a Jython script, wildcard characters in import statements are ignored.    Details are below.
  • DataStreamUtils.py on the server has implemented Python 3 (see below).   

Wildcard Imports

To understand the issue, we'll start with some terminology:

Java Package

"A package is a namespace that organizes a set of related classes and interfaces. Conceptually you can think of packages as being similar to different folders on your computer. You might keep HTML pages in one folder, images in another, and scripts or applications in yet another. Because software written in the Java programming language can be composed of hundreds or thousands of individual classes, it makes sense to keep things organized by placing related classes and interfaces into packages." https://docs.oracle.com/javase/tutorial/java/concepts/package.html

Java Class

"In the real world, you'll often find many individual objects all of the same kind. There may be thousands of other bicycles in existence, all of the same make and model. Each bicycle was built from the same set of blueprints and therefore contains the same components. In object-oriented terms, we say that your bicycle is an instance of the class of objects known as bicycles. A class is the blueprint from which individual objects are created." https://docs.oracle.com/javase/tutorial/java/concepts/class.html


Jython

Jython is a Java implementation of Python, an interpreted, high level programming language that supports objected-oriented and procedural programming. Because Jython has been implemented in Java, it can understand and use existing Java classes and objects.

Jython ScriptA Jython script is a set of instructions (a program) written in the Jython programming language and executed through the use of a Jython interpreter which is embedded in the CWMS CAVI, DSSVue, CWMSVue, ResSim, and other CWMS and HEC software applications.  
Import statements

A statement or instruction to the Jython interpreter to find and load a specific class, package, or module written in Jython, Python, or Java so that it can be used later in the script.  The syntax of an import statement has two basic forms:

import package- This is the most basic type of import, where package identifies a package or module which you want imported in its entirety

from package import name - This form of import, where name identifies a package, class, or function nested in the class, package, or module identified by package.

from package import name as newname - This form of import, where name identifies a package, class, or function nested in the class, package, or module identified by package but allows the user to use a different name, newname, for the imported item.
    Example: from hec.script import MessageBox as Box

wildcardIn the Python language, a * can be used in place of name in an import statement to import all the "names" (packages, classes, etc) in package.   * is a "wildcard". "This form of import is discouraged in the Python community, and is particularly troublesome when importing from Java packages (in some cases it does not work) so you should avoid its use. " per https://jython.readthedocs.io/en/latest/ModulesPackages/#types-of-import-statements


The Issue

Wildcard imports, such as from hec.hecmath import *, are no longer fully supported, and can cause name errors and other runtime exceptions.  In many cases, this change makes it appear that import * statements are ignored. The Jython and Python websites have known issue dialogs describing this situation.  These dialogs suggest that import * works under some conditions but not others.  Specifically, static classes, members, and methods are not found with import * and require explicit imports.  Some background information is available at https://discourse.hecdev.net/t/wildcard-imports-bug-in-jython/408.

For example,

from java.util import *
Arrays.sort(x)

produces:

Error during compute java.lang.RuntimeException: Traceback (most recent call last):
    File "<script>", line 2, in <module>
NameError: global name 'Arrays' is not defined

In the example and its result above, Arrays is a static class contained it the java.util package, but because of the change in how the wildcard imports are treated, the class Arrays was not fully or properly imported.

As a result of this change in Jython, your scripts should be updated to explicitly reference the packages, modules, classes, or class members needed.  Although there are work-arounds available, explicit imports are generally considered better practice.   They may also result in faster loading of your modules and classes.   The performance increase can be significant in certain cases where your scripts are invoked repetitively (e.g., HEC-ResSim state variable and scripted rules).   

The main drawback to creating explicit imports is the additional effort it takes to identify the required modules or classes.   There are two main approaches that we recommend:

  • You can comment out the current wildcard import statements and paste in the list of common imports below. This method is the quickest to get most CAVI scripts working but can result in slower run times (particularly if you do not remove unnecessary imports).
  • Or, you can follow the method described in "Process for Correcting for Wildcard Imports". This method may take a little longer but will result in the most efficient list of imports.

The following list of import statements were collected from one or more field scripters. You can copy and paste this list into your CWMS scripts.  Be sure to comment out your current import statements that use wildcards.

from decimal                 import Decimal
from hec.data.cwmsRating     import RatingSet
from hec.hecmath             import TimeSeriesMath
from hec.io                  import TimeSeriesContainer, PairedDataMath
from hec.heclib.dss          import HecDss, DSSPathname
from hec.heclib.util         import HecTime
from hec.dataui.tx           import TsContainerDataSourceList
from hec.dataui.tx.awt       import VerifyDataDlg
from hec.dataTable           import HecDataTableToExcel
from hec.data.cwmsRating.io  import TableRatingContainer
from hec.dssgui              import ListSelection
from hec.script              import MessageBox, Constants, Plot, Tabulate
from java.awt                import BorderLayout, GridLayout, FlowLayout, Toolkit, GraphicsEnvironment, Rectangle, Color, Font
from java.awt.event          import ActionListener, FocusListener
from java.io                 import FileOutputStream, IOException, TimeSeriesContainer, PairedDataContainer
from java.lang               import System
from java.text               import SimpleDateFormat
from java.util               import TimeZone
from javax.swing             import JDialog, JCheckBox, JComboBox, JPanel, JButton, JOptionPane, JScrollPane, BoxLayout, JLabel, JTextField, SwingConstants, ScrollPaneConstants
from javax.swing.border      import EmptyBorder
from rma.services            import ServiceLookup
from rma.swing               import DateChooser
from time                    import mktime, localtime
from timeit                  import default_timer
import DBAPI, datetime, time, calendar, inspect, java, os, sys, traceback, math, shutil, logging, getpass
import javax.swing.JFrame
import java.lang

With the above imports included in your script. Save and run your script.  If your script runs correctly, it is strongly recommended that you go through and comment out unnecessary imports. Leaving those extra imports in is inefficient and will slow down the loading of the script.

Other resources for common imports can be found at Scripting DssVue or HEC's GitHub.

There IS a Work-Around - but...

In prior Jython versions, the inefficiencies associated with loading all classes from a given package (import *) rather than just the classes or modules needed by the script were offset through the use of caching so that repeated executions of the same script would not need to reload the whole set of classes with each execution.  With Jython 2.7, this caching has apparently been disabled and the use of a wildcard on import (e.g. from myPackage import *) is now not processing everything in the package the way it used to.

However, for backward compatibility, the python.cachedir.skip=false command-line option is available; this option will allow the jython interpreter to believe that caching is "on" and process the package completely.  Although not recommended, to use this option, the following line can be added to the CAVI.config file or your CAVI-Personal.config file to enable the use of import * in your Jython scripts. 

vmparam -Dpython.cachedir.skip=false

NOTE: If you choose to use this work-around, PLEASE consider it a stop-gap until you can revise each of your scripts and remove the wildcards from your import statements. DON'T just set-it-and-forget-it!

ResSim State Variables and Scripted Rules

Like CWMS 3.2.1, ResSim 3.5 includes an upgrade to Jython 2.7.2 and has the same issues with wildcard imports.  However, unlike CWMS, the workaround described above should NOT be used.

The ResSim development team tested the use of the work-around with HEC-ResSim 3.5.  They found that although ResSim state variable scripts and scripted rules that include import * statements would appear to function correctly, when import * is encountered in a script, the whole set of classes from the indentified package  (e.g., from hec.script import *) are loaded with each execution of the script; in other words, no caching is being used.  Since the main script of each state variable and scripted rule may be executed two or more times per time step, per compute pass, if even one import * statement is used, the increase in compute time can be very large. As a result, the ResSim development team recommends that the above workaround SHOULD NOT be used with ResSim.   Instead, the team strongly recommends that all wildcards used in import statements be replaced with an explicit list of the classes needed by each script. 

Similar to CAVI scripts, the two main approaches for creating explicit import statements that we recommend are:

  • You can comment out the current wildcard import statements and paste in the list of common imports below. This method is the quickest to get most state variable and scripted rule scripts working but can result in slower run times (particularly if you do not remove unnecessary imports).  This list is smaller than the common CWMS imports list above since ResSim scripts usually have a different purpose or focus.  However, if it doesn't work, try the longer list or try the next method.
  • Or, you can follow the method found in "Process for Correcting for Wildcard Imports". This method may take a little longer but will result in the most efficient list of imports.

from hec.client     import ClientApp
from hec.heclib.dss import HecDss, DSSPathname
from hec.heclib.util    import HecTime, intContainer, doubleContainer
from hec.data           import Parameter, Units
from hec.model      import RunTimeStep, PairedValuesExt
from hec.hecmath        import TimeSeriesMath, PairedDataMath
from hec.script     import Constants, Plot, ResSim
from java.io            import File
from java.util.prefs    import Preferences
from javax.swing        import JFileChooser, JOptionPane
#from zipfile           import ZipFile
#from __future__        import with_statement
#from bisect            import bisect
#from decimal           import Decimal
#from hec.rss.model import OpValue, OpRule #Used only in scripted rules.
 
import inspect, os, sys, copy, math, calendar

Pure Python

DataStreamUtils.py was updated for python 3 compatibility. NOTE: it was tested for import, and the database username/password retrieval. Other functions may have issues in python3.x.

DataStreamUtils.py was updated with a new function (getDbUserPass) to retrieve the username and password for the cwp user. Here's is an example of the use from shefloader. Districts scripts can and should use this to avoid hardcoding the username and password into their own scripts

...
scriptPath = os.path.join(os.environ["CWMS_EXE"], "bin")
if scriptPath not in sys.path : sys.path.append(scriptPath)
from DataStreamUtils import *
...
cwmsHome = os.environ["CWMS_HOME"]
dbiPropFilename = os.path.join(cwmsHome, "config/properties/dbi.properties")
dbiProperties = getProperties(dbiPropFilename)
dbiConfFile = os.path.join(cwmsHome,"config/properties/dbi.conf")
self.dbSpec = dbiProperties["cwms.dbi.ConnectUsingUrl"].split("@")[1]
dbHost, dbPort, dbSid = self.dbSpec.split(":")
dbPort = int(dbPort)
pdUser,pw = getDbUserPass(self.dbSpec,dbiConfFile)
dsn = cx_Oracle.makedsn(dbHost, dbPort, dbSid)