SHEF Parser and SHEF File Processing (SHEF FILE -> CWMS Database)

Overview

This tutorial shows how to use the SHEF Processing / shef-parser package to:

  • Parse SHEF files
  • Import crit mappings needed for ingest
  • Load parsed time series into a CWMS database using the CWMS Data API (CDA) loader
  • Run the workflow from the command line or Python

References

1) Install shef-parser


python3 -m pip install shef-parser


 

2) Verify the CLI works


shefParser --help


 


Commonly used CLI arguments

Based on the help output in the figure:

  • -i / --in: input SHEF file
  • -o / --out: output file (optional)
  • -l / --log: log file (optional)
  • -f {1,2} / --format {1,2}: output format selection
  • --loader <ts_loader>: load parsed time series to a destination (e.g., CDA)
  • --loglevel {DEBUG,INFO,WARNING,ERROR,CRITICAL}: control verbosity

3) Prerequisite: Configure SHEF Data Acquisition mappings (crit)

Before loading SHEF into CWMS, you typically need crit mappings imported and associated with the SHEF Data Acquisition time series group.



What you’re accomplishing

  • You are ensuring there are defined CWMS time series identifiers that the ingest process can write to.
  • The crit import step below populates/stores these mappings in the CWMS configuration so the loader can resolve incoming SHEF observations correctly.

4) Import a crit file with cwms-cli

4.1 View crit import help


cwms-cli shefcritimport --help


 



4.2 Import the crit file


cwms-cli shefcritimport -f CEMVP_GOES.crit -o MVP -a $CDA_API_ROOT -k CDA_API_KEY


Where:

  • -f: crit file path
  • -o: CWMS office
  • -a: CDA API root URL
  • -k: CDA API key (or key source, depending on your environment)



5) Load a SHEF file into CWMS using the CDA loader

5.1 Command line load


shefParse -i filename --loader cda[$API_ROOT][$API_KEY]

 

shefParse -i your_input.shef --loader cda[$CDA_API_ROOT][$CDA_API_KEY]


 What to look for in the output

  • Startup line showing the parser version
  • Loader initialization (CDA loader)
  • Warnings related to SHEFPARM adjustments can be normal (depends on configuration)
  • “Stored … values in …” lines indicate successful posts



5.2 Load from Python


import os 
from shef import shef_parser 

cda_url = os.getenv("CDA_API_ROOT") 
cda_api_key = os.getenv("CDA_API_KEY") 
shef_parser.parse( input_name=input_filename, 
                   loader_spec=f"cda[{cda_url}][{cda_api_key}]", )


You provide:

  • input_filename: path to the SHEF file you want to ingest
  • CDA_API_ROOT: CDA base URL
  • CDA_API_KEY: API key used to authenticate

6) Quick checklist

  • Install shef-parser
  • Confirm the SHEF Data Acquisition time series group exists
  • Import your .crit file using cwms-cli shefcritimport
  • Load SHEF using:
    • shefParse ... --loader cda[...] (CLI), or
    • shef_parser.parse(... loader_spec="cda[...]") (Python)
  • Verify the ingest by checking:
    • terminal output “Stored … values …”
    • the CWMS time series group / database values (per your normal validation process)