June 10, 2016
Water Data for the Nation changes, 2016
Changes are coming this summer to USGS water data. For most users, the changes will be minor or unnoticeable. Advanced users and data analysts should read on to understand how changes to some data fields and calculations could affect their work. Users who write or maintain software that uses these water data should read on.
These changes result from improvements to the underlying National Water Information System software used to collect, validate, quality-assure and store USGS water data. We have worked hard to minimize end-user impact, but there is some impact.
Most of these changes will we be implemented state-by-state over a 7 month period starting in late summer 2016. But some of these changes—indicated with a * in the list below—will be immediate, national, and should occur in July 2016.
Changes that affect only new data at a site:
- Short rolling outages in real-time feeds. As sites are switched to new underlying software, individual real-time data feeds might slow down or stall for minutes to hours. We will strive to minimize downtime, especially at sites experiencing emergency conditions.
- More Instantaneous Values (IVs) on estimated days. USGS will begin systematically publishing continuous (instantaneous values) data for periods of estimation. Previously, most estimates were available only as daily values and the instantaneous data were filtered.
- Minor rounding changes. Time-series and field-measurement data values may change in the number of significant figures, particularly around powers of 10 (e.g., from 9,999 to 10,000). The specific changes will vary by parameter and, in some cases, will be specific to a particular site.
- New partial-day remarks for daily values (DVs). Daily values that include a gap in data will now have a partial-day remark to indicate that they were computed from less than complete record. When the data gap is too large, no daily value will be calculated.
- Daylight saving time is no longer used for daily-values calculations. This means that in the summer months, where daylight saving time is observed daily values will be calculated from 1:00am to 1:00am in local time. Previously published daily values are not affected.
- More reference lines may appear on graphs. Graphs may begin showing more reference lines, particularly operational limits that never can be exceeded.
- WaterAlert subscriptions should be confirmed. WaterAlert subscriptions should be monitored. Email and text-message WaterAlert subscriptions should be unaffected, but users should report any problems to email@example.com
Changes that may affect all data at a site:
- Use of daylight saving time. Previously, some USGS sites reported data in year-round standard time even when daylight time was locally observed These sites may begin reporting all data in local time as the new software is deployed. Final decisions are made by each local USGS Water Science Center independently.
- Discontinued field-measurement columns. Rating number, shift adjustment, and percent difference columns will no longer be reported; they were inconsistently populated and potentially misleading.
- Longer text in some field measurement columns. Some columns, such as control condition and channel condition, have additional possible values and/or have been converted from using abbreviation codes to full text descriptions.
- Instantaneous values (IVs) before 2007-10-01 are now supported. The Instantaneous Data Archive (IDA) is being integrated into waterdata.usgs.gov and waterservices.usgs.gov. New remark codes, transferred from IDA, are associated with these older IV data to express how well they match published daily values. Pre 2007-10-01 data for other parameters will also begin appearing over time as configured by individual Water Science Centers.
Changes that might affect bookmarked URLs and software that uses water data:
- *Data descriptors (DDs) are replaced with time-series identifiers (TSIDs). DDs were used to discriminate between multiple time series of the same parameter (e.g., water temperature at multiple depths), although end users were discouraged from relying on it. The 3-digit DD has been replaced by a new 6-digit TSID. These TSIDs are considered internal and end users should continue to avoid using these numbers in their programming.
- *Column order may change for time-series tables of multiple parameters. The DD was arbitrarily used to order columns. With the switch to TSID, column orderings likely will change, affecting software that assumes a certain order.
- *URLs for time-series graphs may change. Bookmarks to standalone graphs may break if a site publishes more than one instance of the same parameter (e.g., water temperature at multiple depths). This is because the DD was used to disambiguate these, but now the TSID is used instead.
- *Comment lines in tab-separated (RDB) output may change. Some of the text on comment lines (lines that begin with #) will be in a different format or even removed. End users should avoid relying on information in comment lines in their programming.
- Some less common time-series parameters have changed parameter codes. Major parameters like discharge are still the same (code 00060), but some codes like stream velocity have changed (from 00055 to 72254)
- modifiedSince argument on waterservices may mislead, briefly. Software that relies on this argument to minimize the amount of data requested may see these data requests become much larger as locations are moved to the new underlying system. This should not last for more than a few hours at an individual site.