Identify and Fix Data Gaps

Avatar
Sumati Oraevskiy
Follow

The I&I Analysis Tool requires that all flow and rainfall data channels you plan to use are recorded in a consistent time interval, and are free of data gaps. This article walks you through the process of identifying data gaps, and standardizing the recorded time interval with the Interpolation function.

Checking for data gaps
When the channel receives data on a fixed time interval

Using FlowWorks' Advanced Calculation Engine (FACE), setup a Calculated Channel that uses the Delta Time calculation to record the time between subsequent data points. Next, graph the Calculated Channel to see whether gaps exist; and if so, see just how large (or small) these gaps are. Please follow the steps below.

  1. Set up a new calculated channel (for housing Delta Time values)
  2. Configure the new calculated channel to determine Delta Time between data points in the original channel
  3. Graph the calculated channel to visualize data gaps

1. Setup a new calculated channel for Delta Time values

1.1. Open the Configure Sites (FACE) tool
  1. Click Manage from the main toolbar.
    A drop-down menu appears.
  2. Click Configure Sites (FACE).
    configure-sites-face-1.png

1.2. Add a new channel

  1. Click Add New Channel.
    The Add Channel page populates.
fix-gaps-1.png
  1. Choose the site in which you want to store the delta time channel. Expand the Choose Site drop-down menu to make your selection.
    For easy reference, we recommend you store this channel in the same site as the originating raw channel.
  2. Input a Channel Name.
    We recommend you use the same name as the original channel, with "Delta Time" appended to the end.
  3. For Channel Units, select sec - Seconds.
    Data gaps are determined based on the number of seconds between two sequential values in the original channel.
  4. For Channel Type, select Calculated.
  5. Ensure Visible is checked.
    This enables you to plot this channel in FlowWorks' Graphing tool.
  6. Click Create New Channel.
    You are taken back to the "Configure Sites" page. Continue setup by following the steps below.
fix-gaps-2.png

2. Configure the calculated channel

Now that a calculated channel has been created, it needs to be configured to apply the Delta Time function to incoming data.

2.1. Open the calculated channel you just made

Continuing from the Configure Sites page, click View Calculated Channel.

A page opens that contains a list of calculations associated with the calculated channel. Since this is a new channel, such a list is currently absent.

fix-gaps-3.png

2.2. Setup the Delta Time calculation for this channel

  1. Click Create Calculation.
    The "Create Calculation" page appears.
fix-gaps-4.png
  1. Enter a Calculation Name and Description.
  2. Expand the Type menu and select Delta Time.
    The lower portion of the page refreshes to display the necessary fields for setting-up delta time.
  3. (Optional) Set a Start Date and End Date.
    The Delta Time function will apply to all data within this time frame.
    Leave this area blank if you wish to apply the function to the entire data set.
  4. Select the Source Channel. Use the first drop-down menu to choose the site within which the channel is stored. Then, use the second drop-down menu to select the source channel.
  5. For Ignore Null Values, choose No.
    When checking for data gaps across time, a null value is considered to be a valid entry.
  6. Click Save.
    The "View Calculated Channel" page loads, where the new calculation appears in a list.
delta-time-demo-2.jpg

3. Graph the calculated channel

By graphing the calculated channel, you can visually see whether data gaps exist in your data.

3.1. Open the Graphing tool
  1. Click Graph from the main toolbar.
  2. Click Graphing Tool.
    graphing-tool.jpg

3.2. Select channels for graphing

Choose the raw data channel you want to use in I&I Analysis, and the calculated channel you created for determining Delta Time for the raw data channel.

graph-select-channels.jpg

3.3. Set a Date Range

Using the From and To fields, specify a date range that corresponds with the dates you will use in your I&I study. Or, if you are currently unsure, select a wide range, such as Year To Date or Last 2 Years To Date from the Select a Date Range drop-down menu.

3.4. Choose Time Series for the Graph Type

graph-date-range-1.png

3.5. Click Plot Data to generate the graph

plot-data.png

The time series graph shown below displays two channels: 1) Weir Flow, which is to be used in I&I analysis; and 2) Delta Time - the time difference between data points in the Weir Flow channel. We see that Delta Time is typically 300 seconds (5 minutes), however a spike in the Delta Time graph indicates that delta time jumped up to 1412400 seconds (over 16 days!). This confirms that there is a large gap in the data. This must be fixed before the Raw Flow channel can be used in the I&I Analysis tool.

graphing-gap.png
When the channel does not receive data on a fixed time interval

If you believe that the channel(s) you want to use in I&I Analysis has data gaps, but does not receive data on a fixed time interval, we recommend that you interpolate the data such that the time interval between data points is a consistent 5-minutes. Please follow the directions under Fixing Data Gaps for guidance.

Interpolating data to remove data gaps / standardize time interval between data points

The channels you plan to use in the I&I Analysis tool must contain data that is recorded in a consistent time interval. Whether you have detected data gaps, or you know that the data is inconsistently recorded, you can use the Linear Interpolation function to establish a consistent time interval between data points. The steps described below guide you through the process of setting a data interval length of 5-minutes, however you can choose virtually any other interval length. Once finished, you will use the interpolated channel in lieu of the raw channel in your I&I analysis.

  1. Set up a new calculated channel (for housing interpolated data values)
  2. Configure the calculated channel to apply the Interpolate function to incoming data

1. Set up a new calculated channel for Interpolated data

1.1. Open the Configure Sites (FACE) tool

  1. Click Manage from the main toolbar.
    A drop-down menu appears.
  2. Click Configure Sites (FACE).
    configure-sites-face-1.png

1.2. Add a new channel

  1. Click Add New Channel.
    The Add Channel page populates.
fix-gaps-1.png
  1. Choose the site in which you want to store the delta time channel. Expand the Choose Site drop-down menu to make your selection.
    For easy reference, we recommend you store this channel in the same site as the originating raw channel.
  2. Input a Channel Name.
    We recommend you use the same name as the original channel, with "Interpolated" appended to the end.
  3. For Channel Units, select the same units as used in the original channel.
  4. For Channel Type, select Calculated.
  5. Ensure Visible is checked.
    This enables you to plot this channel in FlowWorks' Graphing tool.
  6. Click Create New Channel.
    You are taken back to the "Configure Sites" page. Continue setup by following the steps below.
weir-flow-interpolated.png

2. Configure the calculated channel

Now that a calculated channel has been created, it needs to be configured to apply the Interpolate function to incoming data.

2.1. Open the calculated channel you just made

Continuing from the Configure Sites page, click View Calculated Channel.

weir-flow-view-calc.png

A page opens that contains a list of calculations associated with the calculated channel. Since this is a new channel, such a list is currently absent.

weir-flow-empty.png

2.2. Set up the Interpolate function for this channel

  1. Click Create Calculation.
    The "Create Calculation" page appears.
  1. Enter a Calculation Name and Description.
  2. Expand the Type menu and select Interpolate.
    The lower portion of the page refreshes to display the necessary fields for completing Interpolate setup.
  3. (Optional) Set a Start Date and End Date.
    The Interpolate function will apply to all data within this time frame.
    Leave this area blank if you wish to apply the function to the entire data set.
  4. Select the Source Channel. Use the first drop-down menu to choose the site within which the channel is stored. Then, use the second drop-down menu to select the source channel.
  5. Set the Interval Length to 5 min, or another interval length of your choosing.
    interpolate-calc.png
  6. Click Save.
    The "View Calculated Channel" page loads, where the new calculation appears in a list.
weir-flow-interpolated-2.png

You can now use the interpolated calculated channel in lieu of the original channel in the I&I Analysis tool.

Have more questions? Submit a request

0 Comments

Article is closed for comments.