Identifying and fixing Data Gaps

For many FlowWorks analysis tools, it is important to ensure there are no gaps in the data you are using. However, when collecting large amounts of data, gaps are sometimes inevitable, so a workaround is necessary.

The I&I Estimator is one example of an analysis tool that requires a complete set of data, absent of gaps. It is also important to note that channels used in the Estimator must receive data every five minutes.

Using the I&I Estimator as our example, this article will discuss:

  • How to create a new calculated channel that shows the time difference between each data point;
  • how to graph the new channel and identify data gaps;
  • how to resolve data gap issues using linear interpolation.

Seeing whether your data has holes

So long as the channel in question is expected to receive data on a fixed time-interval, it is possible to see whether the data has gaps; and if so, how large these gaps are. To find out, you will need to:

  • Create a new channel that calculates the time difference between each data point (called Delta Time);
  • then graph this channel (Time Series graph) to visualize data gaps.

The graph for Delta Time should show a nice flat line at 300 seconds, indicating that data was consistently received every five minutes. Any spikes along the line indicate times when data was not received at the 300-second mark, from which we can conclude that there are gaps in the data.

Setting up a new Calculated Channel to show the difference in time between each data point

The process for creating a new calculated channel is two-fold: first, you need to set up a new channel; second, you need to configure the channel to perform the calculation.

To set up a new calculated channel:

  1. Log-in to your FlowWorks user account.
  2. Hover your mouse over , found in the Main Toolbar.
  3. Select .
  4. Click . You are directed to the "Add Channel" page, where you will setup the Delta Time channel.
  5. Choose site for the channel to reside. This should be the same site that contains the channels you want to use in I&I analysis.
  6. Enter a Channel Name. we recommend 'Delta Time'.
  7. For Channel Units, select 'Seconds (sec)'.

    Data gaps are based on the amount of 'lag time' between each new data entry.

  8. For Channel Type, select 'Calculated'.

    Remember, we want to calculate the time difference between data points.

  9. Ensure Visible is checked, and System Admin Only and Enable Rainfall Tools are both left un-checked.
  10. Click .

    The system may need a few moments to process - you will be directed back to the 'Configure Sites' page once loading is complete.

Now that the calculated channel has been created, you now need to configure it to perform the Delta Time calculation:

  1. From the ‘Configure Sites’ page, select the Delta Time channel. Expand the Choose Site list and select the site that your new calculated channel resides in; then from the Choose Calculated Channel list, select the new channel you created in the previous section (titled ‘Delta Time’ in this example).

    Note: No action is required for the Choose Channel drop-down menu.

  2. Click .
  3. Click .
  4. Give the new calculation a name and description (optional),
  5. Expand the Type drop-down list; select Delta Time.
  6. Set a date range, or leave this blank to have the calculation apply to all channel data.
  7. Select the Source Channel.

    This is the channel that you want to check for data gaps. Start by selecting the site that the channel resides in, then choose the source channel.

  8. Choose 'No' for Ignore Null values.

    For the example described in this article, we do not want to ignore null values: when checking for data gaps across time, a null value is still considered a valid data entry.

  9. Click . You are directed back to the 'View Calculated Channel' page, where the new calculation appears in a list.

Now that your calculated channel is fully set up, you can proceed to graph the channel and see whether data gaps exist.

Graphing the calculated channel to verify data gaps exist

To graph the calculated channel and confirm that data gaps exist:

  1. Click from the Main Toolbar. The Graphing page opens.
  2. Select your channels. Choose the channel you want to use in I&I (in this case, Flow Rate), and also select the new calculated channel. Find both in the Site Explorer (along left side of the page).

  3. Set the date range.
  4. Select Time Series for graph type.
  5. Click . Your graph generates on page's right side.

The graph below shows the channel we want to use in I&I (Flow channel) and the data gaps:

The sudden spike seen in the Delta Time graph indicates that there is indeed a gap in the data that must be fixed before the Flow Rate channel can be used in the I&I Estimator.

Now that you have confirmed that gaps exist in the data, you can move on to fixing them.

Fixing data gaps using Linear Interpolation

So long as the data gaps are relatively small, you can create a new channel that applies a linear interpolation across the entire dataset. Using this method will 'fill in' any gaps and make the data ready for use with analysis tools such as the I&I Estimator.

If the gap is large, you should consider whether interpolating across the gap is an appropriate method of fixing the data. Consider alternative methods such as: filling-in the data gap with 'other representative data'.

Creating a new channel for interpolated data

Below is a graph that shows the channel we want to use in I&I (Flow Rate), the data gaps, and the new interpolated channel:

This graph shows the Flow Rate, Delta Time and Interpolated Flow for a particular site. The Interpolated Flow channel and the original Flow channel look virtually identical – this is because we only had a small data gap to interpolate.

The process for setting up an Interpolated channel consists of almost the same steps as described above, for setting up a Delta Time calculated channel: first create the new channel, then set up the calculation that is to be applied to data in this new channel.

In our I&I example, once we have created a new Interpolated Flow channel, we can use it in the I&I Estimator in lieu of the original Flow Rate channel.

To set up a new calculated channel for Interpolation:

  1. Hover your mouse over , found in the Main Toolbar.
  2. Select . The Configure Sites page opens.
  3. Click . You are directed to the "Add Channel" Page.
  4. Choose Site for the channel to reside. This should be the same site that contains the channels you want to use for analysis.
  5. Enter a Channel Name. In this example, we will call it 'Interpolated Flow'.
  6. Choose Channel Units. For this example, we chose MGD (million gallons per day).
  7. For Channel Type, select 'Calculated'.

    Interpolation involves applying calculations to fill-in data gaps.

  8. Ensure Visible is checked, and System Admin only and Enable Rainfall Tools are both left un-checked.
  9. Click .

    The system may need a few moments to process - you will be directed back to the 'Configure Sites' page once loading is complete.

Now that you have setup a designated channel for interpolated data, you need to configure it to perform interpolation for the channel containing data gaps:

  1. From the ‘Configure Sites’ page, select the Interpolated Flow channel. Expand the Choose Site list and select the site that your new calculated channel resides in; then from the Choose Calculated Channel list, select the new channel you created in the previous section (titled ‘Interpolated Flow’ in this example).

    Note: No action is required for the Choose Channel drop-down menu.

  2. Click .
  3. Click .
  4. Give the new calculation a name and description (optional).
  5. Expand the Type drop-down list; select Interpolate.
  6. Set a date range, or leave this blank to have the calculation apply to all channel data.
  7. Select the Source Channel - this is the channel that you want to check for data gaps. Start by selecting the site that the channel resides in, then choose the channel.
  8. Set the Interval Length to 5 minutes. All data to be used in the I&I Estimator (and most other tools) must be in five-minute intervals.

  9. Save. You will be directed back to the 'View Calculated Channel' page, where the new calculation will appear in a list.

Now that you have set up Interpolation of data for the channel containing data gaps, you can use it in the I&I Estimator (and many other FW tools) in lieu of the original, ‘botchy’ channel.

Further Troubleshooting

If you are still having trouble using a channel in the I&I Estimator, you can proceed to check the Dry Weather Pattern for data gaps. Help on this is coming soon!

Have questions?

We are here to help you! Please open a new support ticket to quickly get in touch with FlowWorks support!

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.