The power of FlowWorks is its ability to transform data collected in the field from multiple vendors' hardware and to import that data into your FlowWorks database, as if it were one homogeneous network. With FlowWorks, you have the ability to analyze and report all of your data over a single, unifying system; not piece-by-piece, thus eliminating the need to use each vendor's proprietary tools.
The following is a discussion of some of the technology, options, and limitations involved in the 'communications' process of getting your data to FlowWorks.
In this topic:
- Device Communications
- Data Format
- Data Processing
- Direct Connection
- Intermediate Server
- Manual Forwarding
- Archiving Data
How your data gets from the field and into FlowWorks often depends on the brand(s) of hardware you have installed in the field (used for relaying the data), and the format of said data.
Occasionally, your hardware is located somewhere with a direct-wired connection to the Internet, such as at a pump station. At other times, you may need to obtain a cellular, GSM, or satellite modem and antenna in order to successfully broadcast your data via the Internet and into our servers. To determine what is the best option for you, it is recommended that you first consult the manufacturer of your hardware or a flow service provider - you will get a more comprehensive list of options from your Internet and communications providers from your service location. FlowWorks' Support staff will be happy to assist as well - what we may know about communications options in your environment would come from troubleshooting.
In selecting and configuring your communications devices, some questions about the interaction with the field hardware need to be considered; most of these questions will be limited by your selection of the hardware. For example, will your field instruments be 'calling' out to FlowWorks or some intermediate Site, or will they be 'called' by the FlowWorks software? In the case of cellular or other similar types of services, will your hardware need to have a controllable reporting schedule, or will it need to be capable of 'answering' a call or poll? Lastly, in terms of reaching the Internet, can your hardware communicate to an IP address, or can your hardware be reached through an IP address?
Important: If your device(s) have IP-based communication, the Sites need to be communicating to us before we can create them in FlowWorks. If your device(s) are using FTP, we can set the Sites up in advance.
There is a large variety of different types and brands of loggers strewn across the globe, all of which relay data that FlowWorks can support. In terms of understanding your locally-available options, we highly recommend that you contact your local communications provider. We have plenty of experience with different communication methods and will be happy to discuss our recommendations once you know what local options are available to you.
The format of your data is typically controlled by your hardware selection. Some hardware (Telog, Isodaq, Isco, etc.) uses a proprietary format which requires that either you or FlowWorks run a copy of their software to process the data communications from the field. Other systems are capable of flexibly being programmed to send a comma-delimited file (also known as 'comma separated values', or CSV file) via FTP to FlowWorks.
Without the need for any custom programming, the quickest account setup can be achieved so long as your data can be configured into FlowWorks' Standard Data Format.
Even if you cannot achieve the recommended standard data format, there is a good chance that the FlowWorks system can already read the data your hardware produces. To find out, send us a sample output file and we will use it to confirm our ability to already read your specific data format. It is only in the most extreme cases where several hours of programming will be required to move non-standard data formats through; in such a case, we will need to consult our developers to negotiate a custom solution for you.
Our formatting requirements for data arriving into FlowWorks are as follows:
- The Site's name (from which data is originating from) must be specified.
- The channel names must be the same in FlowWorks as it is externally at the Site.
- The units of measure assigned to the data values must match in the Site setup.
- The rows of data must each have a date and timestamp, and they must appear in chronological order.
- Each time the Site relays data to FlowWorks' servers, the format of said data must identical - the number of rows may be different, but the format of the date & timestamp and the sequence of the values (ie. chronological) must be consistent.
Additionally, for a successful import of the data from a Site, the Site name, the channel name(s) and the data format must match the configuration expected by the FlowWorks parsing engine. Part of this is to assure that the data is placed into the proper Site/channel, but it is also for maintaining security. Any packet of data received that does not match the formatting tests for the indicated Site is not processed and makes no further inroads into the FlowWorks system, thus protecting FlowWorks from malicious and erroneous objects.
With the considerable amount of experience we have with various hardware vendors, their plethora of different models of field equipment and the data that pass through their proprietary software and ours, we can usually build a parser for data that cannot be configured with FlowWorks' standard data format. Only in a few cases would one require a custom parser - in such a case it is important to note that any custom work requires negotiating billed programming time.
To find out if your data is 'import-ready', please e-mail us with a data sample from the specific hardware in question. We will assess the effort that will be needed to import the data in this format. Please send your sample to firstname.lastname@example.org.
'Direct connection' describes the communications processing used when FlowWorks runs a vendor's software server and that software manages the data communications between FlowWorks and the hardware in the field.
The software server running in FlowWorks communicates with the field equipment, collects data, and relays it over to the FlowWorks database.
Some hardware brands are limited to only using Direct Connection, such as Telog Enterprise, ADS Profile and Isodaq's XDQServer, among others. In these cases, FTPing the data is not possible; instead, you must go with a Direct Connection, using an IP address, server name, etc. It is recommended that you contact FlowWorks' Support Team to confirm the best mode of communication between your loggers and FlowWorks' system.
Some field hardware requires a proprietary server system that is not run on the FlowWorks platform. In these cases, the data from the field is either formatted by that server and then sent over to FlowWorks via FTP, or in cases like Hach's FSData system, Hach enables the web service for the Sites and FlowWorks is given permission to access that remove server on your behalf. FlowWorks then collects the necessary data, not directly from the field but from an intermediate server.
Some hardware cannot support FTPing data, due to their inherent behavior. Since there are so many loggers available, it is recommended that you confirm the ability to FTP, by contacting FlowWorks Support with details of your hardware type.
Some field data is manually submitted to FlowWorks from consulting firms or other entities, who collect the data and then forward it to FlowWorks via FTP. This is useful for those that have historical data that they want to include upon singing-up with FlowWorks. Uploading historical data permits users to include data collected in the past, in addition to new data received going forward. Historical data is typically sent to FlowWorks via comma-delimited (comma-separated-values) files using FTP. Please see topic, "Uploading Data to FlowWorks via FTP" for instructions.
The question of archiving data often comes up with agencies investigating the implementation of FlowWorks into their engineering environment. Some of these agencies have regulations and requirements that put them in the position of needing to get 'hands on' in the archiving of the data originating from their field instruments and the ability to audit their archival.
If you find yourself in this position, FlowWorks can provide you with a nightly file available by FTP for each of your Sites. This is a comma-delimited (comma-separated-values) file, capable of being imported into a wide variety of additional software products or moved to the archival system of your choice. Simply contact FlowWorks Support to request this file.