Load measurement data
Watch our 1 minute importing raw time series data video and the 12 minute measurement campaign tutorial video for a good overview of the data loading process.
To use the data loader you must specify the timestamp format, delimiters and mapping of columns to measurements used in your input data files. WindFarmer requires that you create a settings file first based on one file, then it tries to use the settings file to load many data files. If there is an error loading later files you can modify the settings file and load further data after a change in data format.
The data loading process can be summarised as follows:
- If necessary use the data logger proprietary software to convert the data to delimited text files.
- Click Add Files or Add Folder so all the files for one measurement site are queued in the Data Files table (below)
- Select a file to use as a settings template. If you have many files it is often best to choose a file several files down the list to avoid configuration change issues at the start of a measurement campaign.
- Configure settings by clicking New and working through the wizard (see sections below). You could also open an existing settings file and edit this (if you have another measurement site with a similar configuration).
- Save the settings file. A name is suggested at the same location as the template data file.
- Hit the Load button to process your data!
- Review any header changes you are alerted to.
- Investigate and correct any errors in data loading. You can define new settings based on the errored file and load again in many cases.
- Once you are finished for one measurement site, select all files, Remove them, and then repeat for the next measurement site.
- Review the imported data in Data cleaning or in the Measurement tree.
Tip
Data may also be loaded from existing workbooks from the Map > Measurement site location task
Header and delimiter
| Header settings category | Description |
|---|---|
| Fixed length header | Specify the number of lines contained within your header which must be ignored for each loaded file |
| Symphonie Data Retriever | Support is available to recognise headers of data processed using the NRG Symphonie Data Retriever. Using this option will provide useful column labels that use the information from the processed SDR file header. This option will automatically be selected if you load a RWD file or a converted RWD file. See NRG data loading for more information. |
| Delimiter | |
| Field delimiter | Specifies how data are separated into columns: tab, space, comma etc. |
| Consecutive delimiters as one | This option is designed for the special case where you wish to load a file containing multiple and variable numbers of spaces between data columns. The data in these files looks well aligned in columns when viewed in a text editor. Be wary of using this option in any other situation (including importing data copied, saved or copied from Excel) because if data is missing for one record the columns of data will be misinterpreted at this point and erroneously create channel switches later in the analysis. |
| Excluded text | |
| Text qualifier | This specifies whether or not the date is contained within double or single quotation marks and whether there are any text fields to exclude. |
| Error codes | Text to interpret as a missing value. Add text that should be ignored (e.g. NaN, 9999 etc.) |
| Regional formatting | |
| Number formatting | This setting allows you to use a different thousands and decimal separator to your regional settings (that set the default selection). Select the regional format for the numeric values in the data you wish to import. |
Timestamp columns
First select the time-stamp category depending on how the timestamp is represented in columns:
| Time-stamp category | |
|---|---|
| Combined date and time | for files with the entire timestamp in one column |
| Separate date and time | for files with date and time in separate columns |
| Campbell timestamp | For Julian date-time format. Separate columns are provided for year, day of year (1 to 365) and minute of day (0 to 2350). Note Cambell loggers can also have other timestamp formats that can be specified using the other options. |
| Other time-stamp settings | Description |
|---|---|
| Sample interval (Minutes) | Interval between data points in the series |
| Time offset (minutes) | Offsets each time series record before it is imported into WFA. Note that offsets that are a multiple of the sample interval can be applied later in Time Offset Correction |
| Timestamp period | 10 minute data from data loggers represents the mean, min, max or standard deviation of a continuous signal over the sample period. It is recorded as signal values time stamped at the start, end or middle of each sample period. Specify the format that your data uses. For calculations WindFarmer assumes all data is time stamped at the start of each sample period so the data loader will transform your data to fit WindFarmer's convention on import. |
| Timestamp column(s) | Specify column number (see file preview for columns) |
Timestamp format
To specify the timestamp format either select from the list or specify a custom timestamp by entering the date-time format code. Date and time separators should be specified as they appear in the timestamp. The dates and times should be specified with letter codes as shown in the examples below:
Date / Time codes examples
| Example Timestamp | Input required | Notes |
|---|---|---|
| 07/05/2011 16:30 | dd/MM/yyyy HH:mm | A UK 24-hour clock timestamp |
| 07/05/2011 16:30:00 | dd/MM/yyyy HH:mm:ss | Timestamp includes seconds (ss) |
| 5/7/2011 4:30:00 PM | MM/dd/yyyy h:mm:ss tt | US style, with AM or PM (tt) and a 12-hour timestamp (h) without a preceding zero (e.g. 4 not 04) |
| 07.05.2011 16:30:00 | dd.MM.yyyy HH:mm:ss | European style date with a dot date separator |
| 07-05-2011 16:30:00 | dd-MM-yyyy HH:mm:ss | Dash date separator |
| 7/5/11 16:30 | d/M/yy H:mm | Two digit year (yy) and days, months and hours have no preceding zeros (d, M, H rather than dd, MM, HH) |
| 051111 1010 | ddMMyy HHmm | |
| 20110705 10:10:00 | yyyyMMdd hh:mm:ss | Reversed year-month-day timestamp |
| 07/05/2011 06:40:00 ? | dd/MM/yyyy HH:mm:ss "?" | Extra character at the end of the timestamp. The character needs to be specified in quotes |
Date / Time codes summary
| Code | Example Characters | |
|---|---|---|
| Day | dd or d | 07 or 7 |
| Month, numeric | MM or M | 08 or 8 |
| Month, abbreviated | MMM | Eg: Aug |
| Month, full name | MMMM | Eg: August |
| Year | yyyy or yy | 2012 or 12 |
| Hours: 24 hour clock | HH or H | 23, 03 or 3 |
| Hours: 12 hour clock | hh or h | 09 or 9 |
| Minutes | mm or m | 06 or 6 |
| Seconds | ss | 00 |
| AM or PM | tt | AM |
| Any other text | "text" | unwanted characters should always be specified within quotes |
| Quotation marks in timestamp | Try a Text Qualifier option in the delimiter and header settings page of the wizard. |
Note
It is possible to denote an hour, minute, second, month, or day of 07 by using either HH or H, mm or m, ss or s etc. but it is not possible to define a single digit hour, minute, second, month or day (e.g. 7) using HH (or hh), mm, ss, MM or dd.
Data columns
The data columns page allows you to create new mast and sensor names by simply typing the names into an excel-like grid. You can also choose signals types and mast and sensor names which already exist in the workbook from drop-down lists. Please note that your data has been transposed in this view to allow easy scrolling down the columns of data which makes header information more readable to help define signal names.
Measurements and sites
Within the masts and sensors page you specify the measurement types and heights for any new measurements and details for any new measurement site. You can change the names of masts and sensors if they don't already exist in the project.
Automated data processing and loading for NRG .RWD files. We automatically process raw binary RWD files as follows:
- Auto - creation of loader settings for NRG data.
- Data and exact calibrations applied to data are imported.
- Comparison between each file header for consistency.
- Review settings to load any files rejected based upon header differences.
The first step of data loading is always to add your files to the file preview window. We recommend adding all of your data files.
WindFarmer automates the SDR software. You can specify the path to your version of the SDR using File > Preferences > Data loading.
Note
We automate the SDR using a special "no queue" mode. This allows us to invisibly create text files with all signals converted at 4 dp precision based upon the logger settings and calibrations programmed in the logger for each individual file. We avoid some of the problems inherent in the SDR batch processing software so you don't need a site file. If you do need to process encrypted files please contact us: this is not yet automatable in the standard SDR version.
Loading NRG RWD data
To load data you first need to create some settings.
Select an RWD file as a template, this file will be used to name and assign signals and the file header will be saved as an accepted header which can be compared to other file headers. Unless you have good reason not to, we recommend choosing a file several files down the list to avoid the usual tinkering issues at the start of a measurement campaign (The logger set up is often wrong for the first data files in a measurement campaign).
Click New to auto-create loader settings. Auto-settings creation will also work for converted text SDR files.
Click through to the data columns page. Delete names for any data you don't want to load after looking at the mast documentation. E.g: the channel 15 anemometers are often empty. This is NOT the best place to edit sensor names (you would lose any sensor type or meta-data details)
Click to the last page of the settings wizard. You should add orientation information and you can edit the notes and mast or measurement names if desired.
Save settings file. A name is suggested at the same location as the template data file.
Now you have a settings file you can hit the load button to process your data!
Note, measurement names are automatically assigned as shown below:
| Measurement Type | Prefix used in sensor name | Height interpretable [m] | heights match for 2+ sensors | No heights |
|---|---|---|---|---|
| Anemometer | A | A80 | A80c1 | Ac1 |
| Wind Vane | D | D78 | D78c7 | Dc7 |
| Temperature | T | T60 | T60c5 | Tc5 |
| Pressure | P | P10 | P10c5 | Pc5 |
| Voltage | V | V3 | V3c5 | Vc5 |
| Relative humidity | RH | RH45 | RH45c8 | RHc8 |
| Unrecognised sensor type | C1 (Channel number) | None used | None used | None used |
| Ignored | - | - | - | - |
Note
- If the height is given in feet (ft) then we convert to meters. Otherwise it is assumed height is in m. Heights are rounded to the nearest integer.
- c1 means channel 1