CRBasic Tips to Simplify Data Post-Processing

You’ve collected your measurement data, but how do you easily make sense out of it? How can you simplify the post-processing of your data and start analyzing it sooner? In this article, I’ll explain how adding some code to your CRBasic program can save you from a post-processing headache later.

As a graduate student, I remember my colleagues often talking about the headache of post-processing data before they could even begin analyzing their data.

1. For post-processing, some of us had learned how to write macros in Excel.
2. Other colleagues preferred using resources such as Python, MATLAB, or R scripts to format their data for analysis.
3. A group of us just slogged our way through, copying and pasting data into a new worksheet in Excel until all of the information was available in one spreadsheet—complete with extra columns of categorical labels.

As you might imagine, my colleagues and I found all of these post-processing methods to be cumbersome.

For our experiments in graduate school, we were encouraged to keep detailed notes on dates and times, as well as when an experiment moved from one treatment to the next. I understood the importance of keeping good notes, but I wondered if there was a way to save the information in the data table defined in CRBasic.

Most statistical software packages rely on data frames or input files that have a minimum number of columns and large quantities of rows or records. Too often, however, we format our raw data tables without considering the analysis we hope to complete after the data is collected.

Data tables typically include the default time stamp plus record number, and then we just start adding sensor measurements and system voltages. But normally, an experiment includes more—more treatments, more repetitions with multiples of the same sensor, just more information that is used to group the data for exploratory data analysis as well as T-tests, ANOVAs (analyses of variance), or regressions.

So, how can you store information like this in your data table for easy reference? I’ll step you through an example of how to create and manipulate categorical variables in CRBasic. The variables can be edited while the data is collected and stored in the data table.

Example

Note: Even if you’re not familiar with water quality and turbidity measurements, I think you’ll see in this example how easy it is to improve the helpfulness of your data table.

Background

In this example, the following question was posed: How does optical turbidity vary at six known concentrations (0, 10, 50, 100, 250, and 1000 mg/L)? To answer this question, data were collected using three OBS500 turbidity meters. The details of the experiment follow:

• Two hundred measurements were collected from each turbidity meter at each concentration.
• The measurements were collapsed into groups of 100 for mean, median, and standard deviation calculations.
• Different minerals were used to create sediment suspensions (quartz, kaolinite, bentonite, feldspar, mica, and coal). Natural river sediment was also collected, dried, sieved, and weighed to create natural sediment suspensions.
• The categorical variables included concentration (0, 10, 50, 100, 250, and 1000), sensor name (A, B, and C), and mineral (quartz, kaolinite, bentonite, feldspar, mica, coal, and natural sediment).

Original Data Table

The first round of data collection produced data tables similar to the one shown in Table 1. The reported values included the time stamp, record number, raw backscatter and sidescatter in millivolts, and backscatter and sidescatter readings in FBU (Formazin Backscatter Unit) and FTU (Formazin Turbidity Unit).

Table 1. Raw data collected during the sediment study.

For this experiment, the time stamp was not important because the measurements were not changing relative to time but to known treatments of sediment (concentration and mineral). Using the current data table setup in the CRBasic program running on my datalogger, I had to rely on my detailed notes to post-process the data and figure out which sensor was used, what concentration was measured, and what mineral was used to create the suspension.

Improved Data Table

After making a few changes to the CRBasic program, I was able to generate a more detailed data table (Table 2) that includes categorical information defined as the following:

• obs500_meas(1) is backscatter (FBU).
• obs500_meas(2) is sidescatter (FTU).
• obs500_meas(5) is raw backscatter (mV).
• obs500_meas(6) is raw sidescatter (mV).

Table 2. Data collected for the natural sediment from sensor A at a concentration of 0 mg/L.

How to Do It

To transform my data table from Table 1 to Table 2, I only needed to add a few lines of code to the CRBasic program. Here’s how you can do it too:

1. Define a few new variables for sensor and concentration. The sensor variables are defined as a 16-character string. The sensors are not public variables because they do not change. Concentration, however, is a public variable, so you can enter the known concentration into the public table.
   

   
   Dim SensorA As String * 16
   
   Dim SensorB As String * 16
   
   Dim SensorC As String * 16
   
   Public Concentration As Long
   

2. Define the new data tables like this:
   

   
   DataTable (SamplesA,1,10000)
   
   	Sample (9,obs500_measA(),IEEE4)
   	Sample (1,SensorA,String)
   	Sample (1,Concentration,FP2)
   EndTable
   
   
   DataTable (SamplesB,1,10000)
   
   	Sample (9,obs500_measB(),IEEE4)
   	Sample (1,SensorB,String)
   	Sample (1,Concentration,FP2)
   EndTable
   
   
   DataTable (SamplesC,1,10000)
   
   	Sample (9,obs500_measC(),IEEE4)
   	Sample (1,SensorC,String)
   	Sample (1,Concentration,FP2)
   EndTable
   

3. Define the sensor names within the program. Because we declared the sensors as “Dim” in our program, they will not show up in the public table. So, after your “BeginProg” statement, you will need to define the Name you want for each of the three sensors.
   
   Tip: Remember that the sensors were defined as a 16-character-long string, so the name for each must be less than 16 characters.
   
   In our example, A, B, and C were used. However, you could include serial numbers or model numbers of your sensors. These are not public variables in the sense that they are NOT changing with treatment. I collected 200 measurements per sensor in each treatment.

   
   BeginProg
   
   	SensorA="A"
   	SensorB="B"
   	SensorC="C"
   

Now, every time a data table is called, the sensor string and concentration are added to the record as independent columns in the data table. It’s that easy!

Conclusion

I hope this example helps you understand how some simple additions to your CRBasic program code can save you from a post-processing headache later. Feel free to post your questions or comments below.