Regression Leaner App , data structure for multiple sensors and input possibility?

I am working on a sensor project where I need to calibrate electrochemical sensors using reference data. These sensors are affected by temperature and humidity, so I have created a table that includes the

[Sensor ID, temperature, humidity, and reference data for each sensor]

I input this table into the Regression Model App, and I find that exponential GPR produces the best results. However, each sensor has its own tolerance and sensitivity variation (+/- 10%) when reacting to changes in temperature and humidity. Ideally, I want to create a general model, but also give the user the option to adjust a variable factor to compensate for each sensor's unique characteristics.

If I use the table with Sensor ID included, the model works well for that specific sensor. However, if I remove the Sensor ID and only use the table with temperature, humidity, and reference data, the results are average and do not capture each sensor's unique characteristics.

This is my code

all_timetable=[];                    
for i=1:8    
temptime_table=timetable(app.electrosensors_table{i}.Time, ...
app.electrosensors_table{i}.Temperature,...
app.electrosensors_table{i}.Humidity,...   
app.electrosensors_table{i}.NO2,'VariableNames',["Temperture","Humdity","Sensor"]);                        
%all_timetable=[all_timetable;temptime_table]
%all_ref= [all_ref;app.sensor_table_data(i).ref_timetable_period.locationA]
temptime_table.SensorID = categorical(repmat(i,height(temptime_table),1));
temptime_table.CorrectionFactor = ones(height(temptime_table),1);                    
data = temptime_table(app.sensor_table_data(1).ref_timetable_period.locationA.ref_time,:); % only use reference times
data.Ref = app.sensor_table_data(1).ref_timetable_period.locationA.ref_data; % add reference (used as "response")
all_timetable=[all_timetable;data];
end                    
%data = timetable2table(data);
data = timetable2table(all_timetable);
time = data.Time;
data.Time = []; % remove Time

rng("default") % for repeatability
n = height(data);
p = randperm(n,round(0.8*n)); % 80% for training, 20% for testing
idx = false(n,1);
idx(p) = true;
data_for_regression = data(idx,:);

assignin('base','data_for_regression',data_for_regression)
disp("a");

My question is, how can I instruct the regression app to account for the slight variation in each sensor and create a general model that can also be adjusted by the user to compensate for each sensor's individual differences? Is this possible?

Answer 1

[Amal Raj]

Yes, it is possible to account for the slight variations in each sensor and develop a general model that can be adjusted by the user to account for the individual differences in each sensor.

One approach would be to add an additional input variable that is unique to each sensor, such as a scaling factor or bias term. Calibration experiments can be used to determine the value of this variable, in which you measure the response of each sensor under different temperature and humidity conditions and fit a linear model to relate the response to the input variables and the scaling factor.

To use this method in the Regression Model App, modify your input table to include a column for each sensor's response as well as a column for the scaling factor or bias term for each sensor. Then, using the temperature, humidity, and scaling factor columns as inputs and the response columns as outputs, you can train a regression model, such as a Gaussian Process Regression (GPR). Additional features, such as the product of temperature and humidity, can be included to capture any nonlinear interactions between the input variables.

Once the model has been trained, the user can adjust the scaling factor or bias term for each sensor to account for its unique characteristics. This can be accomplished by either directly modifying the input table or by utilising a user interface that allows the user to adjust the values interactively.

[Dharmesh Joshi]

Thank You for your reply

After my original post, I added a new variable called Sensor ID, simply a categorical variable. This worked for each sensor, but a unique ID was added, and the output was 0.

As you mentioned adding an input variable scaling factor or bias term, could the sensitive factor be a good value to use?

A scaling factor is incorporated into my sensor value prior to the regression app. For example

NO2 = (Sensor Output – PCB Offset) / Sensor Sensitivity.

I am unsure if the “Sensor Sensitivity” factor is directly related to each sensor characteristic, but below is a plot from the sensor datasheet showing how temperature affects the sensor's output. As you can see, there is variation among them,

I got a few questions:

So when you say bias or scaling factor, is it the “Sensor Sensitivity” or another variable I need to create into the input table?

And you mentioned I need to create a column for the response of the sensor. Does this refer to the output of the model?

So if we also use the product of temperature and humidity, is it necessary to input each individually, or is this just an additional parameter input?

[Amal Raj]

Regarding your first question, the "Sensor Sensitivity" factor, which indicates how much the sensor output changes in response to changes in the input variables, may be a suitable scaling factor to account for each sensor's unique characteristics. You could experiment with including this factor in your input table to see how it affects the model's performance.

Regarding your second question, when I mentioned creating a column for sensor response, I was referring to the reference data that you are using to calibrate the sensors. For a given set of input variables, this is the value that the model should predict. This column must be included in your input table because it is the target variable that the model is attempting to learn.

Regarding your third question, if you have reason to believe that the product of temperature and humidity has a significant impact on sensor output, it may be worthwhile to include it as an additional parameter in your input table. However, if you don't have prior knowledge that this interaction is significant, you may not need to include it. Finally, the input variables you choose should be guided by your understanding of the sensor system and the factors that influence its output.

[Dharmesh Joshi]

Just to update you with more information, I have communicated with the sensor manufacturer. They have said that the sensitivity factor or variable converts the sensor's voltage output to a value compared to reference data. However, each sensor's characteristic against temperature is unique, and I believe this reflects each sensor's characteristics. This is shown in my previous post in the image of the plot. As you can see, this is an exponential curve.

Therefore, can the bias or variation factor include some type of exponential input that a user can adjust to create a correction curve similar to the plot's image?

So, can the additional bias input somehow reflect this exponential plot that I attached previously?

[Amal Raj]

Yes, it is possible to include an exponential term in your input table to reflect the non-linear relationship between sensor output and temperature. One approach you could try is to include a new input variable that is the exponential function of temperature, such as exp(k * T), where k is a user-defined constant that determines the steepness of the exponential curve and T is the temperature input variable. The user could adjust the value of k to match the specific characteristics of each sensor.

Alternatively, you could use a more flexible model that can capture non-linear relationships between the input variables and the sensor output, such as a neural network or decision tree. These models can learn to approximate the non-linear relationship without the need for a specific functional form, such as an exponential curve.

Regarding the bias or variation factor, you could still include a scaling factor or bias term to adjust for individual differences in the sensor's sensitivity or offset. This could be a separate input variable that is unique to each sensor and is learned during calibration experiments.

Answer 2

[Dharmesh Joshi]

Thanks,

So, as input, how would I establish "K" as the input for stepness? I don't know if this is possible or even necessary with the Regression App, but can we create another model that works out the variation among data sensors, and then we can use the variation between sensor data? This could give us insight into the stepness range.

When we create a model, GPR produces the best result, but I'm not sure why the neural network has not produced a similar result. The sensitivity or bias is simply to adjust the voltage to a sensor value. This sensitivity can vary over time, so I believe the model would compensate for this, but there could be a possibility that after some time we would need to adjust the sensitivity factor. This factor is affected by the age of the sensor, exposure to the gas, and possibly the environment, so I do not think this would be identical for each sensor. Therefore, a manual process against reference data would need to be done unless, in the future, we could use machine learning to create an additional model, assuming reference data is available.