How do I compare the accuracy of a neural network classifier model to a random classifier?
5 views (last 30 days)
Hi, I am attempting to show that the binary predictions made by my bidirectional LSTM neural network classifier are statistically significant, i.e. that the accuracy is more than that which may occur by chance using a 'random' classifier. I know that I can use McNemar's test to compare the accuracies achieved by 2 different models, but I am unsure as to which model to use as the 'random' or 'baseline' classifier.
Any suggestions would be greatly appreciated. Thanks in advance.
Vineet Joshi on 19 Apr 2021
You can create a ‘random’ classifier by simply using the randsample function. This function can return a randomly sampled class out of provided n classes.
On the other hand, a simple non-random ‘baseline’ classifier can be K nearest neighbor based.
The exact baseline model to use will depend on your use case. For example if you want to show how a bidirectional LSTM neural network can capture the sequential information, you can use a vanilla neural network as a baseline classifier as it will make predictions without any sequence information
Hope this helps.