How to conduct Friedman test to compare metaheuristics and which data sets I should consider?

Dear all,

I am working on metaheuristics. I want to do Friedman test to compare the algorithms. I run each algorithm 32 times and stored the best, mean, median values from each run.

To perform the Friedman test which values I should consider, the best value (last value) from 32 runs of each algorithm, or median or mean values? and please explain the reason for selection of this particle values.

Please share your view about this test.

Thanks

[Oladipo Stephen]

You can use Excel. You can watch this on youtube, "Statistics 101_ Nonparametric Methods, Friedman Test in Excel"

Answer 1

[Hornett]

In practice, the mean or median values are more commonly used for statistical comparisons in algorithm performance because they provide a more comprehensive view of the algorithm's behavior over multiple runs. The best values can be influenced by outliers and might not represent the typical performance of the algorithms.

Here's why you might prefer mean or median over best values:

• Reproducibility: Mean or median values are more reproducible than the best values. An algorithm that consistently performs well on average is generally more reliable than one that performs exceptionally well only occasionally.
• Robustness: Median values, in particular, can provide a more robust comparison when there are outliers in the data, as they are less sensitive to extreme values.
• Statistical Power: The Friedman test assumes that the measurements are independent and identically distributed. Mean or median values, by summarizing the runs, may better meet these assumptions, especially if there is variability in the performance of the algorithms across runs.

In summary, if your goal is to compare the overall performance and reliability of the algorithms, you should consider using the mean or median values for the Friedman test. If you are specifically interested in the best possible performance regardless of consistency, you might opt for the best values, keeping in mind the potential limitations of this approach.

Friedman's test: https://www.mathworks.com/help/stats/friedman.html

Hope it Helps!