How can I prune the weak learners in an ensemble learner?

3 views (last 30 days)
Andrew
Andrew on 4 Oct 2014
Edited: Jan on 10 Aug 2017
The fitensemble method doesn't seem to provide a method for actually pruning the weak learners that comprise it. I understand accuracy may not be improved by doing so, but in my usage space is at a premium, so I'd like to reduce the number of nodes in the trees used as my weak learners -- or at least be able to experiment with the effect that has. And I'm interested in reducing the number of nodes not just by adjusting the "leafiness" via the MinLeaf and MinParent settings, but also use pruning.
Note that enabling "Prune" in the templateTree used as my weak learner does not seem to actually do any pruning, but merely computes the PruneList. However, not only do the CompactClassificationTrees that are my weak learners (inexplicably) lack a prune method, even though they have a PruneList, but also, they are protected members of the ensemble and thus not adjustable anyway.
So how can I prune my weak learners after creating an ensemble?

Accepted Answer

Andrew
Andrew on 4 Oct 2014
I'm answering my own question. (Actually, I had found the answer before asking, after much fiddling, but I thought it might be useful for others.)
NOTE: this is a somewhat disgusting hack. I'd love to know if there is a better way!
Though the "Trained" property of my ensemble is set-protected, and the constituent CompactClassificationTrees don't seem to possess a publicly accessible "prune" method, the hidden "Impl" property of each is publicly set-able and allows me to do what I want. So after using a template tree with pruning on to create an initial ensemble, I was able to loop over the resulting set of trained trees and prune each like the following. (The key is the two usage of "Impl" to access public implementations I could adjust.)
tTree = templateTree( ... , 'Prune', 'on');
ensembleCTrees = fitensemble(xdata, labels, 'AdaBoostM2', 10, tTree);
prunedEnsemble = ensembleCTrees;
for iTree = 1:maxTrees
prunedEnsemble.Impl.Trained{iTree}.Impl = prune( ...
prunedEnsemble.Impl.Trained{iTree}.Impl, 'Level', 20);
end
Then, I could try different settings of the prune level and compare prediction performance between the original and pruned ensembles.

More Answers (1)

Daniel Vieira
Daniel Vieira on 10 Aug 2017
I'm having a similar problem, it appears this doesn't work anymore on 2017a.
  1 Comment
Jan
Jan on 10 Aug 2017
@Daniel: Please do not highjack an existing thread by posting a new question as a pseudo-answer. Open a new thread and add a link to this one, if this is useful. Add an explanation, what "doesn't work" mean. It is easier to solve a problem than to guess, what the problem is. Then delete your answer here. Thanks.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!