Is there a parallel version of splitapply()?
4 views (last 30 days)
Show older comments
Group-based computation is naturally sutiable for parallel computation. I wonder why Matlab has not yet a built-in parfor-splitapply? In Github, there is a repository called Matlab-FunUtils, which has cmap.m. It's like a parfor version of arrayfun. I couldn't find a similar function for splitapply. I have been trying to modify Matlab's splitapply, but the codes are very difficult to understand. Any suggesstoins? Should I write from scratch?
0 Comments
Accepted Answer
Edric Ellis
on 15 Sep 2023
tall arrays support parallel execution of splitapply. Would that work for your case?
6 Comments
Edric Ellis
on 20 Sep 2023
Yes, typically this sort of algorithm cannot work in a nested way. It isn't quite the same restriction as nested parfor (in fact, with thread-pools, you can get genuine 2-level parallelism with parfor, but there's a separate restriction that you need to "hide" the inner parfor inside a function).
As to your original question - does it work to make the groups based on the combinations of variables, as per the 2nd syntax of findgroups ? I.e. something like:
load patients
G = findgroups(Smoker, Age >= 40)
This divides the data into 4 groups for each combination of Smoker and Age >= 40.
More Answers (1)
Bruno Luong
on 15 Sep 2023
To me there are 2 reasons:
- Not everyone have parallel toolbox
- The user function can be multi-threaded and already use efficiently the CPU cores, add parallel on top will then likely reduce the performance. The parallel computation should never be applied automatically anywhere. It should be judged case by case.
See Also
Categories
Find more on Parallel for-Loops (parfor) in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!