Explore - Scientific Workflow Management Tool

Lightweight workflow management tool to automatically retrieve previous intermediate results and track provenance in scientific workflows.
49 Downloads
Updated 3 Sep 2019

Explore is a scientific workflow management tool. You can visualize the data provenance graph and when your code is changed and re-executed, only the necessary functions are re-executed to provide an updated version of your data. This can save a lot of time, especially when experiments are constantly refined. You can easily declare an experiment as a directed acyclic graph (DAG) where the nodes are functions and the edges represent variables that are produced and consumed by functions.

During the first execution of the graph, variables are persisted to the disk which implies a longer graph execution time (due to variable loading and saving duration). However, for future executions, if the node main function and all the sub-functions called during the node execution remain unchanged (a) and the input variables of the node also remain unchanged (b), then the results are retrieved from the disk and not re-computed.

For (a), the major time consumer is the code analysis to find out the sub-functions involved in the execution of a node. Therefore, this "node sub-function dependency" information is also persisted based on the content of the sub-function files. Then, if the sub-function files did not change, it is assumed the node's sub-function dependencies did not change.
For (b), either the variable content is hashed (i) or the variable mat-file date is taken (ii) as signature to check that the variables did not change. Option (i) is advantageous as re-computed nodes might still output the same values. In that case, unchanged child nodes will not be re-computed. However, option (ii) might be faster for bigger variables, where hashing would take too much time.

Thank you to:
- Jan (2019). DataHash (https://www.mathworks.com/matlabcentral/fileexchange/31272-datahash), MATLAB Central File Exchange. Retrieved April 15, 2019.
- Aslak Grinsted (2019). cachedcall (https://de.mathworks.com/matlabcentral/fileexchange/49949-cachedcall), MATLAB Central File Exchange. Retrieved April 15, 2019.
- Aslak Grinsted (2019). rgb.m (https://de.mathworks.com/matlabcentral/fileexchange/1805-rgb-m), MATLAB Central File Exchange. Retrieved April 15, 2019.
- All upstream dependencies

For tutorials, detailed information and bug reports, please check the Github (https://www.github.com/jahsue78/explore) or the corresponding conference paper [1].

[1] Ah Sue, Jonathan, et al. "Optimizing Exploratory Workflows for Embedded Platform Trace Analysis and Its Application to Mobile Devices." International Conference on Human-Computer Interaction. Springer, Cham, 2019.

Cite As

Jonathan A (2024). Explore - Scientific Workflow Management Tool (https://github.com/vacoa/explore), GitHub. Retrieved .

MATLAB Release Compatibility
Created with R2017a
Compatible with any release
Platform Compatibility
Windows macOS Linux
Categories
Find more on Graphics Object Identification in Help Center and MATLAB Answers
Acknowledgements

Inspired by: cachedcall, DataHash, rgb.m

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Versions that use the GitHub default branch cannot be downloaded

Version Published Release Notes
1.0.11

Add link to the paper

1.0.10

Update description and add new basic tutorial.

1.0.9

Change title and summary

1.0.8

Update description

1.0.7

Update description and acknowledgments

1.0.6

New description

1.0.5

Modify readme and add keywords.

1.0.4

Update description

1.0.3

Add Github reference.

1.0.2

Change keywords

1.0.1

Update image

1.0.0

To view or report issues in this GitHub add-on, visit the GitHub Repository.
To view or report issues in this GitHub add-on, visit the GitHub Repository.