checkpoint

Class: matlab.compiler.mlspark.RDD
Namespace: matlab.compiler.mlspark

Mark an RDD for checkpointing

Syntax

checkpoint(obj)


Description

checkpoint(obj) marks an RDD for checkpointing.

Input Arguments

obj — Input RDD

RDD object

An input RDD that is to be marked for checkpointing, specified as an RDD object.

Examples

Mark an RDD for Checkpointing

Use the checkpoint method to save an RDD to a file inside the checkpoint directory.

%% Connect to Spark
sparkProp = containers.Map({'spark.executor.cores'}, {'1'});
conf = matlab.compiler.mlspark.SparkConf('AppName','myApp', ...
                        'Master','local[1]','SparkProperties',sparkProp);
sc = matlab.compiler.mlspark.SparkContext(conf);

%% checkpoint
sc.setCheckpointDir('myDir')
myFile = sc.parallelize({1,2,3});
mapRDD = myFile.map(@(x)({x,1}));
mapRDD.checkpoint();

Version History

Introduced in R2016b