Main Content

importNetworkFromPyTorch

Import PyTorch network as MATLAB network

Since R2022b

    Description

    example

    net = importNetworkFromPyTorch(modelfile) imports a pretrained and traced PyTorch® model from the file modelfile. The function returns the network net as an uninitialized dlnetwork object.

    importNetworkFromPyTorch requires the Deep Learning Toolbox™ Converter for PyTorch Models support package. If this support package is not installed, then importNetworkFromPyTorch provides a download link.

    Note

    The importNetworkFromPyTorch function can generate a custom layer when you import a PyTorch layer. For more information, see Algorithms. The function saves the generated custom layers in the +modelfile package.

    example

    net = importNetworkFromPyTorch(modelfile,Name=Value) imports a pretrained and traced PyTorch network with additional options specified by one or more name-value arguments. For example, PackageName="CustomLayers" saves any generated custom layers and associated functions in the +CustomLayers package in the current folder. If the PyTorchInputSizes name-value argument is specified, then the function may return the network net as an initialized dlnetwork.

    For information about how to trace a PyTorch model, see https://pytorch.org/docs/stable/generated/torch.jit.trace.html.

    Examples

    collapse all

    Import a pretrained and traced PyTorch model as an uninitialized dlnetwork object. Then, add an input layer to the imported network.

    This example imports the MNASNet (Copyright© Soumith Chintala 2016) PyTorch model. MNASNet is an image classification model that is trained with images from the ImageNet database. Download the mnasnet1_0 file, which is approximately 17 MB in size, from the MathWorks website.

    modelfile = matlab.internal.examples.downloadSupportFile("nnet", ...
        "data/PyTorchModels/mnasnet1_0.pt");

    Import the MNASNet model by using the importNetworkFromPyTorch function. The function imports the model as an uninitialized dlnetwork object without an input layer. The software displays a warning that contains information about the number of input layers, what type of input layer to add, and how to add an input layer.

    net = importNetworkFromPyTorch(modelfile)
    Warning: Network was imported as an uninitialized dlnetwork. Before using the network, add input layer(s):
    
    inputLayer1 = imageInputLayer(<inputSize1>, Normalization="none");
    net = addInputLayer(net, inputLayer1, Initialize=true);
    
    net = 
      dlnetwork with properties:
    
             Layers: [152×1 nnet.cnn.layer.Layer]
        Connections: [161×2 table]
         Learnables: [210×3 table]
              State: [104×3 table]
         InputNames: {'TopLevelModule_layers_0'}
        OutputNames: {'aten__linear12'}
        Initialized: 0
    
      View summary with summary.
    
    

    Specify the input size of the imported network and create an image input layer. Then, add the image input layer to the imported network and initialize the network by using the addInputLayer function.

    InputSize = [224 224 3];
    inputLayer = imageInputLayer(InputSize,Normalization="none");
    net = addInputLayer(net,inputLayer,Initialize=true);

    Analyze the imported network and view the input layer. The network is ready to use for prediction.

    analyzeNetwork(net)

    analyzenetwork_addinputlayer.png

    Import a pretrained and traced PyTorch model as an initialized dlnetwork object using the name-value argument PyTorchInputSizes.

    This example imports the MNASNet (Copyright© Soumith Chintala 2016) PyTorch model. MNASNet is an image classification model that is trained with images from the ImageNet database. Download the mnasnet1_0.pt file, which is approximately 17 MB in size, from the MathWorks website.

    modelfile = matlab.internal.examples.downloadSupportFile("nnet", ...
        "data/PyTorchModels/mnasnet1_0.pt");

    Import the MNASNet model by using the importNetworkFromPyTorch function with the name-value argument PyTorchInputSizes. We know that a 224x224 color image is a valid input size for this PyTorch model. The software automatically creates and adds the input layer for a batch of images. This allows the network to be imported as an initialized network in one line of code.

    net = importNetworkFromPyTorch(modelfile,PyTorchInputSizes=[NaN,3,224,224])
    net = 
      dlnetwork with properties:
    
             Layers: [153x1 nnet.cnn.layer.Layer]
        Connections: [162x2 table]
         Learnables: [210x3 table]
              State: [104x3 table]
         InputNames: {'InputLayer1'}
        OutputNames: {'aten__linear12'}
        Initialized: 1
    
      View summary with summary.
    
    

    The network is ready to use for prediction.

    Import a pretrained and traced PyTorch model as an uninitialized dlnetwork object. Then, initialize the imported network.

    This example imports the MNASNet (Copyright© Soumith Chintal 2016) PyTorch model. MNASNet is an image classification model that is trained with images from the ImageNet database. Download the mnasnet1_0 file, which is approximately 17 MB in size, from the MathWorks website.

    modelfile = matlab.internal.examples.downloadSupportFile("nnet", ...
        "data/PyTorchModels/mnasnet1_0.pt");

    Import the MNASNet model by using the importNetworkFromPyTorch function. The function imports the model as an uninitialized dlnetwork object.

    net = importNetworkFromPyTorch(modelfile)
    Warning: Network was imported as an uninitialized dlnetwork. Before using the network, add input layer(s):
    
    inputLayer1 = imageInputLayer(<inputSize1>, Normalization="none");
    net = addInputLayer(net, inputLayer1, Initialize=true);
    
    net = 
      dlnetwork with properties:
    
             Layers: [152×1 nnet.cnn.layer.Layer]
        Connections: [161×2 table]
         Learnables: [210×3 table]
              State: [104×3 table]
         InputNames: {'TopLevelModule_layers_0'}
        OutputNames: {'aten__linear12'}
        Initialized: 0
    
      View summary with summary.
    
    

    Specify the input size of the imported network. Then, create a random dlarray object that represents the input to the network. The data format of the dlarray object must have the dimensions "SSCB" (spatial, spatial, channel, batch) to represent a 2-D image input. For more information, see Data Formats for Prediction with dlnetwork.

    InputSize = [224 224 3];
    X = dlarray(rand(InputSize),"SSCB");

    Initialize the learnable parameters of the imported network by using the initialize function.

    net = initialize(net,X);

    Now the imported network is ready to use for prediction. Analyze the imported network.

    analyzeNetwork(net)

    analyzenetwork_initialize.png

    Import a pretrained and traced PyTorch model as an uninitialized dlnetwork object to classify an image.

    This example imports the MNASNet (Copyright© Soumith Chintala 2016) PyTorch model. MNASNet is an image classification model that is trained with images from the ImageNet database. Download the mnasnet1_0 file, which is approximately 17 MB in size, from the MathWorks website.

    modelfile = matlab.internal.examples.downloadSupportFile("nnet", ...
        "data/PyTorchModels/mnasnet1_0.pt");

    Import the MNASNet model by using the importNetworkFromPyTorch function. The function imports the model as an uninitialized dlnetwork object.

    net = importNetworkFromPyTorch(modelfile)
    Warning: Network was imported as an uninitialized dlnetwork. Before using the network, add input layer(s):
    
    inputLayer1 = imageInputLayer(<inputSize1>, Normalization="none");
    net = addInputLayer(net, inputLayer1, Initialize=true);
    
    net = 
      dlnetwork with properties:
    
             Layers: [152×1 nnet.cnn.layer.Layer]
        Connections: [161×2 table]
         Learnables: [210×3 table]
              State: [104×3 table]
         InputNames: {'TopLevelModule_layers_0'}
        OutputNames: {'aten__linear12'}
        Initialized: 0
    
      View summary with summary.
    
    

    Specify the input size of the imported network and create an image input layer. Then, add the image input layer to the imported network and initialize the network by using the addInputLayer function.

    InputSize = [224 224 3];
    inputLayer = imageInputLayer(InputSize,Normalization="none");
    net = addInputLayer(net,inputLayer,Initialize=true);

    Read the image you want to classify.

    Im = imread("peppers.png");

    Resize the image to the input size of the network. Show the image.

    InputSize = [224 224 3];
    Im = imresize(Im,InputSize(1:2));
    imshow(Im)

    The inputs to MNASNet require further preprocessing. Rescale the image. Then, normalize the image by subtracting the training images mean and dividing by the training images standard deviation. For more information, see Input Data Preprocessing.

    Im = rescale(Im,0,1);
    
    meanIm = [0.485 0.456 0.406];
    stdIm = [0.229 0.224 0.225];
    Im = (Im - reshape(meanIm,[1 1 3]))./reshape(stdIm,[1 1 3]);

    Convert the image to a dlarray object. Format the image with the dimensions "SSCB" (spatial, spatial, channel, batch).

    Im_dlarray = dlarray(single(Im),"SSCB");

    Get the class names from squeezenet, which is also trained with ImageNet images.

    squeezeNet = squeezenet;
    ClassNames = squeezeNet.Layers(end).Classes;

    Classify the image and find the predicted label.

    prob = predict(net,Im_dlarray);
    [~,label_ind] = max(prob);

    Display the classification result.

    ClassNames(label_ind)
    ans = categorical
         bell pepper 
    
    

    Import a pretrained and traced PyTorch model as an uninitialized dlnetwork object. Then, find the custom layers that the software generates.

    This example uses the findCustomLayers helper function.

    This example imports the MNASNet (Copyright© Soumith Chintala 2016) PyTorch model. MNASNet is an image classification model that is trained with images from the ImageNet database. Download the mnasnet1_0 file, which is approximately 17 MB in size, from the MathWorks website.

    modelfile = matlab.internal.examples.downloadSupportFile("nnet", ...
        "data/PyTorchModels/mnasnet1_0.pt");

    Import the MNASNet model by using the importNetworkFromPyTorch function. The function imports the model as an uninitialized dlnetwork object.

    net = importNetworkFromPyTorch(modelfile);
    Warning: Network was imported as an uninitialized dlnetwork. Before using the network, add input layer(s):
    
    inputLayer1 = imageInputLayer(<inputSize1>, Normalization="none");
    net = addInputLayer(net, inputLayer1, Initialize=true);
    

    The importNetworkFromPyTorch function generates custom layers for the PyTorch layers that the function cannot convert to built-in MATLAB layers or functions. For more information, see Algorithms. The software saves the automatically generated custom layers to the +mnasnet1_0 package in the current folder and the associated functions to the +ops subpackage. To see the custom layers and associated functions, inspect the package.

    custom_layers_package.png

    You can also find the indices of the generated custom layers by using the findCustomLayers helper function. Display the custom layers.

    ind = findCustomLayers(net.Layers,'+mnasnet1_0');
    net.Layers(ind)
    ans = 
      13×1 Layer array with layers:
    
         1   'aten__add0'         Custom Layer   mnasnet1_0.aten__add0
         2   'aten__add1'         Custom Layer   mnasnet1_0.aten__add1
         3   'aten__add2'         Custom Layer   mnasnet1_0.aten__add2
         4   'aten__add3'         Custom Layer   mnasnet1_0.aten__add3
         5   'aten__add4'         Custom Layer   mnasnet1_0.aten__add4
         6   'aten__add5'         Custom Layer   mnasnet1_0.aten__add5
         7   'aten__add6'         Custom Layer   mnasnet1_0.aten__add6
         8   'aten__add7'         Custom Layer   mnasnet1_0.aten__add7
         9   'aten__add8'         Custom Layer   mnasnet1_0.aten__add8
        10   'aten__add9'         Custom Layer   mnasnet1_0.aten__add9
        11   'aten__dropout_11'   Custom Layer   mnasnet1_0.aten__dropout_11
        12   'aten__linear12'     Custom Layer   mnasnet1_0.aten__linear12
        13   'aten__mean10'       Custom Layer   mnasnet1_0.aten__mean10
    

    Helper Function

    The findCustomLayers helper function returns the indices of the custom layers that importNetworkFromPyTorch automatically generates.

    function indices = findCustomLayers(layers,PackageName)
    
    s = what(['.' filesep PackageName]);
    
    indices = zeros(1,length(s.m));
    for i = 1:length(layers)
        for j = 1:length(s.m)
            if strcmpi(class(layers(i)),[PackageName(2:end) '.' s.m{j}(1:end-2)])
                indices(j) = i;
            end
        end
    end
    
    end

    This example shows how to import a network from PyTorch and train the network to classify new images. Use the importNetworkFromPytorch function to import the network as a uninitialized dlnetwork object. Train the network by using a custom training loop.

    This example uses the modelLoss, modelPredictions, and preprocessMiniBatchPredictors helper functions.

    This example also uses the supporting file new_fcLayer. To access the supporting file, open the example in Live Editor.

    Load Data

    Unzip the MerchData data set, which contains 75 images. Load the new images as an image datastore. The imageDatastore function automatically labels the images based on folder names and stores the data as an ImageDatastore object. Divide the data into training and validation data sets. Use 70% of the images for training and 30% for validation.

    unzip("MerchData.zip");
    imds = imageDatastore("MerchData", ...
        IncludeSubfolders=true, ...
        LabelSource="foldernames"); 
    [imdsTrain,imdsValidation] = splitEachLabel(imds,0.7);

    The network you use in this example requires input images with a size of 224-by-224-by-3. To automatically resize the training images, use an augmented image datastore. Randomly translate the images up to 30 pixels in the horizontal and vertical axes. Data augmentation helps prevent the network from overfitting and memorizing the exact details of the training images.

    inputSize = [224 224 3];
    
    pixelRange = [-30 30];
    scaleRange = [0.9 1.1];
    imageAugmenter = imageDataAugmenter(...
        RandXReflection=true, ...
        RandXTranslation=pixelRange, ...
        RandYTranslation=pixelRange, ...
        RandXScale=scaleRange, ...
        RandYScale=scaleRange);
    augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain, ...
        DataAugmentation=imageAugmenter);

    To automatically resize the validation images without performing further data augmentation, use an augmented image datastore without specifying any additional preprocessing operations.

    augimdsValidation = augmentedImageDatastore(inputSize(1:2),imdsValidation);

    Determine the number of classes in the training data.

    classes = categories(imdsTrain.Labels);
    numClasses = numel(classes);

    Import Network

    Download the MNASNet (Copyright© Soumith Chintala 2016) PyTorch model. MNASNet is an image classification model that is trained with images from the ImageNet database. Download the mnasnet1_0 file, which is approximately 17 MB in size, from the MathWorks website.

    modelfile = matlab.internal.examples.downloadSupportFile("nnet", ...
        "data/PyTorchModels/mnasnet1_0.pt");

    Import the MNASNet model as an uninitialized dlnetwork object by using the importNetworkFromPyTorch function.

    net = importNetworkFromPyTorch(modelfile)
    Warning: Network was imported as an uninitialized dlnetwork. Before using the network, add input layer(s):
    
    inputLayer1 = imageInputLayer(<inputSize1>, Normalization="none");
    net = addInputLayer(net, inputLayer1, Initialize=true);
    
    net = 
      dlnetwork with properties:
    
             Layers: [152×1 nnet.cnn.layer.Layer]
        Connections: [163×2 table]
         Learnables: [210×3 table]
              State: [104×3 table]
         InputNames: {'TopLevelModule_layers_0'}
        OutputNames: {'aten__linear12'}
        Initialized: 0
    
      View summary with summary.
    
    

    Display the final layer of the imported network. The aten__linear12 layer is a custom layer generated by the importNetworkFromPyTorch function and the last learnable layer of the imported network. This layer contains information about how to combine the features that the network extracts into class probabilities and a loss value.

    net.Layers(end)
    ans = 
      aten__linear12 with properties:
    
                                      Name: 'aten__linear12'
                                 NumInputs: 2
                                InputNames: {'in'  'in_rank'}
    
       Learnable Parameters
        TopLevelModule_classifier_1_weight: [1280×1000 single]
          TopLevelModule_classifier_1_bias: [0.0493 -0.0804 -0.0906 -0.1006 0.1332 -0.0767 -0.0788 -0.0026 -0.0525 -0.1215 -0.1635 -0.1147 -0.1421 -0.1148 -0.0586 -0.2150 -0.0970 -0.0798 -5.4136e-04 -0.0968 0.0479 0.0780 0.0015 -0.1375 -0.0485 -0.1223 … ]
    
       State Parameters
        No properties.
    
      Show all properties
    
    

    Replace Final Layer

    To retrain the imported network to classify new images, replace the final layers with a new fully connected layer. The new layer new_fclayer is adapted to the new data set and must also be a custom layer because it has two inputs.

    Initialize the new_fcLayer layer and replace the aten__linear12 layer with new_fcLayer.

    newLayer = new_fcLayer("fc1","Custom Layer", ...
        {'in' 'in_rank'},{'out'},numClasses);
    net = replaceLayer(net,"aten__linear12",newLayer);

    Add a softmax layer to the network and connect the softmax layer to the new fully connected layer.

    net = addLayers(net,softmaxLayer(Name="sm1"));
    net = connectLayers(net,"fc1","sm1");

    Add Input Layer

    Add an image input layer to the network and initialize the network.

    inputLayer = imageInputLayer(inputSize,Normalization="none");
    net = addInputLayer(net,inputLayer,Initialize=true);

    Analyze the network. View the first layer and the final layers.

    analyzeNetwork(net)

    analyzenetwork_inputlayer.png

    analyzenetwork_finallayers.png

    Define Model Loss Function

    Training a deep neural network is an optimization task. By treating a neural network as a function f(X;θ), where X is the network input and θ is the set of learnable parameters, you can optimize θ so that it minimizes some loss value based on the training data. For example, optimize the learnable parameters θ such that, for inputs X with corresponding targets T, they minimize the error between the predictions Y=f(X;θ) and T.

    Create the modelLoss function, listed in the Model Loss Function section of the example, which takes as input the dlnetwork object and a mini-batch of input data with corresponding targets. The function returns the loss, the gradients of the loss with respect to the learnable parameters, and the network state.

    Specify Training Options

    Train for 15 epochs with a mini-batch size of 20.

    numEpochs = 15;
    miniBatchSize = 20;

    Specify the options for SGDM optimization. Specify an initial learning rate of 0.001 with a decay of 0.005, and a momentum of 0.9.

    initialLearnRate = 0.001;
    decay = 0.005;
    momentum = 0.9;

    Train Network

    Create a minibatchqueue object that processes and manages mini-batches of images during training. For each mini-batch, perform these steps:

    1. Use the custom mini-batch preprocessing function preprocessMiniBatch (defined at the end of this example) to convert the labels to one-hot encoded variables.

    2. Format the image data with the dimension labels "SSCB" (spatial, spatial, channel, batch). By default, the minibatchqueue object converts the data to dlarray objects with the underlying type single. Do not format the class labels.

    3. Train on a GPU if one is available. By default, the minibatchqueue object converts each output to a gpuArray object if a GPU is available. Using a GPU requires a Parallel Computing Toolbox™ license and a supported GPU device. For information about supported devices, see GPU Computing Requirements (Parallel Computing Toolbox).

    mbq = minibatchqueue(augimdsTrain,...
        MiniBatchSize=miniBatchSize,...
        MiniBatchFcn=@preprocessMiniBatch,...
        MiniBatchFormat=["SSCB" ""]);

    Initialize the velocity parameter for the gradient descent with momentum (SGDM) solver.

    velocity = [];

    Calculate the total number of iterations for the training progress monitor.

    numObservationsTrain = numel(imdsTrain.Files);
    numIterationsPerEpoch = ceil(numObservationsTrain/miniBatchSize);
    numIterations = numEpochs*numIterationsPerEpoch;

    Initialize the trainingProgressMonitor object. Because the timer starts when you create the monitor object, create the object immediately after the training loop.

    monitor = trainingProgressMonitor(Metrics="Loss",Info=["Epoch","LearnRate"],XLabel="Iteration");

    Train the network using a custom training loop. For each epoch, shuffle the data and loop over mini-batches of data. For each mini-batch, perform these steps:

    1. Evaluate the model loss, gradients, and state using the dlfeval and modelLoss functions and then update the network state.

    2. Determine the learning rate for the time-based decay learning rate schedule.

    3. Update the network parameters using the sgdmupdate function.

    4. Update the loss, learning rate, and epoch values in the training progress monitor.

    5. Stop if the Stop property is true. The Stop property value of the TrainingProgressMonitor object changes to true when you click the Stop button.

    epoch = 0;
    iteration = 0;
    
    % Loop over epochs.
    while epoch < numEpochs && ~monitor.Stop
        
        epoch = epoch + 1;
    
        % Shuffle data.
        shuffle(mbq);
        
        % Loop over mini-batches.
        while hasdata(mbq) && ~monitor.Stop
    
            iteration = iteration + 1;
            
            % Read mini-batch of data.
            [X,T] = next(mbq);
            
            % Evaluate the model gradients, state, and loss using dlfeval and the
            % modelLoss function and update the network state.
            [loss,gradients,state] = dlfeval(@modelLoss,net,X,T);
            net.State = state;
            
            % Determine learning rate for time-based decay learning rate schedule.
            learnRate = initialLearnRate/(1 + decay*iteration);
            
            % Update the network parameters using the SGDM optimizer.
            [net,velocity] = sgdmupdate(net,gradients,velocity,learnRate,momentum);
            
            % Update the training progress monitor.
            recordMetrics(monitor,iteration,Loss=loss);
            updateInfo(monitor,Epoch=epoch,LearnRate=learnRate);
            monitor.Progress = 100*iteration/numIterations;
        end
    end

    Classify Validation Images

    Test the classification accuracy of the model by comparing the predictions on the validation set with the true labels.

    After training, making predictions on new data does not require the labels. Create a minibatchqueue object containing only the predictors of the test data:

    • To ignore the labels for testing, set the number of outputs of the mini-batch queue to 1.

    • Specify the same mini-batch size that you use for training.

    • Preprocess the predictors using the preprocessMiniBatchPredictors function, listed at the end of the example.

    • For the single output of the datastore, specify the mini-batch format "SSCB" (spatial, spatial, channel, batch).

    numOutputs = 1;
    
    mbqTest = minibatchqueue(augimdsValidation,numOutputs, ...
        MiniBatchSize=miniBatchSize, ...
        MiniBatchFcn=@preprocessMiniBatchPredictors, ...
        MiniBatchFormat="SSCB");

    Loop over the mini-batches and classify the images using the modelPredictions function, listed at the end of the example.

    YTest = modelPredictions(net,mbqTest,classes);

    Evaluate the classification accuracy.

    TTest = imdsValidation.Labels;
    accuracy = mean(TTest == YTest)
    accuracy = 0.9500
    

    Visualize the predictions in a confusion chart. Large values on the diagonal indicate accurate predictions for the corresponding class. Large values on the off-diagonal indicate strong confusion between the corresponding classes.

    figure
    confusionchart(TTest,YTest)

    Helper Functions

    Model Loss Function

    The modelLoss function takes as input a dlnetwork object net and a mini-batch of input data X with corresponding targets T. The function returns the loss, the gradients of the loss with respect to the learnable parameters in net, and the network state. To compute the gradients automatically, use the dlgradient function.

    function [loss,gradients,state] = modelLoss(net,X,T)
    
    % Forward data through network.
    [Y,state] = forward(net,X);
    
    % Calculate cross-entropy loss.
    loss = crossentropy(Y,T);
    
    % Calculate gradients of loss with respect to learnable parameters.
    gradients = dlgradient(loss,net.Learnables);
    
    end

    Model Predictions Function

    The modelPredictions function takes as input a dlnetwork object net, a minibatchqueue of input data mbq, and the network classes. The function computes the model predictions by iterating over all the data in the minibatchqueue object. The function uses the onehotdecode function to find the predicted class with the highest score.

    function Y = modelPredictions(net,mbq,classes)
    
    Y = [];
    
    % Loop over mini-batches.
    while hasdata(mbq)
        X = next(mbq);
    
        % Make prediction.
        scores = predict(net,X);
    
        % Decode labels and append to output.
        labels = onehotdecode(scores,classes,1)';
        Y = [Y; labels];
    end
    
    end

    Mini Batch Preprocessing Function

    The preprocessMiniBatch function preprocesses a mini-batch of predictors and labels using these steps:

    1. Preprocess the images using the preprocessMiniBatchPredictors function.

    2. Extract the label data from the incoming cell array and concatenate the result into a categorical array along the second dimension.

    3. One-hot encode the categorical labels into numeric arrays. Encoding into the first dimension produces an encoded array that matches the shape of the network output.

    function [X,T] = preprocessMiniBatch(dataX,dataT)
    
    % Preprocess predictors.
    X = preprocessMiniBatchPredictors(dataX);
    
    % Extract label data from cell and concatenate.
    T = cat(2,dataT{1:end});
    
    % One-hot encode labels.
    T = onehotencode(T,1);
    
    end

    Mini-Batch Predictors Preprocessing Function

    The preprocessMiniBatchPredictors function preprocesses a mini-batch of predictors by extracting the image data from the input cell array and concatenating the result into a numeric array. For grayscale input, concatenating over the fourth dimension adds a third dimension to each image to use as a singleton channel dimension.

    function X = preprocessMiniBatchPredictors(dataX)
    
    % Concatenate.
    X = cat(4,dataX{1:end});
    
    end

    Input Arguments

    collapse all

    Name of the PyTorch model file, specified as a character vector or string scalar. modelfile must be in the current folder, or you must include a full or relative path to the file. The PyTorch model must be pretrained and traced over one inference iteration.

    For information about how to trace a PyTorch model, see https://pytorch.org/docs/stable/generated/torch.jit.trace.html.

    Example: "mobilenet_v3.pt"

    Name-Value Arguments

    Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

    Example: importNetworkFromPyTorch(modelfile,PackageName="CustomLayers") imports the network in modelfile and saves the custom layers package +PackageName in the current folder.

    Name of the custom layers package in which importNetworkFromPyTorch saves custom layers, specified as a character vector or string scalar. importNetworkFromPyTorch saves the custom layers +PackageName package in the current folder. If you do not specify PackageName, then importNetworkFromPyTorch saves the custom layers in the +modelfile package in the current folder. For more information about packages, see Packages Create Namespaces.

    importNetworkFromPyTorch tries to generate a custom layer when you import a custom PyTorch layer or when the software cannot convert a PyTorch layer into an equivalent built-in MATLAB® layer. importNetworkFromPyTorch saves each generated custom layer to a separate MATLAB code file in +PackageName. To view or edit a custom layer, open the associated MATLAB code file. For more information about custom layers, see Custom Layers.

    The +PackageName package can also contain the +ops subpackage. This subpackage contains MATLAB functions corresponding to PyTorch operators that the automatically generated custom layers use. importNetworkFromPyTorch saves the associated MATLAB function for each operator in a separate MATLAB code file in the +ops subpackage. The object functions of dlnetwork, such as the predict function, use these operators when it interacts with the custom layers. The +ops subpackage can also contain placeholder functions. For more information, see Placeholder Functions.

    Example: PackageName="mobilenet_v3"

    Dimension sizes of the PyTorch network inputs, specified as a numeric array, string scalar, or cell array. The dimension input order is the same as in the PyTorch network. You can specify PyTorchInputSizes as a numeric array only when the network has a single nonscalar input. If the network has multiple inputs, PyTorchInputSizes must be a cell array of the input sizes. For an input whose size or shape is not known specify PyTorchInputSize as "unknown". For an input that corresponds to a 0-dimensional scalar in PyTorch, specify PyTorchInputSize as "scalar".

    The standard input layers that importNetworkFromPyTorch supports are ImageInputLayer (SSCB), FeatureInputLayer (CB), ImageInputLayer3D (SSSCB), and SequenceInputLayer (CBT). Here, S is spatial, C is channel, B is batch, and T is time. importNetworkFromPyTorch also supports nonstandard inputs using PyTorchInputSizes. For example, import the network and specify the input dimension sizes with this function call: net = importNetworkFromPyTorch("nonStandardModel.pt",PyTorchInputSizes=[1 3 224]). Then, initialize the network with a U-labelled dlarray object, where U is unknown, with these function calls: X = dlarray(rand(1 3 224),"UUU") and net = initialize(net,X). The software interprets the U-labelled dlarray as data in PyTorch order.

    Example: PyTorchInputSizes=[NaN 3 224 224] is a network with one input that is a batch of images.

    Example: PyTorchInputSizes={[NaN 3 224 224],"unknown"} is a network with two inputs. The first input is a batch of images and the second input has unknown size.

    Data Types: numeric array | string | cell array

    Output Arguments

    collapse all

    Pretrained PyTorch network, returned as an uninitialized dlnetwork object. Before using the imported network, you must add an input layer or initialize the network. For examples, see Import Network from PyTorch and Add Input Layer and Import Network from PyTorch and Initialize.

    Limitations

    • The importNetworkFromPyTorch function fully supports PyTorch version 1.10.0. The function can import most models created in other PyTorch versions.

    • The importNetworkFromPyTorch function can import only image classification and segmentation models.

    More About

    collapse all

    Conversion of PyTorch Layers and Functions into Built-In MATLAB Layers and Functions

    The importNetworkFromPyTorch function supports the PyTorch layers, functions, and operators listed in this section for conversion into built-in MATLAB layers and functions with dlarray support. For more information about functions that operate on dlarray objects, see List of Functions with dlarray Support. The conversion process often has limitations.

    Conversion of PyTorch Layers

    This table shows the correspondence between PyTorch layers and Deep Learning Toolbox layers. In some cases, when importNetworkFromPyTorch cannot convert a PyTorch layer into a MATLAB layer, the software converts the PyTorch layer into a Deep Learning Toolbox function with dlarray support.

    PyTorch LayerCorresponding Deep Learning Toolbox LayerAlternative Deep Learning Toolbox Function
    torch.nn.AdaptiveAvgPool2dnnet.pytorch.layer.AdaptiveAveragePoolingLayerpyAdaptiveAvgPool2d
    torch.nn.AvgPool2daveragePooling2dLayerNot applicable
    torch.nn.BatchNorm2dbatchNormalizationLayerNot applicable
    torch.nn.Conv1dconvolution1dLayerpyConvolution
    torch.nn.Conv2dconvolution2dLayerNot applicable
    torch.nn.ConvTranspose1dtransposedConv1dLayerpyConvolution
    torch.nn.ConvTranspose2dtransposedConv2dLayerpyConvolution
    torch.nn.DropoutdropoutLayerNot applicable
    torch.nn.GELUgeluLayerpyGelu
    torch.nn.GroupNormgroupNormalizationLayerNot applicable
    torch.nn.LayerNormlayerNormalizationLayerNot applicable
    torch.nn.LeakyReLUleakyReluLayerpyLeakyRelu
    torch.nn.LinearfullyConnectedLayerpyLinear
    torch.nn.MaxPool2dmaxPooling2dLayerNot applicable
    torch.nn.PReLUnnet.pytorch.layer.PReLULayerNot applicable
    torch.nn.ReLUreluLayerrelu
    torch.nn.SiLUswishLayerpySilu
    torch.nn.SigmoidsigmoidLayerpySigmoid
    torch.nn.Softmaxnnet.pytorch.layer.SoftmaxLayerpySoftmax
    torch.nn.Upsampleresize2dLayer (Image Processing Toolbox)pyUpsample (requires Image Processing Toolbox™)
    torch.nn.UpsamplingNearest2dresize2dLayer (Image Processing Toolbox)pyUpsample (requires Image Processing Toolbox)
    torch.nn.UpsamplingBilinear2dresize2dLayer (Image Processing Toolbox)pyUpsample (requires Image Processing Toolbox)
    Conversion of PyTorch Functions

    This table shows the correspondence between PyTorch functions and Deep Learning Toolbox functions.

    PyTorch FunctionCorresponding Deep Learning Toolbox Function
    torch.nn.functional.adaptive_avg_pool2dpyAdaptiveAvgPool2d
    torch.nn.functional.avg_pool2dpyAvgPool2d
    torch.nn.functional.conv1dpyConvolution
    torch.nn.functional.conv2dpyConvolution
    torch.nn.functional.dropoutpyDropout
    torch.nn.functional.gelupyGelu
    torch.nn.functional.hardsigmoidpyHardSigmoid
    torch.nn.functional.hardswishpyHardSwish
    torch.nn.functional.layer_normpyLayerNorm
    torch.nn.functional.leaky_relupyLeakyRelu
    torch.nn.functional.linearpyLinear
    torch.nn.functional.log_softmaxpyLogSoftmax
    torch.nn.functional.max_pool2dpyMaxPool2d
    torch.nn.functional.relurelu
    torch.nn.functional.silupySilu
    torch.nn.functional.softmaxpySoftmax
    Conversion of PyTorch Mathematical Operators

    This table shows the correspondence between PyTorch mathematical operators and Deep Learning Toolbox functions. The importNetworkFromPyTorch first tries to convert the cat PyTorch operator to a concatenation layer, then to a function.

    PyTorch OperatorCorresponding Deep Learning Toolbox Layer or FunctionAlternative Deep Learning Toolbox Function
    +, -, /pyElementwiseBinaryNot applicable
    torch.argmaxpyArgMaxNot applicable
    torch.bmmpyMatMulNot applicable
    torch.catconcatenationLayerpyConcat
    torch.chunkpyChunkNot applicable
    torch.clamp_minpyClampMinNot applicable
    torch.concatpyConcatNot applicable
    torch.detachpyDetachNot applicable
    torch.floor_divpyElementwiseBinaryNot applicable
    torch.gatherpyGatherNot applicable
    torch.matmulpyMatMulNot applicable
    torch.maxpyMaxBinary/pyMaxUnaryNot applicable
    torch.meanpyMeanNot applicable
    torch.mul, *multiplicationLayerpyElementwiseBinary
    torch.normpyNormNot applicable
    torch.permutepyPermuteNot applicable
    torch.powpyElementwiseBinaryNot applicable
    torch.repeatpyRepeatNot applicable
    torch.reshapepyViewNot applicable
    torch.sizepySizeNot applicable
    torch.splitpySplitWithSizesNot applicable
    torch.sqrtpyElementwiseBinaryNot applicable
    torch.squarepySquareNot applicable
    torch.squeezepySqueezeNot applicable
    torch.stackpyStackNot applicable
    torch.sumpySumNot applicable
    torch.tpyTNot applicable
    torch.topyToNot applicable
    torch.transposepyTransposeNot applicable
    torch.unsqueezepyUnsqueezeNot applicable
    torch.zerospyZerosNot applicable
    Conversion of PyTorch Matrix Operators

    This table shows the correspondence between PyTorch matrix operators and Deep Learning Toolbox functions.

    PyTorch OperatorCorresponding Deep Learning Toolbox Function or Operator
    Indexing (for example, X[:,1])pySlice
    torch.tensor.contiguous=
    torch.tensor.expandpyExpand
    torch.tensor.expand_aspyExpandAs
    torch.tensor.selectpySlice
    torch.tensor.viewpyView

    Placeholder Functions

    When the importNetworkFromPyTorch function cannot convert a PyTorch layer into a built-in MATLAB layer or generate a custom layer with associated MATLAB functions, the function creates a custom layer with a placeholder function. You must complete the placeholder function before you can use the network.

    This code snippet defines a custom layer with the pyAtenUnsupportedOperator placeholder function.

    classdef UnsupportedOperator < nnet.layer.Layer
    
    function [output] = predict(obj,arg1)
    % Placeholder function for aten::<unsupportedOperator>
    output= pyAtenUnsupportedOperator(arg1,params);
    end
    
    end

    Tips

    • To use a pretrained network for prediction or transfer learning on new images, you must preprocess your images in the as same way the images that you use to train the imported model. The most common preprocessing steps are resizing images, subtracting image average values, and converting the images from BGR format to RGB format.

      • To resize images, use imresize. For example, imresize(image,[227 227 3]).

      • To convert images from RGB to BGR format, use flip. For example, flip(image,3).

      For more information about preprocessing images for training and prediction, see Preprocess Images for Deep Learning.

    • The members of the +PackageName package are not accessible if the package parent folder is not on the MATLAB path. For more information, see Packages and the MATLAB Path.

    • MATLAB uses one-based indexing, whereas Python® uses zero-based indexing. In other words, the first element in an array has an index of 1 and 0 in MATLAB and Python, respectively. For more information about MATLAB indexing, see Array Indexing. In MATLAB, to use an array of indices (ind) created in Python, convert the array to ind+1.

    • For more tips, see Tips on Importing Models from TensorFlow, PyTorch, and ONNX.

    Algorithms

    The importNetworkFromPyTorch function imports a PyTorch layer into MATLAB by trying these steps in order:

    1. The function tries to import the PyTorch layer as a built-in MATLAB layer. For more information, see Conversion of PyTorch Layers.

    2. The function tries to import the PyTorch layer as a built-in MATLAB function. For more information, see Conversion of PyTorch Layers.

    3. The function tries to import the PyTorch layer as a custom layer. importNetworkFromPyTorch saves the generated custom layers and the associated functions in the +PackageName package. For an example, see Import Network from PyTorch and Find Generated Custom Layers.

    4. The function imports the PyTorch layer as a custom layer with a placeholder function. You must complete the placeholder function before you can use the network, see Placeholder Functions.

    In the first three cases, the imported network is ready for prediction after you initialize it.

    Version History

    Introduced in R2022b

    expand all