Segment CT Scan Using MONAI Label

This example shows how to apply deep learning models from the Medical Open Network for AI (MONAI) Label platform to segment a CT scan.

The MONAI Label [1][2] platform provides fully automated and interactive deep learning models for segmenting radiology images. You can use MONAI Label within the Medical Image Labeler app by connecting to a MONAI Label server. A server is an instance of the MONAI Label software that contains the deep learning models. You can install and run a MONAI label server on your local machine or on a remote machine. Installing a local server requires the Medical Imaging Toolbox™ Interface for MONAI Label Library support package. For a detailed introduction, see Get Started with MONAI Label in Medical Image Labeler. The support package requires Medical Imaging Toolbox, Computer Vision Toolbox™, and Deep Learning Toolbox™. For more information about installing add-ons, see Get and Manage Add-Ons.

Download Data

This example uses a subset of the TotalSegmentator data set (version 2) [3]. The full data set contains 1228 CT images, including abdominal and whole body scans, with segmentation labels for up to 117 anatomical structures per scan. The subset of data used here includes two scans and the total size is approximately 92 MB. Download the data from the MathWorks® website, then unzip the file.

zipFile = matlab.internal.examples.downloadSupportFile("medical","TotalSegmentatorDataV2_0_1.zip");
filepath = fileparts(zipFile);
unzip(zipFile,filepath)
dataFolder = fullfile(filepath,"TotalSegmentatorDataV2_0_1");

Open Medical Image Labeler

Open the Medical Image Labeler app from the Apps tab on the MATLAB® Toolstrip, under Image Processing and Computer Vision. You can also load the app by using the medicalImageLabeler command.

Create New Volume Labeling Session

To start a new 3-D labeling session, on the app toolstrip, click New Session and select New Volume Session (3-D). In the Create a new session folder dialog box, specify a location in which to save the new session folder by entering a path, or select Browse and navigate to your desired location. In the New Session Folder dialog box, specify a name for the folder for this labeling session. Then, select Create Session.

Load Image Data into Medical Image Labeler

To load an image into the Medical Image Labeler app, on the app toolstrip, click Import. Then, under Data, select From File. Browse to the location where you downloaded the data, specified by the dataFolder workspace variable, and select the file lung_043.nii. For a volume session, the imported data file can be a single DICOM or NIfTI file containing a 3-D image volume, or a directory containing multiple DICOM files corresponding to a single image volume.

The name of the imported image is visible in the Data Browser pane. The no labels symbol next to the filename indicates that the volume does not contain any pixel labels. You can import multiple 3-D image files into a volume session. All files imported into a single app session must label the same regions of interest, such as a tumor or lung, and the app exports all labels together as one groundTruthMedical object.

Start MONAI Label

To use MONAI Label, you must connect to a MONAI Label server. You can install and run a MONAI server locally, or connect to a server running on a remote machine. To learn more about servers, see Get Started with MONAI Label in Medical Image Labeler. In this example, you connect to a local server installed using the Medical Imaging Toolbox Interface for MONAI Label Library support package.

To start a server, on the MONAI Label tab of the app toolstrip, select Start MONAI Label > On the local machine. The first time you try to connect to a local server, Medical Image Labeler prompts you to install the Medical Imaging Toolbox Interface for MONAI Label Library support package, which includes the MONAI Label software. Follow the link in the dialog box to open the Add-Ons Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. When you install the support package, MATLAB downloads the MONAI Label models, which requires an internet connection.

Once the installation is complete, try starting the server again. You do not need to restart the app.

Choose Model

Select a model to try from the Deep Learning Models gallery. Choose a model based on the labels you want to predict. Some models, such as the segmentation spleen and pancreas ct dints segmentation models, focus on one organ, while others, such as segmentation and deepedit, predict multiple organ labels. If multiple models predict the labels you want, you can compare models by using trial and error, or pick the model whose training data is similar to the data to segment. For a list of links to MONAI Label documentation for each model, see Get Started with MONAI Label in Medical Image Labeler.

For this example, the target label regions are the lungs, liver, kidneys, and spleen. To start, select the segmentation model, which includes labels for all of the target regions.

Perform Label Mapping

After you select a model for the first time in a labeling session, the Label Mapping dialog box opens automatically. Select the labels you want to predict, then map each option to a new or existing label definition. Label definition refers to labels in the Label Definitions pane of Medical Image Labeler.

If you map any MONAI Label labels to the <Create New> option, the new labels appear in the Label Definitions pane.

Run the Model

In the app toolstrip, select Run. The model predicts the labels you selected to predict in the Label Mapping dialog box. Certain models can take several minutes to run. When the model finishes, the slice planes update to show the predicted labels.

Examine the results. To view the label overlays more clearly, consider adjusting the contrast of the underlying CT image and the overlay opacity. To change the contrast, in the Labeler tab, select the Window Level tool. Click in a slice plane, and drag left to increase the contrast, or drag up to increase the brightness. You can also set the values directly by typing values for the Window Level and Window Width parameters in the app toolstrip. For this data, a level value of 120 and window value of 400 work well. To adjust the label opacity, drag the Label Opacity slider in the app toolstrip to about one-third of the maximum value.

Zoom within a slice pane by using the scroll wheel.

Overall, the model has performed well. Depending on your application and the precision it requires, you may need to refine the labels.

Refine Labels Using Label Mapping

Assume that instead of labeling the lobes of each lung, you decide to label each lung as one region. To create unified lung labels, map all the MONAI Label labels for each lung to one Medical Image Labeler label definition. If you know you want to combine labels upfront, you can follow this approach the first time you run the model.

First, delete the app labels for the lobes. In the Label Definitions pane, right-click the lung_upper_lobe_left label and select Delete. In the dialog box, select Yes to confirm the choice. Repeat to delete all of the lung labels. Then, click Create Label Definition, and create a new label definition named lung_left. Repeat to create a lung_right label.

Reopen the Label Mapping dialog box by selecting Label Mapping in the MONAI Label tab of the app toolstrip. Reselect the lung lobe labels, and map the left lobe labels to the lung_left mask and the right lobe labels to the lung_right label definition.

Click Run to rerun the model. As expected, each lung label contains all of the corresponding lobes.

Refine Labels Using Automated Model

You can also refine a label by trying a different automated model. For example, assume you notice a few areas of the spleen label that need refinement.

The spleen is predicted by several models, including segmentation spleen and spleen ct segmentation. These models have all been trained to label the spleen, but differences in the model architecture and training data may yield different results for different image data sets. For this example, compare the results from a dedicated spleen model to the initial segmentation model prediction.

Select the segmentation spleen model from the Deep Learning Models gallery in the app toolstrip. In the Label Mapping dialog box, select the spleen label, and map it to the existing spleen label definition. Click OK.

Click Run to run the model, and then visually inspect the updated spleen label to assess whether the segmentation improved. You can keep the updated label, or press Ctrl+Z to revert the label.

Refine Labels Using Interactive Model

The support package provides two interactive models, deepedit and deepgrow 3d, that accept drawn foreground and background markers as input to the model. The deepedit model is a hybrid model that can be run with or without markers. The deepgrow 3d model requires at least one foreground marker. For this example, use the deepedit model interactively to refine the kidney_right label.

From the Deep Learning Models gallery, select the deepedit model. In the Label Mapping dialog box, select the rightKidney label and map it to the existing kidney_right label definition. Click OK.

To draw foreground markers for the right kidney, first select kidney_right in the Label Definitions pane. Then, on the MONAI Label tab of the app toolstrip, select Mark Foreground. Click in a slice pane and drag to start drawing inside the kidney region. Once you release the mouse button, the foreground drawing becomes a green freehand shape. Drag the green waypoints to adjust the shape, or right-click the line and select Add Waypoint to add a new draggable waypoint. To add background markers, in the app toolstrip, select Mark Background. Then, click in the same slice pane, outside the kidney region, and drag to draw the markers. The background markers appear red.

You can draw markers in multiple slices. When you are done adding markers, click Run to run the model. Visually inspect the updated kidney_right label to assess whether the segmentation has improved. You can keep the updated label, or press Ctrl+Z to revert the label. If the segmentation does not improve, you can try drawing additional markers and rerun the model. Otherwise, you can manually refine the labels.

Refine Labels Manually

For many data sets, MONAI Label models can provide a helpful starting point that requires manual refinement. Once you feel you have achieved the best results possible with MONAI Label, you can clean up labels using any of the tools in the Draw tab of the app toolstrip. For example, add missing pixels to the kidney_right label by using the Paint Brush tool, as shown in this image. To learn more about the manual and semi-automated drawing tools in Medical Image Labeler, see Get Started with Medical Image Labeler.

Disconnect from Server and End Labeling

When you are done labeling, disconnect from the MONAI Label server and save your results before closing the app session. To disconnect from the server, in the MONAI Label tab of the app toolstrip, select Disconnect Server. If you are using a local server, the app disconnects from and stops the server. If you are connected to a remote server, the app disconnects from the server, but does not stop the server.

To save your labeling session, in the Labeler tab of the app toolstrip, select Save. Throughout the labeling session, the app saves the label images and a groundTruthMedical object in the session folder you selected when you created the session. For more details about saving and accessing label results, see How Medical Image Labeler Manages Ground Truth Labels.

References

[1] Diaz-Pinto, Andres, Sachidanand Alle, Vishwesh Nath, Yucheng Tang, Alvin Ihsani, Muhammad Asad, Fernando Pérez-García, et al. “MONAI Label: A Framework for AI-Assisted Interactive Labeling of 3D Medical Images,” 2022. https://doi.org/10.48550/ARXIV.2203.12362.

[2] Diaz-Pinto, Andres, Pritesh Mehta, Sachidanand Alle, Muhammad Asad, Richard Brown, Vishwesh Nath, Alvin Ihsani, et al. “DeepEdit: Deep Editable Learning for Interactive Segmentation of 3D Medical Images.” In Data Augmentation, Labelling, and Imperfections, edited by Hien V. Nguyen, Sharon X. Huang, and Yuan Xue, 11–21. Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 2022. https://doi.org/10.1007/978-3-031-17027-0_2.

[3] Wasserthal, Jakob, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, et al. “TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images.” Radiology: Artificial Intelligence 5, no. 5 (September 1, 2023): e230024. https://doi.org/10.1148/ryai.230024.