Deep Learning with Point Clouds

Lidar sensors record point cloud data that provides rich 3-D geometric information of their surroundings. You can process this data to get a better understanding of an environment, and use it for various applications in driving, robotics, medicine, forestry, construction, urban planning, and oceanography.

Point cloud data is highly unordered and sparse as it stores points in a 3-D space without any discretion. Additionally, factors such as sensor range, occlusions, and uneven sampling of points also affect the nature of point cloud data. These factors make point cloud processing a challenging task.

Deep learning addresses various challenges in processing point cloud data. It is easier to perform complex point cloud processing tasks such as segmentation, detection, and tracking, by training deep learning networks.

• Segmentation clusters points in a point cloud and assigns class labels such as car, tree, and building to those clusters.

• Detection identifies and locates objects in a 3-D point cloud scene.

• Tracking tracks the state of objects across different point cloud frames.

These are the common steps for any point cloud processing workflow using deep learning.

Import Data

Lidar Toolbox™ provides functions to import and read raw point cloud data from several file formats. For more information, see I/O.

The toolbox enables you to divide this data into training and test data sets, and store them as datastore objects. For example, you can store point cloud files by using the fileDatastore object. For more information on datastore objects, see Datastores for Deep Learning (Deep Learning Toolbox).

The Import Point Cloud Data For Deep Learning example shows you how to import a large point cloud data set, and then configure and load a datastore.

Augment and Preprocess Data

In this step you prepare training data by labeling, augmenting, and preprocessing it.

Label Data

You need a large, labeled data set to train a deep learning network. If you have an unlabeled data set, you can use the Lidar Labeler app to label your training data. For information on how to use the app, see Get Started with the Lidar Labeler.

Preprocess Data

You can preprocess your data before training the network. Lidar Toolbox provides function to perform various preprocessing tasks.

• Denoise, downsample, and filter point clouds.

• Convert unorganized data into the organized format.

• Divide aerial point cloud data into blocks to perform block-by-block processing.

For more information on preprocessing, see Lidar Processing Applications (Deep Learning Toolbox).

To interactively visualize, analyze, and preprocess point cloud data, use the Lidar Viewer app. For more information on how to use the app, see Get Started with Lidar Viewer.

Augment Data

Data augmentation adds variety to the existing training data. The robustness of a network to data transformations increases when you train it on a data set with a lot of variety.

Augmentation techniques reduce overfitting problems and enable the network to better learn and infer features.

For more information on how to perform data augmentation on point clouds, see Data Augmentations for Lidar Object Detection Using Deep Learning.

Create Network

Every network has a unique architecture specific to the task you design it for. You define the network architecture based on the network input and the layers. Most deep learning networks either encode the point cloud into an image-like format, voxelize the point cloud, or operate directly on individual points.

Lidar Toolbox provides functions to create deep learning networks, as well as functions to use them for specific workflows.

• For segmentation workflows, see Segmentation.

• For object detection workflows, see Detection and Tracking.

To programmatically create a custom network layer-by-layer, use the functions specified in List of Deep Learning Layers (Deep Learning Toolbox). You can also interactively create the network by using the Deep Network Designer (Deep Learning Toolbox) app.

To visualize the network structure, use the analyzeNetwork (Deep Learning Toolbox) function.

Train Network

Specify training options by using the trainingOptions (Deep Learning Toolbox) function, and train the network by using the trainnet (Deep Learning Toolbox) function.

Test and Evaluate Results

Run the trained network on your test data set and evaluate the performance of the network. Depending on the task, you must use different functions to evaluate the results.

• To evaluate segmentation results, use the evaluateSemanticSegmentation function.

• To evaluate object detection results, use the evaluateObjectDetection and bboxOverlapRatio functions.

Note

This functionality requires Deep Learning Toolbox™ licence.