Get Started with Lidar Camera Calibrator

The Lidar Camera Calibrator app enables you to interactively estimate the rigid transformation between a lidar sensor and a camera.

This topic shows you the Lidar Camera Calibrator app workflow, as well as features you can use, to analyze and improve your results. The first, and most important, part of the calibration process is to obtain accurate and useful data. For guidelines and tips for capturing data, see Calibration Guidelines.

Load Data

To open the Lidar Camera Calibrator app, at the MATLAB^® command prompt, enter this command.

lidarCameraCalibrator

Alternatively, you can open the app from the Apps tab, under Image Processing and Computer Vision.

The app opens to an empty session. The app reads point cloud data in the PLY and point cloud data (PCD) formats, and images in any format supported by imformats. If your data is stored in a rosbag file, see the Read Lidar and Camera Data from Rosbag File tutorial to convert it accordingly.

Load the calibration data into the app.

1. On the app toolstrip, select Import > Import Data, opening the Import Data dialog box.

2. In the Folder for images box, enter the path to the folder that contains the image files you want to load. Alternatively, select the Browse button next to the box, navigate to the folder containing the images,and click Select Folder.

3. In the Folder for point clouds box, enter the path to the folder that contains the sequence of PCD or PLY files you want to load. Alternatively, select the Browse button next to the box, navigate to the folder containing the files,and click Select Folder.

4. In the Checkerboard Settings section, enter the calibration checkerboard parameters. Specify the size for each checkerboard square in the Square Size box, and select the units of measurement from the list next to the box.

5. In the Padding box, enter the padding values for the checkerboard. For more information on padding, see Checkerboard Padding. Click OK to import your data.

To add more images and point clouds to the session at any point in the session, select Import > Add Data to Session.

Feature Detection

The app loads the image and point cloud data and performs an automatic feature detection pass on the data using the specified checkerboard parameters. It detects the checkerboard corners from the image data and the checkerboard plane from the point cloud data. The app interface displays the detection progression and results.

The Accepted Data pane displays the image and point cloud pairs that the app accepts for calibration. The app accepts an image or point cloud if it detects checkerboard features in both of them. The app uses file names to pair the image and point cloud data. It compares images to the corresponding point clouds with the same file name. The Rejected Data pane displays the data pairs for which the app could not detect features in the image, the point cloud, or both. You can use Select Region of Interest or Select Checkerboard Region on the Rejected Data to obtain better detections.

The app displays the data in the visualization area as separate panes for image and point cloud data. Each pane has tabs with the image or point cloud data file name. You can select a data pair from the Accepted Data or Rejected Data pane to visualize it in this area. When you select a data pair, the app highlights it in blue. To delete the selected data pair, press Backspace (PC) or delete (Mac). For more keyboard shortcuts for the data browser, see Data Browser.

The image display pane shows the image from the selected pair and the detected checkerboard corners. To detect the checkerboard corners, the app uses the estimateCheckerboardCorners3d function. The app computes camera intrinsics to perform feature detection. If you have camera intrinsic values, you can load them into the app, in the Camera Intrinsics section, by selecting Use Fixed Intrinsics. In the dialog box that opens, browse to your camera intrinsics file and load it into the app. After loading, in the Feature Detection section, select Detect to detect features with the new intrinsics.

The point cloud display pane shows the point cloud from the selected pair, with the detected checkerboard plane rendered in white. To detect the plane, the app uses the detectRectangularPlanePoints function.

Use the various display options for point clouds in the app to improve visualization and detection.

Select Region of Interest

Select Snap To ROI to visualize a particular region of interest (ROI) in the point cloud. The app sets a default value for the ROI, but you can set a custom ROI using the Edit ROI tool. This tool enables you to manually pinpoint the region of the point cloud. Specifying an ROI to the region where the checkerboard is present can reduce data rejections and improve performance by focusing feature detection on a specific region.

1. Select Edit ROI, which opens the Edit ROI tab. The tab contains the same Accepted Data and Rejected Data panes as the Calibration tab, but the point cloud display pane takes up the rest of the window.

2. Select a data pair, which the app highlights in blue. You can select a rejected data pair to tune the ROI.

3. Clear the Snap To ROI button to view the whole point cloud.

4. The ROI is highlighted in yellow. Point to the ROI, and the cursor turns into a hand symbol.

5. Click on any side of the ROI and drag it to toggle the size. You can select Snap To ROI to view the contents of the ROI and change the size accordingly.

6. Select Apply to save your changes or Cancel to discard them.

Because the app applies the new ROI on all the point cloud frames, you must define an ROI that covers all areas in which you placed the checkerboard. Clear the Snap To ROI button to view the whole point cloud and select the Hide ROI Cuboid to remove the ROI highlighting. Select Detect to detect features in the selected ROI. Alternatively, you can use keyboard shortcuts to perform these tasks. For more information, see Edit ROI.

Select Checkerboard Region

To further tune the detections, you can use the Select Checkerboard feature to manually select checkerboard points in any point cloud frame.

1. On the app toolstrip, select Select Checkerboard. The app opens the Select Checkerboard tab. This tab contains the same Accepted Data and Rejected Data panes as the Calibration tab, but the point cloud display pane takes up the rest of the window.

2. Select a data pair. The app highlights it in blue. You can select a rejected data pair and find the checkerboard in the point cloud.

3. Use the zoom and rotate options in the axes toolbar of the point cloud display to locate the checkerboard.

4. Select Select Checkerboard. The cursor changes into a crosshair.

5. Click and drag the cursor over the checkerboard. A selection rectangle appears, with the points inside it highlighted in red. Alternatively, you can also select Brush/Select Data on the axes toolbar.

6. After selecting the points, rotate the point cloud to check whether any background points have been selected. If your selection contains unwanted points, select Clear Selection to start over.

7. Select Apply to save the selected points, or Cancel to discard them.

This checkerboard selection applies only to the current point cloud. Select Detect to detect features using the manually selected checkerboard.

Feature Detection Settings

The app provides these feature detection settings in which you can tune parameters.

• Remove Ground — Remove ground points from the point cloud. The app uses the pcfitplane function to estimate the ground plane. The Remove Ground feature is enabled by default. Select Remove Ground to clear it.

• Cluster Threshold — Clustering threshold for two adjacent points in the point cloud, specified in meters. The clustering process is based on the Euclidean distance between adjacent points. If the distance between two adjacent points is less than the clustering threshold, both points belong to the same cluster. Low-resolution lidar sensors require a higher Cluster Threshold, while high-resolution lidar sensors benefit from a lower Cluster Threshold.

• Dimension Tolerance — Tolerance for uncertainty in the rectangular plane dimensions, specified in the range [0,1]. A higher Dimension Tolerance indicates a more tolerant range for the rectangular plane dimensions.

Select Detect to detect features using the new parameters.

Calibration

When you are satisfied with the detection results, select Calibrate button to calibrate the sensors. If you have an estimated transformation matrix, select Initial Transform to load the transformation matrix from a file or the workspace. The app assumes the rotation angle between the lidar sensor and the camera is in the range [–45 45 ], in degrees, along each axis. For a rotation angle outside this range, use Initial Transform to specify an initial transformation to improve calibration accuracy.

After calibration, the app interface displays the image with the checkerboard points from the point cloud projected onto it. The app uses the projectLidarPointsOnImage function to project the lidar points onto the image. The color information of the images is fused with the point cloud data using the fuseCameraToLidar function.

The app also provides the inaccuracy metrics for the transformation matrix using error plots. The plots specify these errors in each data pair:

• Translation Errors — The difference between the centroid coordinates of the checkerboard planes in the point clouds and those in the corresponding images. The app returns the error values in meters.

• Rotation Errors — The difference between the normal angles defined by the checkerboard planes in the point clouds and those in the corresponding images. The app estimates the plane in the image using the checkerboard corner coordinates. The app returns the error values in degrees.

• Reprojection Error— The difference between the projected (transformed) centroid coordinates of the checkerboard planes from the point clouds and those in the corresponding images. The app returns the error values in pixels.

When you select a data pair in the data browser, the corresponding bars in the error plot are highlighted in dark blue. You can tune the calibration results by removing outliers. Drag the red line on each plot vertically to set error limits. The app selects all the data pairs with an error value greater than error limit as outliers, and highlights the error bars and their corresponding data pairs in the data browser in blue. Right-click any of the selected data pairs on the data browser and select Remove and Recalibrate to delete the outliers and recalibrate the sensors. Deleting outliers can improve calibration accuracy. For a list of keyboard shortcuts to use with the error plots, see Error Plots.

Export Results

You can export the transformation matrix and error metrics, as variables, into the workspace or a MAT-file. You can generate a MATLAB script of the complete app workflow to use in your projects.

Keyboard Shortcuts and Mouse Actions

Note

On Macintosh platforms, use the Command (⌘) key instead of Ctrl.

Use keyboard shortcuts and mouse actions to increase productivity while using the Lidar Camera Calibrator app.

Data Browser

Task	Action
Navigate through data pairs in the Accepted Data or Rejected Data pane	Up or down arrow
Select all data pairs in the data browser	Ctrl+A
Select multiple data pairs above or below the currently selected data pair.	Hold Shift and press the up arrow or down arrow
Select multiple data pairs. Note Deselecting a selected data pair is not currently supported using the same shortcut.	Hold Ctrl and click on data pairs
Delete the selected data pair from the data browser.	PC: Backspace or Delete Mac: delete A dialog box appears to deletion.
Select the data pairN above the currently selected data pair. N is the number of data pairs fully displayed in the data browser at the current time.	PC: Page Up Mac: Hold Fn and press the up arrow
Select the data pairN below the currently selected data pair. N is the number of data pairs fully displayed in the data browser at the current time.	PC: Page Down Mac: Hold Fn and press the down arrow
Select the first data pair in the data browser.	PC: Home Mac: Hold Fn and press the left arrow
Select the last data pair in the data browser.	PC: End Mac: Hold Fn and press the right arrow

Error Plots

Use these shortcuts on the error plots to analyze the data. Operations on any of the three error plots affect the corresponding error bars on all three plots.

Task	Action
Select the error bar left of the currently selected error bar.	Left arrow
Select the error bar right of the currently selected error bar.	Right arrow
Select the error bar right or left of the currently selected error bar, in addition to the currently selected error bar.	Hold Shift and press the left arrow or right arrow
Select multiple error bars. Note Deselecting a selected error bar is not currently supported using the same shortcut.	Hold Ctrl and click error bars
Select all error bars.	Ctrl+A
Delete the selected error bar and corresponding data pair from the data browser. The app then recalibrates the sensors.	PC: Backspace or Delete Mac: delete A dialog box appears to confirm deletion.

Edit ROI

Shortcuts to use on the Edit ROI tab.

Task	Action
Undo ROI size change. Note The app stores only the last three sizes of the ROI, so you cannot undo more than three times in a row.	Ctrl+Z
Redo ROI size change.	Ctrl+Y
Clear ROI selection	Esc

Limitations

The Lidar Camera Calibrator app has these limitations:

• The script generated from Export > Generate MATLAB Script does not include any checkerboard regions manually selected using the Select Checkerboard feature. In the script, the checkerboard region is detected in the specified ROI.

• After manually selecting checkerboard regions using the Select Checkerboard feature, when you return to the Calibration tab, you can see the selected points (highlighted in red) only while viewing the whole point cloud ( when SnapToROI is cleared).