estimateLidarCameraTransform

Estimate rigid transformation from lidar sensor to camera

Since R2020b

Syntax

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners)

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners,intrinsics)

[tform,errors] = estimateLidarCameraTransform(___)

[___] = estimateLidarCameraTransform(___,Name,Value)

Description

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners) estimates the transformation between a lidar sensor and a camera using the checkerboard planes extracted from lidar sensor data and 3-D image corners of the checkerboard extracted from camera data, respectively.

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners,intrinsics) uses the checkerboard planes extracted from a lidar sensor, 2-D or 3-D image corners of the checkerboard extracted from a camera, and the camera intrinsic parameters to estimate the transformation between the lidar sensor and the camera.

[tform,errors] = estimateLidarCameraTransform(___) returns the inaccuracy in estimating the transformation matrix errors using any combination of input arguments in previous syntaxes.

[___] = estimateLidarCameraTransform(___,Name,Value) specifies options using one or more name-value arguments in addition to any combination of arguments in previous syntaxes. For example, 'Verbose',true sets the function to display progress.

Examples

Estimate Rigid Transform from Lidar to Camera

Estimate the rigid transformation from a lidar sensor to a camera using data captured from the lidar sensor and camera calibration parameters. Use these three steps:

1. Load the data into the workspace.

2. Extract the required features from images and point cloud data.

3. Estimate the rigid transformation using the extracted features.

Load Data

Load images and point cloud data into the workspace.

imageDataPath = fullfile(toolboxdir('lidar'),'lidardata',...
    'lcc','vlp16','images');
imds = imageDatastore(imageDataPath);
imageFileNames = imds.Files;
ptCloudFilePath = fullfile(toolboxdir('lidar'),'lidardata',...
'lcc','vlp16','pointCloud');
pcds = fileDatastore(ptCloudFilePath,'ReadFcn',@pcread);
pcFileNames = pcds.Files;

Load camera calibration files into the workspace.

cameraIntrinsicFile = fullfile(imageDataPath,'calibration.mat');
intrinsic = load(cameraIntrinsicFile);

Feature Extraction

Specify the size of the checkerboard squares in millimeters.

squareSize = 81;

Estimate the checkerboard corner coordinates for the images.

[imageCorners3d,planeDimension,imagesUsed] = estimateCheckerboardCorners3d( ...
    imageFileNames,intrinsic.cameraParams,squareSize);

Filter the point clouds based on the images used.

pcFileNames = pcFileNames(imagesUsed);

Detect the checkerboard planes in the filtered point clouds using the plane parameters planeDimension.

[lidarCheckerboardPlanes,framesUsed] = detectRectangularPlanePoints( ...
pcFileNames,planeDimension,'RemoveGround',true);

Extract the images, checkerboard corners, and point clouds in which you detected features.

imagFileNames = imageFileNames(imagesUsed);
imageFileNames = imageFileNames(framesUsed);
pcFileNames = pcFileNames(framesUsed);
imageCorners3d = imageCorners3d(:,:,framesUsed);

Estimate Transformation

Estimate the transformation using checkerboard planes from the point clouds and 3-D checkerboard corner points from the images.

[tform,errors] = estimateLidarCameraTransform(lidarCheckerboardPlanes, ...
imageCorners3d,intrinsic.cameraParams);

Display translation, rotation, and reprojection errors as bar graphs.

figure
bar(errors.TranslationError)
xlabel('Frame Number')
title('Translation Error (meters)')

figure
bar(errors.RotationError)
xlabel('Frame Number')
title('Rotation Error (degrees)')

figure
bar(errors.ReprojectionError)
xlabel('Frame Number')
title('Reprojection Error (pixels)')

Input Arguments

ptCloudPlanes — Segmented checkerboard planes

P-by-1 array of pointCloud objects

Segmented checkerboard planes, specified as a pointCloud object or P-by-1 array of pointCloud objects. P is the number of point clouds. Each pointCloud object must contain points that represent a checkerboard (rectangular) plane.

P must be equal for both the ptCloudPlanes and imageCorners arguments. This means that number of point clouds and number of images used for detection must also be equal.

imageCorners — Checkerboard corners extracted from camera data

4-by-2-by-P array | 4-by-3-by-P array

Checkerboard corners extracted from camera data, specified in 2-D or 3-D coordinates.

• 2-D coordinates, specified as a 4-by-2-by-P array. Each row of a channel is of the form of [x y], of a checkerboard corner extracted from the corresponding camera image. The values are in pixel coordinates

• 3-D coordinates, specified as a 4-by-3-by-P array. Each row of a channel is of the form of [x y z], of a checkerboard corner extracted from the corresponding camera image. The values are in world coordinate system.

P represents the number of camera images used for detection. P must be equal for both the ptCloudPlanes and imageCorners arguments. This means that number of point clouds and number of images used for detection must also be equal.

Note

When imageCorners value is in 2-D coordinates, you must specify the camera intrinsic parameters, intrinsics.

Data Types: single | double

intrinsics — Camera intrinsic parameters

cameraIntrinsics object

Camera intrinsics parameters, specified as a cameraIntrinsics object.

Note

When imageCorners value is in 2-D coordinates, you must specify the camera intrinsic parameter, intrinsics. When the imageCorners are in 3-D coordinates, intrinsics is an optional input.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: Verbose=true sets the function to display progress.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'Verbose',true sets the function to display progress.

Lidar3DCorners — Checkerboard corners in lidar frame

4-by-3-by-P array

Checkerboard corners in the lidar frame, specified as a 4-by-3-by-P array where P is the number of point clouds.

If the user specifies the checkerboard corners in the lidar frame, then the function does not calculate them internally.

Data Types: single | double

InitialTransform — Initial rigid transformation

identity transformation as a rigidtform3d object (default) | rigidtform3d object

Initial rigid transformation, specified as a rigidtform3d object.

The function assumes the rotation angle between the lidar sensor and the camera is in the range [-45 45] along each axis. For any other range of the rotation angle, use this name-value pair to specify an initial transformation to improve function accuracy.

Verbose — Display function progress

false or 0 (default) | true or 1

Display function progress, specified as a logical 0 (false) or logical 1 (true).

Data Types: logical

Output Arguments

tform — Lidar to camera rigid transformation

rigidtform3d object

Lidar to camera rigid transformation, returned as a rigidtform3d object. The returned object registers the point cloud data from a lidar sensor to the coordinate frame of a camera.

errors — Inaccuracy in the transformation matrix estimation

structure | P-element numeric array

Inaccuracy of the transformation matrix estimation, returned as a structure or P-element numeric array.

• The function returns a structure when image corners are in 3-D coordinates. The structure contains these fields.

  • RotationError — The difference between the normal angles defined by the checkerboard planes in the point clouds (lidar frame) and those in the images (camera frame). The function estimates the plane in the image using the checkerboard corner coordinates. The function returns the error values in degrees, as a P-element numeric array. P is the number of point clouds.

  • TranslationError — The difference between the centroid coordinates of checkerboard planes in the point clouds and those in the images. The function returns the error values in meters, as a P-element numeric array. P is the number of point clouds.

  If you specify camera intrinsic parameters to the function using intrinsics argument, then the structure contains this additional field.

  • ReprojectionError — The difference between the projected (transformed) centroid coordinates of the checkerboard planes from the point clouds and those in the images. The function returns the error values in pixels, as a P-element numeric array. P is the number of point clouds.

• For 2-D image corners, the function only returns the reprojection error.

Version History

Introduced in R2020b

R2022b: Supports rigidtform3d objects

You can now specify the InitialTransform name-value argument as a rigidtform3d object, which uses the premultiply convention. Although you can still specify InitialTransform as a rigid3d object, this object is not recommended because it uses the postmultiply convention. For more information, see Migrate Geometric Transformations to Premultiply Convention.

When you specify InitialTransform, the estimateLidarCameraTransform function returns tform as an object of the same type. When you do not specify InitialTransform, the estimateLidarCameraTransform function now returns tform as a rigidtform3d object. Before, the function returned tform as a rigid3d object.

R2022a: Support for 2-D image corners input

You can specify the imageCorners input in 2-D coordinates. You must additionally specify the camera intrinsic parameters by using the instrinsics input to estimate the transformation matrix when image corners are 2-D.