Main Content

Control NVIDIA Carter Robot in Isaac Sim Using ROS 2

This example introduces how to interact with the NVIDIA Isaac Sim® [1] application from MATLAB® through a ROS 2 interface. NVIDIA Isaac Sim, powered by Omniverse® [2], is a scalable robotics simulation application and synthetic data generation tool that powers photorealistic, physically accurate virtual environments. You can use NVIDIA Isaac Sim to develop, test, and manage AI-based robots. NVIDIA Omniverse is a computing platform that enables individuals and teams to develop universal-scene-description-based (USD-based) [3] 3-D workflows and applications.

Overview

In this example, you publish ROS 2 messages from MATLAB to the Carter robot. You also subscribe to the topics that the Carter robot publishes in MATLAB, such as odometry, image, and camera data. This example was tested using Isaac Sim version 2022.2.1 on a Windows® platform.

Configure Workstation

  1. Ensure your local workstation meets the System Requirements and Isaac Sim Driver Requirements [4] for running Omniverse Isaac Sim.

  2. Visit Omniverse Downloads to download the standard version of Omniverse Launcher, which is the native client for downloading, installing, and updating Omniverse applications, extensions, and connectors.

  3. Install Omniverse Launcher [5].

  4. Install Isaac Sim and Omniverse Cache [6] from the Exchange tab in the Omniverse Launcher.

  5. Install Nucleus [7] from the Omniverse Launcher to access Isaac environments, assets, and samples.

Install and Activate ROS 2 Environment

  1. Install the ROS 2 Foxy distribution on your Windows workstation by following the ROS 2 Windows binary installation instructions from ROS 2 Documentation: Foxy [8].

  2. From the start menu, look for developer command prompt: run the x64 Native Tools Command Prompt for VS 2019 shortcut as an administrator.

  3. Activate the ROS 2 environment by sourcing the ROS 2 installation. For example, run this command in the command prompt: call C:\opt\ros\foxy\x64\setup.bat in the command prompt.

Launch Isaac Sim

Launch NVIDIA Isaac Sim from the same command prompt in which you activate the ROS 2 installation environment. Navigate to your Isaac Sim installation location using cd <path_to_isaac_sim_install_path>. If you do not know where the Isaac Sim application is installed, follow these steps to locate the installation path.

1. Go to the Library tab in the Omniverse Launcher. Select Isaac Sim in the Apps pane on the left.

2. Open the Isaac Sim Settings panel by clicking Settings.

3. Click the folder icon in the settings panel to open the Isaac Sim installation location in the explorer.

4. Copy the Isaac Sim location and navigate to that folder in the same command prompt in which you activate the ROS 2 environment.

5. Set the FASTRTPS_DEFAULT_PROFILES_FILE environment variable to <path_to_isaac_sim_install_path>\ros2_workspace\fastdds.xml before launching Isaac Sim. For example, run this command in the command prompt:set FASTRTPS_DEFAULT_PROFILES_FILE=C:\Users\Paul\AppData\Local\ov\pkg\isaac_sim-2022.2.1\ros2_workspace\fastdds.xml.

6. Run isaac-sim.bat to launch Isaac Sim. The first run of the Isaac Sim application takes up to five minutes to warm up the shader cache.

Enable ROS 2 Bridge Extension

To enable the ROS 2 bridge (omni.isaac.ros2_bridge) extension, go to the Extensions menu under the Window tab. Then, search for ROS2 Bridge and enable it. This extension connects Isaac Sim to ROS 2.

IsaacSimROS2Bridge.gif

Set Up Scene in Isaac Sim

To load the warehouse scene in Isaac Sim, follow these steps:

1. Navigate to this location under Content tab in Isaac Sim : omniverse://localhost/NVIDIA/Assets/Isaac/2022.2.1/Isaac/Samples/ROS2/Scenario.

2. Drag the carter_warehouse_navigation file with.usd extension to the Viewport in Isaac Sim. This action loads a warehouse scenario containing a Carter robot that uses its RTX lidar and a camera to perceive the world. Loading the scene for the first time takes several minutes.

3. To begin the simulation, click Play.

IsaacSimWareHouseSceneROS2.gif

Interact with Carter Robot from MATLAB

To get started, create a node in MATLAB. This action starts the ROS 2 network automatically.

node = ros2node("carterNode");

Query available topics in the ROS 2 network. Ensure that the warehouse scene is playing in NVIDIA Isaac Sim to see the topics that the Carter robot publishes.

ros2 topic list
/camera_info_left
/clock
/cmd_vel
/odom
/parameter_events
/rgb_left
/rosout
/scan
/tf

You can control the movement of the Carter robot by publishing a message to the /cmd_vel topic. The message must have the type geometry_msgs/Twist, which contains data specifying the linear and angular velocities. The linear velocity along the x-axis controls forward and backward motion. The angular velocity around the z-axis controls the rotation speed of the robot base.

Specify the velocity in m/s.

velocity = 0.1;

Create a ROS 2 publisher for the /cmd_vel topic and the corresponding message containing the velocity values.

robotCmd = ros2publisher(node,"/cmd_vel");
velMsg = ros2message(robotCmd);

Set the forward velocity of the robot using the velocity variable and publish the command to the robot. Let the robot move for a moment, and then bring it to a stop.

velMsg.angular.z = 0;
velMsg.linear.x = velocity;
send(robotCmd,velMsg)
pause(4)
velMsg.linear.x = 0;
send(robotCmd,velMsg)

Receive Robot Position and Orientation

The Carter robot uses the /odom topic to publish its current position and orientation (collectively referred to as pose). Because the Carter robot is not equipped with a global positioning system (GPS) system, the pose is relative to the initial pose of the robot.

Create a ROS 2 subscriber for the odometry messages

odomSub = ros2subscriber(node,"/odom");

Wait for the subscriber to return data, then extract the data and assign it to variables x, y, and z. If the software returns an error, then the receive command likely timed out. Ensure that the robot is publishing odometry and that your network is configured properly.

odomMsg = receive(odomSub,3);
pose = odomMsg.pose.pose;
x = pose.position.x;
y = pose.position.y;
z = pose.position.z;

Display the position of the robot.

[x y z]
ans = 1×3

    0.3959   -0.0000    0.0016

The pose object stores the orientation of the Carter robot as a quaternion in the Orientation property. Use quat2eul (Robotics System Toolbox) to convert the orientation to the more convenient representation of Euler angles. Display the current orientation, theta, of the robot in degrees.

quat = pose.orientation;
angles = quat2eul([quat.w quat.x quat.y quat.z]);
theta = rad2deg(angles(1))
theta = -0.0037

Receive Camera Data

Subscribe to the /rgb_left topic, on which the Carter robot publishes the camera RGB images.

cameraSub = ros2subscriber(node,"/rgb_left");

Wait for the image data, convert the received ROS 2 message into image matrix using rosReadImage, and then display the result by using imshow.

cameraMsg = receive(cameraSub);
I = rosReadImage(cameraMsg);
imshow(I)

Display continuously updating camera data while the Carter robot turns for a short duration.

velMsg.angular.z = velocity;
send(robotCmd,velMsg);
tic;
while toc < 20
  imgMsg = receive(cameraSub);
  I = rosReadImage(imgMsg);
  imshow(I)
  drawnow
end

Set the angular velocity of the robot to 0 and publish the command to the robot, for it to stop rotating.

velMsg.angular.z = 0;
send(robotCmd,velMsg);

Receive Lidar Data

Subscribe to the /scan topic on which the Carter robot publishes lidar data.

lidarSub = ros2subscriber(node,"/scan");

Wait for the data and then display it with rosPlot.

scanMsg = receive(lidarSub);
figure
rosPlot(scanMsg)

Continuously display updating lidar scans while the Carter robot turns for a short duration.

velMsg.angular.z = velocity;
send(robotCmd,velMsg)
tic
while toc < 20
  scanMsg = receive(lidarSub);
  rosPlot(scanMsg)
end

Set the angular velocity of the robot to 0 and publish the command to the robot, for it to stop rotating.

velMsg.angular.z = 0;
send(robotCmd,velMsg)

References

[1] NVIDIA Developer. “Isaac Sim,” December 11, 2019. https://developer.nvidia.com/isaac-sim.

[2] “NVIDIA Omniverse Documentation.” Accessed July 27, 2023. https://docs.omniverse.nvidia.com/.

[3] NVIDIA. “Universal Scene Description (USD) at NVIDIA.” Accessed July 27, 2023. https://www.nvidia.com/en-us/omniverse/usd/.

[4] “1. Isaac Sim Requirements — Isaacsim Latest Documentation.” Accessed July 27, 2023. https://docs.omniverse.nvidia.com/isaacsim/latest/requirements.html.

[5] “Installing Launcher — Launcher Latest Documentation.” Accessed July 27, 2023. https://docs.omniverse.nvidia.com/launcher/latest/installing_launcher.html.

[6] “Workstation — Utilities Latest Documentation.” Accessed July 27, 2023. https://docs.omniverse.nvidia.com/utilities/latest/cache/installation/workstation.html.

[7] “Installation — Nucleus Latest Documentation.” Accessed July 27, 2023. https://docs.omniverse.nvidia.com/nucleus/latest/workstation/installation.html.

[8] “Windows (Binary) — ROS 2 Documentation: Foxy Documentation.” Accessed July 27, 2023. https://docs.ros.org/en/foxy/Installation/Windows-Install-Binary.html.