How To Command Simulated Isaac Robot

This tutorial requires a machine with Isaac Sim 2023.1.x (recommended) or Isaac Sim 2022.2.x installed. For Isaac Sim requirements and installation please see the Omniverse documentation. To configure Isaac Sim to work with ROS 2 please see this guide.

This tutorial has the following assumptions on system configuration:

  1. NVIDIA Isaac Sim is installed in the default location. Docker based installations of Isaac sim are also supported but it is up to the user to configure the system.

  2. Docker is installed. If you plan to use your GPU with MoveIt, you will need to install nvidia-docker.

  3. You clone this repo so that you can build a Ubuntu 22.04 Humble based Docker image that can communicate with Isaac and run this tutorial.

Introduction to ros2_control

One of the recommended ways to execute trajectories calculated by MoveIt is to use the ros2_control framework to manage and communicate with your robot, real or simulated. It comes highly recommended because it offers a developers a common API that allows your software to switch between many different robot types, and the sensors they have built in, by simply changing some launch arguments. For example if we inspect the Panda Robot’s ros2_control.xacro we can see it uses a flag use_fake_hardware to switch between being simulated or connecting to a physical robot.

<hardware>
  <xacro:if value="${use_fake_hardware}">
    <plugin>mock_components/GenericSystem</plugin>
  </xacro:if>
  <xacro:unless value="${use_fake_hardware}">
    <plugin>franka_hardware/FrankaHardwareInterface</plugin>
    <param name="robot_ip">${robot_ip}</param>
  </xacro:unless>
</hardware>

Hardware Components can be of different types, but the plugin <plugin>mock_components/GenericSystem</plugin> is very a simple System that forwards the incoming command_interface values to the tracked state_interface of the joints (i.e., perfect control of the simulated joints).

For us to expand our Panda robot to Isaac Sim we first have to introduce topic_based_ros2_control. This Hardware Interface is a System that subscribes and publishes on configured topics. For this tutorial the topic /isaac_joint_states will contain the robot’s current state and /isaac_joint_commands will be used to actuate it. The moveit_resources_panda_moveit_config we are using in this tutorial does not support connecting to hardware, so our ros2_control.xacro is now updated to load the TopicBasedSystem plugin when the flag ros2_control_hardware_type is set to isaac.

<xacro:if value="${ros2_control_hardware_type == 'mock_components'}">
    <plugin>mock_components/GenericSystem</plugin>
</xacro:if>
<xacro:if value="${ros2_control_hardware_type == 'isaac'}">
    <plugin>topic_based_ros2_control/TopicBasedSystem</plugin>
    <param name="joint_commands_topic">/isaac_joint_commands</param>
    <param name="joint_states_topic">/isaac_joint_states</param>
</xacro:if>

In this tutorial we have included a Python script that loads a Panda robot and builds an OmniGraph to publish and subscribe to the ROS topics used to control the robot. The OmniGraph also contains nodes to publish RGB and Depth images from the camera mounted on the hand of the Panda. The RGB image is published to the topic /rgb, the camera info to /camera_info, and the depth image to /depth. The frame ID of the camera frame is /sim_camera. To learn about configuring your Isaac Sim robot to communicate with ROS 2 please see the Joint Control tutorial on Omniverse.

Computer Setup

  1. Install Isaac Sim.

  2. Perform a shallow clone of the MoveIt 2 Tutorials repo.

git clone https://github.com/moveit/moveit2_tutorials.git -b main

  1. Go to the folder in which you cloned the tutorials and then switch to the following directory.

cd moveit2_tutorials/doc/how_to_guides/isaac_panda

  1. Build the Docker image. This docker image also contains pytorch.

docker compose build base

Running the MoveIt Interactive Marker Demo with Mock Components

This section tests out the mock_components/GenericSystem hardware interface, as opposed to using Isaac Sim.

  1. To test out the mock_components/GenericSystem hardware interface run:

docker compose up demo_mock_components

This will open up RViz with the Panda robot using mock_components to simulate the robot and execute trajectories.

Please see the Quickstart in RViz tutorial if this is your first time using MoveIt with RViz.

After you are done testing press Ctrl+C in the terminal to stop the container.

Running the MoveIt Interactive Marker Demo with Isaac Sim

  1. On the host computer, go to the tutorials launch directory.

cd moveit2_tutorials/doc/how_to_guides/isaac_panda/launch

  1. Then run the following command to load the Panda Robot pre-configured to work with this tutorial.

Note

This step assumes that a compatible version of Isaac Sim is installed on the host in the $HOME/.local/share/ov/pkg/" directory. This step also takes a few minutes to download the assets and setup Isaac Sim so please be patient and don’t click the Force Quit dialog that pops up while the simulator starts.

./python.sh isaac_moveit.py

  1. From the moveit2_tutorials/doc/how_to_guides/isaac_panda directory start a container that connects to Isaac Sim using the topic_based_ros2_control/TopicBasedSystem hardware interface.

docker compose up demo_isaac

This will open up RViz with the Panda robot using the TopicBasedSystem interface to communicate with the simulated robot and execute trajectories.