Creating a dataset

First, build a virtual space with Omniverse

The flow of building a 3D virtual space using Omniverse is as follows.

・Build a facility building using 3D assets such as floors, walls, and ceilings

-Added ceiling lighting

・Add assets such as shelves and furniture to the facility

・Add detailed assets such as floor lines and casters

・Added cardboard etc. for shelves

　

The virtual space of Macnica distribution warehouse is now complete. Using Omniverse's rendering technology, we were able to recreate a photorealistic virtual space even with multiple lights.

Creating a dataset

The virtual space created with Omniverse can be used like a "studio" to capture dataset images necessary for AI model generation.

・Using Omniverse Replicator's randomization function, it is possible to automatically generate a wide variety of datasets from a single shooting scene.

- In addition to the position, size, and color parameters of objects that define the shooting scene, it is possible to randomly change various parameters such as camera angle and light source conditions.

・Omniverse Replicator can automatically generate dataset images paired with training data for images taken in this way.

　

This time, we were able to automatically generate approximately 10,000 dataset images in a few hours using the GPU environment recommended by NVIDIA.

AI model training

NVIDIA TAO Toolkit is a framework for generating AI models, and includes a series of steps including input processing of AI models, specification of datasets, examples of learning parameter descriptions, and conversion processing for using the generated AI models on NVIDIA GPUs. A sample script containing the process is provided.

This time, we built an AI model based on a sample case of image segmentation using UNET. The created model was intended to be implemented in the Isaac ROS image segmentation sample program, and the people detection model (PeopleSemSegNet) available in Isaac ROS was used as the input model.

By performing transfer learning on the AI model that detects people using the dataset generated by Omniverse Replicator, we were able to generate an AI model that detects the floor.

Implementation of AI model (robot simulation)

We placed a two-wheeled robot in the virtual space of Macnica distribution warehouse that we had created, and created an environment in which its movement could be controlled from a program created using ROS2.

- Added the Isaac ROS image segmentation sample package to the ROS2 program execution environment.

・Replace the AI model to be used with the AI model that detects the floor created earlier and run the robot simulation.

As shown in the figure below, a robot is placed in the created three-dimensional virtual environment, and the image from the stereo camera (StereoCamera: Right) mounted on the robot is taken out.

As a result of running Isaac ROS image segmentation AI processing, we were able to see an image segmentation image in which the floor area was filled in red.

[Supplement] The execution example in this article uses Isaac Sim (2022.2.1), Isaac ROS (DP3.1), and TAO Toolkit (5.1.0).

In Episode 2, we explain how to create a dataset.

In this article, we introduced the sequence of data set generation, AI model learning, and AI model implementation (robot simulation). What did you think?

There are many published examples of downloading AI model files and running robot simulations in the ROS2 simulation environment.

However, when customizing and running an original model, many people find the steps of generating various variations of dataset images and then ``combining them with training data'' a hurdle.

By utilizing the NVIDIA Omniverse platform, it is possible to automatically generate dataset images in a photorealistic virtual space without having to take images at the actual site.

In addition, the AI development process from transfer learning with TAO Toolkit to robot simulation with Isaac ROS can be tried many times in a short period of time, and additional datasets can be generated to customize the desired AI model and the learning period can be shortened. We expect it to improve significantly. We hope that you will try using this article as a reference.

In the next episode, Episode 2, we will dig deeper into creating a dataset and explain the execution steps in detail.

If you are considering robot simulation in virtual space, please contact us.

Macnica provides NVIDIA software solutions, mainly Omniverse, and has a strong track record of supporting robotics-related companies. If you are considering introducing AI and need help, please feel free to contact us.