Data format conversion

Next, prepare the dataset needed for training.
The data required for training has already been created using Omniverse Replicator, but the output synthetic data needs to be converted into a data format that can be used for training using the TAO Toolkit.
When you unzip the zip file output by Omniverse Replicator, you will see that it contains three types of data: png, npy, and json.
Each file contains the following:

・png: composite RGB image data
・npy: Bounding box data that indicates the location of the asset in the composite image
・json: Class data that indicates the contents of the assets in the composite image


These data are converted into a data format called KITTI format in order to be trained with the TAO Toolkit.
The KITTI format is a data format that outputs annotation information, including class data and bounding box data, as a txt file for each image as shown below.

Now, let's extract the necessary information from the npy and json files and convert them into KITTI format.
A Python script is used for the conversion.

Once the above process is complete, the dataset will be organized as follows:

Let's check how much data has been organized.

It is possible to perform learning in this state, but since the amount of data is somewhat small, we next perform a process called data augmentation.

Data Augmentation

Data augmentation is a technique for augmenting training data by changing the environmental conditions of the training data.
Data augmentation is an effective method when there is not enough training data available, and is also known as Data Augmentation.
This data augmentation is generally performed using Python libraries, etc., but the TAO Toolkit has a function for performing data augmentation, so data augmentation can be easily performed with a single command, just like training.

Data augmentation using the NVIDIA TAO Toolkit is performed based on the contents of the following spec file.

This will rotate the image 90 degrees, rotate the hue from 0 to 20 degrees, blur the image, and change the brightness.
The following command will perform data expansion:

Let's check the image and bounding box after data augmentation.

You can see that the resulting image is similar to the original, but different.
By performing data augmentation in this way, we can increase the amount of data used for training.

The dataset preparation is now complete.

Executing transfer learning

We will perform transfer learning using the TAO Toolkit using the dataset and pre-trained model we have prepared so far.
Transfer learning is performed based on the contents of the following spec file. Learning-related settings such as the number of epochs and batch size can be changed by editing this spec file.

The following command will run transfer learning.

Once training is complete, the trained model is output.

Running inference

Once the training is complete, check the results.

Since the AI model can detect the doll as expected, we will stop training.