Introduction

In order to use Hailo's product, which is an AI processor for edge devices, you need to create a binary file that is optimized for the Hailo product. This article introduces the steps and publishes the Python script.

We will actually run the Python script, convert the AI model (ONNX file) learned with the deep learning development framework to Hailo, and confirm high performance.

Tools used

We are using "Hailo AI Software Suite (2024-01)" provided by Hailo with Docker.

Hailo AI Software Suite contains Hailo's main development tools in one package, making it convenient to create a development environment just by installing it.

It can be downloaded for free by registering in the Developer Zone of Hailo's web site.

Hailo's tools are as follows.

The flow is to create a HEF file (Hailo Executable binary File) from the ONNX file in the Model Build Environment on the left, copy the HEF file to the inference machine (Runtime Environment), and execute inference using the HailoRT API. The script released this time is exactly what creates a HEF file from an ONNX file using the Model Build Environment.

HEF file generation flow from ONNX file

Enter Docker in Hailo AI Software Suite and create an appropriate folder. Save the following files there.

・Target ONNX file (here yolox_s_leaky.onnx)

・Python script (save the source code described below as dfc_script.py)

Then just run the Python command.

$ python ./dfc_script.py

Below is the contents of the script file. It can be used with other models by modifying the "#Define model information" part according to the ONNX model.

*This is for reference only, and operation is not guaranteed.

import numpy as np
import os
from hailo_sdk_client import ClientRunner
#Define model information
model_name = 'yolox_s_leaky'
onnx_path = 'yolox_s_leaky.onnx'
start_node = 'images'
end_node = ['Conv_197','Sigmoid_199','Sigmoid_200','Conv_213','Sigmoid_215','Sigmoid_216','Conv_229','Sigmoid_231','Sigmoid_232']
input_shape = {'images': [1, 3, 640, 640]}
chosen_hw_arch = 'hailo8'
input_height = 640
input_width = 640
input_ch = 3

alls_lines = [
    'model_optimization_flavor(optimization_level=0, compression_level=1)
',
    'resources_param(max_control_utilization=1.0, max_compute_utilization=1.0,max_memory_utilization=1.0)
',
    'performance_param(fps=250)
'
]
#Parsing
runner = ClientRunner(hw_arch=chosen_hw_arch)
hn, npz = runner.translate_onnx_model(onnx_path, model_name, start_node_names=[start_node], end_node_names=end_node, net_input_shapes=input_shape)
parsed_model_har_path = f'{model_name}_parsed_model.har'
runner.save_har(parsed_model_har_path)
#Optimize
calibData = np.random.randint(0, 255, (1024, input_height, input_width, input_ch))
runner.load_model_script(''.join(alls_lines))
runner.optimize(calibData)
quantized_model_har_path = f'{model_name}_quantized_model.har'
runner.save_har(quantized_model_har_path)
#Compile
hef = runner.compile()
file_name = f'{model_name}.hef'
with open(file_name, 'wb') as f:
    f.write(hef)
compiled_model_har_path = f'{model_name}_compiled_model.har'
runner.save_har(compiled_model_har_path)

*The input notation for alls_lines is incorrect, but it is written as follows.
alls_lines = [
'model_optimization_flavor(optimization_level=0, compression_level=1)\n',
'resources_param(max_control_utilization=1.0, max_compute_utilization=1.0,max_memory_utilization=1.0)\n',
'performance_param(fps=250)\n'
]

*Please download and use the source code from the button below.

Script file explanation

#Define model information

Please enter the information of the AI model to be processed by Hailo here.

model_name: Any name

onnx_path: Specify the path of the onnx file

This time I am using yolox_s_leaky.onnx published on Hailo Model Zoo (link)

start_node, end_node: Specify the first and last nodes of the part to be processed by Hailo

This time, the following parts are specified. end_node specifies multiple nodes

input_shape: The shape of the input node

chosen_hw_arch: Target Hailo device (hailo8, hailo8l, hailo15h, etc.)

input_height, input_width, input_ch: Input resolution and number of channels for calibration data

alls_lines: Various settings during compilation (please refer to the Dataflow Compiler manual for details)

Results and validation

After running the Python script, several HAR files (Hailo ARchive files) will be created along with the HEF file. Since the state of each process (Parsing, Optimize, Compile) is saved, you can return to the previous process at any time.

You can also use this archive file to view the resulting report file with the command below.

$ hailo profiler yolox_s_leaky_compiled_model.har

An html file will be created as a report file, and you can check the compilation results as shown below.

As shown above, you can expect a performance of 250fps with this model.

You can get various information such as the shape of the graph after compilation and the bottleneck layer from this report file, so please check it out.

Finally, let's check how well this hef file performs on actual hardware.

You can actually perform input/output on the hardware with the following command and measure the performance.

(Of course, this is not possible if a Hailo device is not implemented)

$ hailortcli benchmark yolox_s_leaky.hef

In this case, the results were as follows.

The summary is listed in the red frame above, and you can see that 250fps can be achieved with actual hardware.

Summary

It may be difficult to use an unknown device, but with Hailo, it is not that difficult to create a binary file as introduced here. We hope you will take advantage of this script and experience Hailo's high performance.

Inquiry

If you have any questions regarding this article, please contact us below.