Overview of HailoRT

HailoRT is a tool for loading the hef file created by DataflowCompiler onto an actual Hailo device and running inference.

*Please refer to the link below for instructions on how to create a hef file.

　　①Hardware and installation requirements for AI Software Suite

　　②How to use Dataflow Compiler (how to create hef files)

HailoRT is part of the AI Software Suite, but ultimately we think it will primarily be installed on embedded devices and used to perform inference.

Therefore, unlike the AI Software Suite, it is compatible with a wide range of platforms.

　Host CPU：x86、ARM (aarch64)

OS: Linux (Yocto embedding recipe also available)/Windows

HailoRT itself is open software available on Github.

　https://github.com/hailo-ai/hailort

In this article, we will explain how to use HailoRT in the AI Software Suite and provide sample source code.

HailoRT CLI Tool

HailoRT comes with a CLI (Command Line Interface) tool, so you can easily check the operation with commands. If you just want to check the operation of the hef file you created, this is probably the quickest way to do it.

As a test, let's try running the hef file created in the article linked below.

　　②How to use Dataflow Compiler (how to create hef files)

The Dataflow Compiler tutorial folder is below, so first move there.

$ cd /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_tutorials/notebooks

You can check the performance (FPS) by specifying the hef file created with the hailortcli run command.

$ hailortcli run resnet_v1_18.hef

By adding options, you can also check latency and power consumption.

$ hailortcli run resnet_v1_18.hef --measure-latency
$ hailortcli run resnet_v1_18.hef --measure-power

The results for each are shown below.

Python

The tutorial explains how to access the hardware through HailoRT in Python, as well as how to create a hef file.

Since you probably use Python a lot during training, you may find it useful to check accuracy.

$ hailo tutorial

For the HailoRT tutorial, select the notebook “DFC_4_Inference_Tutorial.ipynb”.

This is the preparation step before inputting images into the AI model. Basically, you can use this part as is.

For information about the functions used here, please refer to the HailoRT User Guide.

Since this is a sample image, the input data is random.

This cell actually inputs image data and receives the output.

This is a blocking process, so I think it will be useful for verifying results, such as checking accuracy.

This notebook also contains information on how to process streaming, so please refer to it.

C/C++

When it is finally implemented in an embedded device, C/C++ will likely be used. There are also various ways to use it, such as switching between multiple AI models or using multiple Hailo devices.

Sample files for various purposes are available for C/C++, so please select one according to your needs.

The sample files are available on Github. There is also a README, so please use it as a reference for your selection.

https://github.com/hailo-ai/hailort/tree/master/hailort/libhailort/examples

These sample files are merely source code for the input and output parts of the Hailo device.

When actually building an application, there are image acquisition, pre-processing, inference, post-processing, etc. Hailo has prepared several application-level sample codes, so please refer to them as well.

　https://github.com/hailo-ai/Hailo-Application-Code-Examples/tree/main/runtime/cpp

bonus

The AI Software Suite also includes an application demonstration tool called Tappas.

It is mainly based on GStreamer and can be customized.

Finally, let's try moving this.

$ hailo_run_app --help

You can check how to use it and what demos are included here.

Let's try running yolov8.

$ hailo_run_app detection detection --show-fps

You can easily run the demo with just commands, so please give it a try.

Tapas itself is also available on Github, so please check that out as well.

　https://github.com/hailo-ai/tappas

Summary

This concludes the explanation of Hailo's development flow.

I hope this helps with the development of Hailo.

If you have any questions, please contact us using the "Contact Us" button below.

Inquiry

If you have any questions about this article, please contact us using the form below.