Running VLM on an edge device - Qwen2.5-VL-7B-Instruct

Narrow down by specifying conditions

現在2201件がヒットしています。check

Design AI/Artificial Intelligence Software Microcomputer/Processor/DSP Qualcomm

AI that "sees and understands" camera footage through words is becoming commonplace in the field.
Visual inspections of manufacturing lines, situational awareness of stores and facilities, and surrounding recognition for vehicles and robots—tasks that were previously judged by human visual inspection are beginning to be taken over by VLM (Vision-Language Model), which understands images and responds in natural language. Unlike conventional image recognition that simply "detects objects," VLM can describe "what is happening" in text and answer questions. This takes the expressive power of edge AI to the next level.

On the other hand, there are always constraints that prevent sending video to the cloud in the field. These include environments with unstable communication, applications where response times cannot be affected by network delays or line instability, and cases where the video itself contains sensitive information that cannot be shared externally. Under these constraints, on-device execution that is self-contained on a single board becomes the practical solution. Only by running VLM at the edge can we bring the eyes of AI to these field locations.

Using the Qualcomm Dragonwing IQ-9075, we will run Qwen2.5-VL-7B, a 7B-class VLM (Vision-Language Model = AI that handles images and language), on the board alone without relying on the cloud. By simply obtaining the model from Qualcomm AI Hub and starting the server with GenieX (Qualcomm's on-device inference runtime), visual language understanding using VLM will run as an OpenAI-compatible API. The inference stack is bundled with the execution environment on the board, so manual setup of libraries is almost unnecessary.
This article will explain the procedure and the actual measurement.

Writer

Hidetaka Iwakura
Macnica Macnica Finesse Company
Technical Management Department, Technical Division 5, Section 2

What you will learn from this article

- Will 7B VLM run on-device with the IQ-9075, and at what speed?

How to obtain models from Qualcomm AI Hub

- Steps for performing inference using Qualcomm GenieX as an OpenAI-compatible API

- Steps to use as an LLM API

Execution environment

The Qualcomm Linux environment on the IQ-9075 comes bundled with the GenAI inference stack as part of the board's OS, providing all the necessary runtime components for immediate inference once a model is prepared. The main components included are as follows:

Included components	role
QNN SDK 2.47	Qualcomm AI Runtime
genie-app / genie-t2t-run	Genie inference execution binary
libGenie.so	Genie Runtime Body
libgenie-vlm.so	VLM-specific runtime
Skel kit for V73	Hexagon NPU Library

Dragonwing IQ-9075 Specifications

This article discusses a high-performance edge AI SoC for IoT and embedded systems, specifically the EVK version (IQ9075-AA) with 100 TOPS. Key specifications are as follows:

item	content
SoCs	Qualcomm QCS9075
NPU	Hexagon V73 x 2 (Dual CDSP)
Arithmetic performance	100 TOPS (IQ9075-AA, total across 2 NPUs; 1 NPU used for single-model execution)
operating temperature	-40〜+115℃
Evaluation unit memory	36GB

Click here for product details

testing environment

Qualcomm IQ9075 evaluation board appearance

element	Structure of this article
board	Dragonwing IQ-9075 EVK (IQ9075-AA)
OS	Qualcomm Linux 2.0 (kernel 6.18.30)
model	Qwen2.5-VL-7B-Instruct (W4A16 Genie, Qualcomm AI Hub)
runtime	GenieX v0.3.13

The QAIRT version required by Qwen2.5-VL-7B-Instruct is 2.45 at the time of this article's verification. The OS-included version 2.47 is higher than this and therefore meets the requirement.
Since the requirements can change with model updates,we will checkthe metadata.json file in the bundle we actually obtained.

Obtaining the model (Qualcomm AI Hub)

Model acquisition is done via GenieX CLI. The GenieX installation command and instructions can be found in the "Quick Start (Linux)" section of the AI Hub model page.geniex infer ai-hub-models/Qwen2.5-VL-7B-InstructThe following is a description of the model.Geniex PullIt will be automatically retrieved (this article also uses this method).
If you want to download the zip file directly, you can do so from the "download the model" link on the same page.

• Model page: Qualcomm AI Hub: Qwen2.5-VL-7B-Instruct

No sign-in required

The device is QCS9075 (IQ-9075), and the runtime/precision is Genie / W4A16.

Execution procedure in GenieX

GenieX is an on-device inference runtime released by Qualcomm. It can run Hugging Face GGUF models and pre-compiled bundles from Qualcomm AI Hub on NPUs/GPUs/CPUs, and features multiple interfaces including CLI, Python, and an OpenAI-compatible server.

The Quick Start guide on the AI Hub model page provides instructionsfor a one-off test using`geniex infer <model>`. In this article, we will use it in two parts: model retrieval (`geniex pull`)and an OpenAI-compatible server (`geniex serve`) so that it can be continuously accessed from apps and agents.

1. インストール ``` curl -fsSL https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/install.sh | sh ``` ---------------------------------------------------------- 2. チップセット設定 → モデル取得 ``` geniex config set chipset qcs9075 geniex pull ai-hub-models/Qwen2.5-VL-7B-Instruct:w4a16 ``` ---------------------------------------------------------- 3. OpenAI互換サーバの起動 ``` geniex serve --host 0.0.0.0:18181 ``` これで http://<board-ip>:18181/v1 にOpenAI互換エンドポイントが立ちます。 ---------------------------------------------------------- 4. 画像を投げてみる（VLM） OpenAIのマルチモーダル形式そのままで、画像はimage_urlにbase64データURIで渡します。 ``` {"model":"qualcomm/Qwen2.5-VL-7B-Instruct", "messages":[{"role":"user","content":[ {"type":"text","text":"Describe this image in detail."}, {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}]}]} ``` OpenAI互換なので、任意のOpenAI互換クライアントをこのエンドポイントに向けるだけで使えます。

The following video demonstrates how an image is sent from a browser's OpenAI-compatible client and how a description is returned.

Actual performance

The 7B VLM decodes an image at approximately 7.6 tok/s and responds with a TTFT (Time to First Token) of approximately 1.1 seconds.

The measurements are based on tokens received via streaming from a browser's OpenAI-compatible client. tok/s represents the pure decoding speed (the generation time) after the initial token, and does not include image upload or network round-trip time. TTFT (Time To First Token) is the one-time waiting time required for image encoding and prefill. Looking at these two separately reveals the breakdown of the perceived performance.

The actual measurements for the three prompts shown in the demo video above (same board, same settings, and the exact values displayed on the screen) are as follows:

Prompt (image)	Generated token	Decode	TTFT
Standing buffet party	56	7.6 tok/s	1.2s
Lecture scene	75	7.6 tok/s	1.0s
Self-ordering kiosk operation	36	7.7 tok/s	1.1s

All tests were performed on the on-device NPU. The 3-prompt example is representative and not a rigorous benchmark. Decoding speed stabilized at 7.6-7.7 tok/s, with a TTFT of approximately 1 second. The user's perceived wait time is determined by "TTFT + number of generated tokens × decoding time per token," and since prefill is performed only once, the longer the response, the closer the effective throughput gets to the decoding rate.

The IQ-9075 has two Hexagon V73 processors, but single-model inference runs on only one of them. Therefore, the above figures represent the performance of one processor, leaving the other processor available as reserve power for simultaneous execution of other models or streams.

The trade-off between convenience and speed: GenieX is an execution path that prioritizes ease of use as an OpenAI-compatible API. If speed is the top priority, the native path using the OS-bundledgenie-appdirectly is faster. In the test environment described in this article, comparing the same decoding rate, GenieX achieved approximately 7.6 tok/s, while direct use of genie-app yielded approximately 10 tok/s (consistent with the Response Rate published on the model page by Qualcomm AI Hub). Depending on the application, you can choose betweenthe convenience of GenieX and the speed of direct use ofgenie-app.

Regarding version consistency

For the writing of this article, we used a model bundle compiled with QAIRT 2.45.
GenieX includes its own QAIRT 2.45 runtime, so the model and runtime versions automatically match. Furthermore, we have confirmedthat models compiled with 2.45 are backward compatible and work correctly with the newer OS standard QNN 2.47 (directly accessed viagenie-app).

Implemented as an alternative to the Cloud LLM API.

GenieX uses an OpenAI-compatible API, so you can replace the cloud with the edge simply by changing the connection URL. Current LLM/VLM applications and agents use the "OpenAI-compatible API" as a common language. Services like OpenRouter act as intermediaries, bundling numerous cloud models together using that common language. What GenieX does is make the edge SoC speak that same common language on its own.

The advantages of replacing it with an edge are as follows:

- Works even on closed networks / Does not stop even if the network is shut down

- No images or text data are ever released outside the system.

- Zero token charges

In short, this single device provides an OpenAI-compatible endpoint that can be set up in a closed network, serving as an alternative to the cloud.

Summary

This article demonstrates how to run 7B VLM (Qwen2.5-VL) visual language understanding on a Qualcomm Dragonwing IQ-9075 using Qualcomm AI Hub and GenieX. From model acquisition to launching an OpenAI-compatible server, the board can explain images and answer questions using just a few commands, without relying on the cloud.

- The board's OS includes the GenAI inference stack, allowing 7B VLM to run simply by "acquiring the model and starting the server."
- Since the inference is output as an OpenAI-compatible API, cloud-based applications and agents can be directly directed to the local edge device. This makes it a cloud alternative not only for VLM but also for text-based LLM.

The era of on-device generation AI running as a standard OS feature is becoming a reality for IoT and embedded SoCs.
If you are interested in Qualcomm SoCs, GenieX, or AI Hub, please feel free to contact us.

Reference links
・Qualcomm AI Hub: https://aihub.qualcomm.com/
・Qwen2.5-VL-7B-Instruct (AI Hub): https://aihub.qualcomm.com/models/qwen2_5_vl_7b_instruct
・GenieX: https://aihub.qualcomm.com/geniex