Writer

Hidetaka Iwakura
Macnica Macnica Finesse Company
Technical Management Department, Technical Division 5, Section 2

What you will learn from this article

- A mechanism that allows GGUF to run on the Hexagon NPU

- Differences between the two NPU execution paths that GenieX offers

- Reasons for selecting Q4_0 when using the NPU with GGUF

- Measured differences between the three paths (prefill, response start time, CPU usage)

- Ease of bringing in items and NPU performance vary depending on the route.

Verification environment

• Board: Dragonwing IQ-9075 EVK (CPU has 8 cores)

• Model: Qwen3-4B (llama.cpp route is unsloth/Qwen3-4B-GGUF:Q4_0, QAIRT route is qualcomm/Qwen3-4B:W4A16)

Runtime: GenieX v0.4.0, QAIRT 2.45, llama.cpp Hexagon backend (included with GenieX v0.4.0)

The measurements primarily focus on speed and CPU utilization. This is not a comparison to demonstrate that the output quality of Q4_0 and W4A16 are equivalent.

Dragonwing IQ-9075 Specifications

This article discusses a high-performance edge AI SoC for IoT and embedded systems, specifically the EVK version (IQ9075-AA) with 100 TOPS. Key specifications are as follows:

item	content
SoCs	Qualcomm QCS9075
NPU	Hexagon V73 x 2 (Dual CDSP)
Arithmetic performance	100 TOPS (IQ9075-AA, total across 2 NPUs; 1 NPU used for single-model execution)
operating temperature	-40〜+115℃
Evaluation unit memory	36GB

Click here for product details

GenieX has two inference runtimes.

GenieX is an inference environment for running LLM and VLM on Qualcomm devices. Two runtimes are available through a common interface including a CLI, Python API, and OpenAI-compatible API. llama_cpp uses the llama.cpp engine, with its internal GGML Hexagon backend handling NPU execution. qairt is another runtime that uses Qualcomm AI Engine Direct (QNN/QAIRT).

The official documentation distinguishes between the runtime that handles GGUF as llama_cpp and the runtime that handles pre-compiled bundles from Qualcomm AI Hub as qairt.

Neither of these two approaches is always superior. The best approach differs depending on whether you want to quickly test any model or aim for high NPU performance on your target SoC.

The runtime backend determines where GGUF is executed.

GGUF is a file format and does not specify the execution destination itself. If GenieX supports the GGUF file, it can be executed on the CPU, GPU, or Hexagon NPU using the same file.

The GGUF stores information such as the model's weights, quantization format, and network structure. However, the llama.cpp backend, not the GGUF itself, determines which processor performs the matrix operations.

The GenieX llama.cpp runtime supports CPUs, Adreno GPUs, and Hexagon NPUs. Therefore, you can use the same file to change the processing destination as long as the GGUF supports the GenieX target architecture, quantization, and computation. This does not mean that any GGUF will work on all Qualcomm devices.

GenieX can retrieve models from various sources using the --model-hub option, supporting Qualcomm AI Hub, Hugging Face, ModelScope, Docker registries, and local file systems. It's also possible to load GGUF models that have already been retrieved from the local machine.

The following is a minimal example using an IQ-9075 EVK with GenieX v0.4.0 already installed. A network connection and sufficient storage space are required for the initial model download. For GenieX installation and supported platforms, please see below.Official QuickstartPlease check.

# Hexagon NPUで実行 geniex infer unsloth/Qwen3-4B-GGUF:Q4_0 \ --compute npu \ --ngl -1 \ --think=false # CPUで実行 geniex infer unsloth/Qwen3-4B-GGUF:Q4_0 \ --compute cpu \ --ngl 0 \ --think=false # 対応tensorをNPU、残りをCPUへ割り当てるhybrid経路（`--think=false`はQwen3の思考モードを無効化する指定） geniex infer unsloth/Qwen3-4B-GGUF:Q4_0 \ --compute hybrid \ --ngl -1 \ --think=false

--ngl -1 is a specification that requests llama.cpp to offload all layers. The name originates from llama.cpp and is an abbreviation for GPU layers, but the same specification is used for offloading to the Hexagon NPU. --compute npu is a fixed path to the HTP (Hexagon Tensor Processor, the computational part of the Hexagon NPU), and model loading or inference may fail due to unsupported calculations or memory allocation failures on the NPU side.

On the other hand, --compute hybrid allocates the corresponding tensors to the NPU and processes the rest with the CPU. Hybrids that include a fixed NPU and CPU fallback should be evaluated separately.

The quantization formats that can be implemented on the Hexagon NPU are limited.

GGUF has several quantization formats, including Q4_0, Q8_0, and Q4_K_M. K-quant is commonly used for CPU applications, but the official GenieX documentation recommends Q4_0 for the Hexagon NPU backend. This is because the Hexagon backend can only handle a limited number of quantization formats.
Upon examining the source code of upstream llama.cpp at the time of this verification, it was found that the Hexagon backend accepts F16, F32, IQ4_NL, MXFP4, Q4_0, Q4_1, and Q8_0, but K-quant is not supported. Furthermore, only Q4_0, Q8_0, and MXFP4 are repacked on the NPU side.

This is where the practical pitfall lies. Most GGUFs distributed by the public are Q4_K_M, and if you use them as is, you won't get an error, but they will run on the CPU without being loaded onto the NPU. It doesn't manifest as "it won't run on the NPU," but rather as "it's running on the CPU without you realizing it."

The quantization name is read as follows:

Q4_K_M does not mean that all tensors are uniformly 4 bits; the effective bits/weight varies depending on the model. In the Llama 3.1 8B example in llama.cpp, it is 4.8944 bits/weight.

Only Q4_0 was measured in this verification. Models that have a Q4_0 version in their distribution can run that file directly via the NPU.

If a Q4_0 version is not available, the standard tools in llama.cpp involve a two-step process: first, creating a GGUF from the original BF16 or FP16 checkpoint, and then quantizing that GGUF to Q4_0.

python convert_hf_to_gguf.py \
  --remote <organization>/<model> \
  --outfile model-bf16.gguf \
  --outtype bf16
./build/bin/llama-quantize \
  model-bf16.gguf \
  model-Q4_0.gguf \
  Q4_0

Double quantization, which involves converting already quantized GGUFs such as Q4_K_M to Q4_0, is not recommended. The official llama.cpp documentation explains that requantizing a quantized model can significantly degrade quality compared to quantizing from 16-bit or 32-bit source data.

QAIRT routes run on W4A16 bundles.

Another QAIRT route retrieves model bundles compiled for the target chipset from Qualcomm AI Hub. LLM primarily uses W4A16, which combines 4-bit weights and 16-bit activations.

# GenieX v0.4.0での実測時モデルID geniex infer qualcomm/Qwen3-4B:W4A16 \ --compute npu \ --think=false

The model namespace varies depending on the version of GenieX. At the time of this article's verification, it is qualcomm/Qwen3-4B, while the official documentation refers to similar AI Hub models in the format ai-hub-models/Qwen3-4B. You can check the actual notation in the model guide for the version you are using and in the output of geniex infer -h.

QAIRT bundles are for NPUs only. Unlike GGUF, which switches to the CPU at runtime, the quantization method, target chipset, context length, etc., are determined when the bundle is created.

While using a pre-configured model makes implementation easy, bundling an arbitrary LLM yourself requires considering model-specific export pipelines, quantization data, KV caches, partitioned graphs, and context binaries. GGUF cannot be directly converted to W4A16; you must revert to the original model checkpoint and corresponding export procedure.

Three pathways were measured using IQ-9075.

In subsequent measurements, the terminology will be used with the following meanings:

On the Dragonwing IQ-9075 EVK, the same Qwen3-4B sequence was measured using the following three paths: QAIRT is qualcomm/Qwen3-4B:W4A16, and llama.cpp is unsloth/Qwen3-4B-GGUF:Q4_0.

・llama.cpp CPU

・llama.cpp Q4_0 / Hexagon NPU

・QAIRT W4A16 / Hexagon NPU

The NPU route in llama.cpp is fixed to --compute npu execution. The hybrid route is not included in this measurement.

First, let's compare the speed of prefilling the input. Comparing the CPU and NPU of llama.cpp at the same measurement boundary in llama-bench, the input processing equivalent to 512 tokens was 444.69 tok/s for the NPU and 75.82 tok/s for the CPU, meaning llama.cpp NPU was approximately 5.9 times faster. Since QAIRT cannot be measured in llama-bench, this 5.9 times is a comparison within the llama.cpp path.

Note that this value represents the pure input processing throughput measured by llama-bench and does not match the value calculated inversely from TTFT, which will be discussed later. This is because TTFT includes time other than input processing, and the processing cost per token increases as the input length increases. This 5.9x is the value obtained by running the distributed GGUF as is. Hexagon NPU achieves this difference with prefills without needing to prepare a dedicated bundle.

Next, we look at how much CPU was used for the same process. We generated 128 tokens from a short prompt and converted the process CPU time during the measurement interval to the number of logical cores by dividing it by the actual time.

The generation speed of short responses was similar across all three paths, with differences ranging from a few percent to 20%. The biggest difference was in CPU utilization. The QAIRT NPU path maintained approximately 94% of the generation speed of the CPU path while reducing the average CPU utilization from 6.64 cores to 0.83 cores. This means that approximately 5.8 out of 8 cores are freed up for other processing.

In other words, even comparing only the generation speed of short responses, the difference between the three paths is small, and this metric alone does not make it clear why NPUs are useful. NPUs are effective in edge LLMs for processing long inputs, which we will discuss next, and for the aforementioned CPU utilization.

The value of NPUs lies in long text input and CPU resource management.

The LLM process is divided into two main parts: prefill, which reads the input, and decode, which generates tokens one by one.

Prefill is a process that easily leverages the parallel processing capabilities of the NPU because it processes large matrix operations in batches. On the other hand, decoding involves repeatedly performing small calculations and continuously reading large model weights, so it is strongly affected by memory bandwidth.

Therefore, for applications involving inputting long prompts, search results, and conversation history, speeding up prefill directly translates to a reduction in response time. On the other hand, decoding is heavily influenced by memory bandwidth, and using an NPU does not necessarily guarantee faster decoding.

Furthermore, the difference in CPU utilization becomes significant when running other processes concurrently on the same device. In the measurements in the previous section, the CPU path consistently utilized an average of 6.64 out of 8 cores, while the QAIRT NPU path utilized only 0.83 cores.

In a configuration where most of the 8 cores are dedicated to LLM inference, there is less capacity left for the following processes:

- Processing of camera and sensor inputs

- Speech recognition and speech synthesis

• Database search

- Execute agent tools

- Responses from Web APIs and UIs

This verification measured the CPU usage of the LLM inference process, and did not measure the performance when these processes are combined.

The NPU is not simply a computing unit that speeds up LLMs; it also plays a role in freeing up the CPU from LLM inference, creating processing headroom for the entire device.

Differences in personality revealed in the two paths

Summary

Where a GGUF is executed is determined by the runtime backend, not its format. Using the GenieX llama.cpp runtime, you can run the corresponding GGUF on a Hexagon NPU. GenieX has two NPU paths:

• GGUF + llama.cpp Hexagon: A versatile path that makes it easy to import a wide range of models.

Qualcomm AI Hub + QAIRT: A pre-compiled NPU path for the target chipset.

In actual measurements with the IQ-9075, the llama.cpp NPU prefilled 512 tokens approximately 5.9 times faster than the same llama.cpp CPU path. In a TTFT with 3,840 tokens input, the QAIRT NPU was 17.6 times faster than the llama.cpp CPU, while the llama.cpp NPU was 7.1 times faster. On the other hand, for short decoding, the CPU was sometimes faster. Looking only at generation speed, the difference between paths was small, but the ranking changed when input length, response start time, and CPU usage were also included.

The above are the results of actual measurements under the conditions of IQ-9075 and Qwen3-4B. The values and rankings between paths will change if the target SoC, model, and bundle availability change.

Inquiry

If you are interested in building on-device LLMs using Qualcomm Dragonwing SoCs, or implementing models using GenieX and Qualcomm AI Hub, please feel free to contact us.