- Semiconductor BusinessHOME
- Products and Services of Macnica,Inc.
-
technical information
-
Events and Seminars
- Handling Manufacturer
- Support
- Inquiry
- Click here to purchase products
- Semiconductor business e-mail magazine registration
Introduction
With advancements in generative AI and image recognition technologies, there is a growing need to run AI not only in the cloud, but also at the edge, such as on cameras, robots, and industrial equipment.
SiMa.ai provides the MLSoC™ "Modalix" (hereinafter, Modalix) and the software development environment "Palette Neat" for this type of physical AI.
Modalix is an MLSoC targeting a wide range of AI workloads, including CNNs, ViTs, and LLMs. SiMa.ai is deploying Modalix as a physical AI platform with 50 TOPS and under 10W of power consumption.
In this article, Modalix It can be evaluated on actual hardware. Modalix SoM DevKit(Hereafter, Modalix DevKit) and, that development environment Palette Neat I will introduce you to this.
What is Modalix SoM DevKit?
Modalix SoM DevKit is a platform for developing and evaluating edge AI applications using SoMs equipped with SiMa.ai 's MLSoC™ Modalix.
SiMa.ai positions Modalix DevKit as a development platform for evaluating workloads such as computer vision, LLM, and real-time multimodal processing.
Modalix features an ML Accelerator for AI processing, as well as an 8-core Arm Cortex-A65 processor. Furthermore, the Modalix DevKit comes pre-installed with eLxr Linux.
It supports a wide range of AI workloads, from vision AI to generative AI, and by using Modalix DevKit, you can evaluate applications such as the following on actual devices:
- Object detection and image classification
- segmentation
- Vision AI using multiple cameras
- AI generated using LLM/VLM
- An application that combines Vision AI and Generative AI
One of Modalix 's key features is that it supports a wide range of AI models, from Vision to GenAI.
What is Palette Neat?
For Modalix AI A software environment that supports application development Palette Neat is.
Palette Neat is SiMa.ai 's software development toolkit that covers everything from model preparation and application building to verification on Modalix DevKit.
Palette Neat's software stack
The main components are the following four:
①Neat SDK
Neat SDK Modalix To develop applications for Container-based development environment is.
It runs as a container on the host PC, providing a cross-compilation environment for C++ applications, Neat Library headers, integration with DevKit, Neat Insight, and context for AI agents, all in one place.
This development environment is ideal for those who want to perform large-scale application development for Modalix, model preparation, and verification using DevKit primarily on a host PC.
②Neat Library
Neat Library is Modalix Above AI To build and run applications C++ / Python runtime library is.
In addition to C++, PyNeat is also provided for using Neat from Python. Palette Neat uses the Neat Library to run models and build pipelines for AI applications.
③Model Compiler
Model Compiler is a tool for converting AI models such as ONNX into a format that can be executed with Modalix 's ML Accelerator.
If you are using your own AI model, you will need to use Model Compiler to perform quantization and compilation.
On the other hand, if you use pre-compiled models obtained from sources such as Model Zoo (described later), you can start development without installing Model Compiler.
④LLiMa
LLiMa is a runtime for handling generative AI models such as LLM, VLM, and ASR on Modalix.
SiMa.ai provides a development flow for obtaining models for generative AI and running them on Modalix DevKit using LLiMa.
A major feature of Palette Neat is "Agentic Development".
One particularly noteworthy feature of Palette Neat is its Agentic Development Environment, which allows you to develop AI agents.
SiMa.ai positions Palette Neat as an environment that supports the development of physical AI applications by utilizing AI agents.
The Neat SDK includes: Neat Sources and samples AI To make it easier for agents to refer to Agent-ready context If available, Codex and Claude Code Skills for this purpose are provided.
moreover, Neat SDK 2.1.2.3 From now on, it will be available from your browser. Code UI to Codex and Claude Code The extension is pre-installed. It will be done.
Therefore, Neat SDK Within the development environment Claude Code or Codex Using Modalix Develop applications for this purpose. You can.
Graph Surgery by AI Agents
Specific examples of how Agentic Development can be used 1 But, AI Model Graph Surgery is.
The Neat SDK provides the sima-model-surgery skill for Codex/Claude.
You can have an AI agent analyze a model and assist with tasks such as the following:
- Check compatibility with Model Compiler.
- Suggest necessary graph changes.
- Modify ONNX Graph
- Verify the modified model.
- Prepare the model for recompilation.
The SiMa.ai documentation specifically showcases examples of using Graph Surgery for YOLO models, among others, to improve compatibility with the Model Compiler and optimize model output for Modalix.
In other words, Palette Neat provides not only an environment for running AI, but also an environment for developing AI applications while utilizing AI.
DevKit Sync connects the Host PC, Neat SDK, and Modalix DevKit.
In the Neat SDK,
Host PC → Neat SDK → Modalix DevKit
You can build a development environment that integrates these elements.
DevKit Sync is the feature that supports this development flow.
DevKit Sync allows you to access a common /workspace from your host PC, Neat SDK container, and Modalix DevKit.
Because application source code, build results, logs, models, etc., can be shared, there is no need to manually copy files from the host PC to DevKit every time.
Additionally, within the Neat SDK, you can use the `dk` command to execute commands on a paired Modalix DevKit.
One of the key features of the Neat SDK is its ability to seamlessly transition between development on a host PC and actual device testing on Modalix DevKit.
What is Neat Insight?
The Neat SDK includes Neat Insight, a browser-based development and testing UI.
Neat Insight is a particularly useful tool when developing Vision AI applications, and it allows you to perform the following actions from your browser:
- Checking source code, models, and generated artifacts within the workspace
- Management of test videos and images
- Generate an RTSP stream from a video file.
- Verification of video output from the AI application
- Metadata display for Object Detection, Classification, Pose Estimation, Segmentation, Tracking, etc.
For example, when evaluating an object detection application, you can register a test video in Neat Insight to create an RTSP stream, and then input that video into an AI application on Modalix DevKit.
By returning the AI-processed video and the metadata of the detection results to Neat Insight, you can view them in Video Viewer with bounding boxes and other elements superimposed.
In short, Neat Insight is a tool that allows you to perform everything from preparing test inputs for AI applications to verifying inference results, all within your browser.
Summary
This article introduced SiMa.ai 's Modalix SoM DevKit and Palette Neat.
Modalix SoM DevKit allows you to evaluate a wide range of physical AI workloads on actual devices, from Vision AI to Generative AI such as LLM.
Palette Neat allows you to develop AI applications for Modalix by combining the following features:
- Development environment for Modalix using Neat SDK
- AI application development using Neat Library / PyNeat
- Model quantization and compilation using Model Compiler
- Running the GenAI model using LLiMa
- Integration of Host PC, SDK, and DevKit via DevKit Sync
- Checking video and inference results using Neat Insight
- Using compiled models with Model Zoo
- Aggressive Development using Claude Code/Codex
In particular, Palette Neat is characterized by its ability to integrate AI agents such as Claude Code and Codex into the development environment, in addition to runtimes and development tools for running models, enabling efficient development of physical AI applications that leverage AI.
Furthermore, a major feature of Palette Neat is its support for "Agentic Development," which incorporates these AI agents into the development process. By utilizing AI-powered code generation and development support, it reduces the workload on developers and streamlines the process from idea to implementation.
If you'd like to try out Modalix SoM DevKit, the following article explains how to set up Palette Neat.
Inquiry
Please feel free to contact us with any questions about our products, technical inquiries, sample requests, or estimates.