Introduction
AI Agents are now capable of more than just answering questions; they can understand the situation, call up the necessary tools, and execute tasks. NVIDIA The company uses these agents in a Box (*1 As a foundation for running within ) NVIDIA NemoClaw™ (*2) is offered. NVIDIA DGX Spark™ (*3) If you use this, LLM From inference of a large-scale language model to agent execution, on your local machine 1 It can all be done on one table.
*1 Box: A secure execution environment isolated from the outside world.
*2 NVIDIA NemoClaw: OpenClaw and Hermes Agent are open-source AI agents, and this is a secure execution environment for AI agents. NVIDIA OpenShell™ A stack made up of these elements.
*3 NVIDIA DGX Spark: NVIDIA GB10 Grace Blackwell Superchip and 128GB Desktop size with integrated memory AI Supercomputer.
Macnica, NVIDIA Our partner companies (NPN: NVIDIA Partner Network) targeting Japan NVIDIA NPN Partner Agentic AI Hackathon Our company members participated in "[ ]" Smart Home Agent × Digital Twin on NVIDIA DGX Spark We developed and presented our results based on the theme of "[theme name]".
This article will describe the deliverables developed at this hackathon. AI The agent receives instructions in natural language and operates and checks the equipment and sensors in the miniature house, and then monitors its status. NVIDIA Omniverse™ (*4) Digital twin (*5 Regarding the system which will also be reflected in the following, the overall structure and the methods used NVIDIA We will introduce our company's technology and implementation innovations.
*4 NVIDIA Omniverse: NVIDIA It is provided as open source by the company. 3D A platform for simulation and digital twin construction.
*5 Digital twin: A virtual representation of a real-world object or space, like a twin.
Challenges and approaches taken
Whether in a home, office, or factory, each piece of equipment and sensor is manufactured and operated independently. It's primarily human responsibility to interpret situations like "it's hot," "it's dark," or "someone has arrived," and then translate that information into instructions for which equipment to operate and how. Even with automation, the only solution is to continuously add rules for each piece of equipment and each specific situation.
In other words, the ability to treat all devices as a single system and make flexible decisions based on the situation is still not fully developed. We thought that this problem could be solved by utilizing AI agents. Therefore, for this project, we chose a smart home as our theme, where an AI agent connects with all devices, assesses the situation, and controls them accordingly.
System configuration
The system consists of four layers: ① edge/device, ② agent + control system, ③ AI model, and ④ digital twin. Everything except the edge runs on two DGX Spark servers, one dedicated to LLM and the other to agents, database, voice input, and Omniverse (digital twin). It does not rely on the cloud, and all processing is completed locally.
For the edge microcontroller, we adopted the Texas Instruments (TI) CC3235MODSF wireless microcontroller (LaunchPad development kit). It controls sensors, fans, servo motors, LCDs, etc. via SPI, I2C, PWM, and GPIO, and also acts as its own Wi-Fi access point to communicate with the DGX Spark (Macnica also handles TI products).
The microcontroller transmits temperature, humidity, and the status of each device every second, and the DGX Spark transmission/reception system records this data in a database (DB: SQLite). The agent checks the status and operates the devices through the REST API exposed by this transmission/reception system. The main technologies used are shown in Table 1.
|
role |
Technology used |
point |
|---|---|---|
|
Computing infrastructure |
NVIDIA DGX Spark × 2 |
#1: Agent, DB, Omniverse; #2: LLM only. All processing runs locally. |
|
inference |
NVIDIA Nemotron™ 3.5 Lightning 30B-A3B (*6), vLLM + DSpark (speculative decoding) (*7) |
Improved real-time response with high throughput |
|
agent |
NVIDIA NemoClaw(OpenShell + OpenClaw) |
Control all devices from a single agent. |
|
digital twin |
NVIDIA Omniverse |
Confirmation of the design and coordination with the actual device's operation. |
|
Edge (microcontroller) |
TI CC3235MODSF LaunchPad |
Each device is controlled by a signal. It communicates with DGX Spark as an access point. |
|
Database (DB) |
FastAPI + SQLite |
Record values every second and provide them via a REST API. |
|
Voice input |
faster-whisper |
The system detects when the speaker has finished speaking and automatically sends the transcript. |
|
Design and Development |
Claude Code(※8)、Blender |
It can be used in a wide range of applications, from part modeling to coding. |
Table 1 Main technologies used
*6 NVIDIA Nemotron 3.5 Lightning 30B-A3B: Nemotron is an open LLM family released by NVIDIA.
"30B-A3B" represents a MoE (Mixture of Experts) structure in which approximately 3 billion out of a total of approximately 30 billion parameters are used during inference.
*7 vLLM /Speculative Decoding: vLLM is an open-source LLM inference engine. Speculative decoding involves predicting and verifying subsequent tokens in advance.
A method to speed up production while maintaining quality.
*8 Claude Code: An AI coding agent from Anthropic.
Implementation points
1. Use an AI agent to control devices throughout your home using natural language.
We used NVIDIA NemoClaw as the execution platform for the agents. A single agent handles everything from checking sensor values to operating equipment.
The devices in the house are exposed as a REST API for the transmission and reception system. The agent was given the "skill to operate the house (*9)".
*9 Skills: Instruction manuals that are given to agents, outlining how to perform specific tasks (such as available APIs and how to call them). In this case, the skills describe how to call the APIs of each device.
|
Usage |
APIs |
|---|---|
|
Fan rotation speed (10 to 100%) |
POST /api/fan |
|
Door opening and closing |
POST /api/servo |
|
Opening and closing curtains |
POST /api/servo2 |
|
LED lights up |
POST /api/led |
|
Display on LCD |
POST /api/lcd |
|
Current status (temperature, humidity, equipment status) |
GET /api/telemetry/latest |
Table 2 APIs called from the agent
The only skills I've written down are the bare minimum, like opening the door when someone says "I'm home." I leave decisions like turning on the LED lights when someone says "It's dark" to the LLM (Live Learner).
Even without covering all the detailed rules, the agent will understand your intentions and act accordingly. Furthermore, OpenShell 's communication policy (*10) restricts the agent's access to only the sending and receiving systems.
*10 OpenShell communication policy: A rule that restricts where the agent can communicate.
We also prepared a small program to monitor the room temperature. When the room temperature exceeds 30°C, this program requests the agent to "turn on the fan" in natural language. The equipment operates autonomously through the same agent, without requiring any human intervention.
2. Digital twin synchronized with the real thing
We assigned two roles to the digital twin: "to be used for design and operational verification" and "to reflect the actual state of the object."
1. Design and verify functionality using Claude Code.
We created 3D assets and a house model from parts photos and datasheets to build a digital twin of the smart home. We then created and refined the agent's skills while verifying their operation within that digital twin.
2. Create the actual object using 3D printing.
We printed the same model as the digital twin and then assembled and installed the equipment.
3. Synchronize the physical house with the digital twin.
The microcontroller sends the house status every second, and the transmission/reception system records this data in a database. The Omniverse then retrieves the latest status via an API and reflects it in each part of the digital twin. This synchronizes the status of elements such as fans, doors, LEDs, and LCDs with that of the actual house.
demo
During the presentation, we conducted a live demonstration, showing how a real house moved in approximately 5 to 8 seconds after the request was made, and how the same state was reflected in the digital twin.
|
request |
Agent operation |
The actual house and the digital twin |
|---|---|---|
|
"Open the curtains." |
Instructions to open the curtains |
The curtains open |
|
"I'm home" |
Understand the intention and give the instruction to open the door. |
The door opens |
|
"It's so dark." |
The instruction was to understand the intent and turn on the LED. |
LED lights up |
|
(Autonomous operation) Room temperature is 30℃ or higher |
The monitoring program instructed the fan to rotate. |
The fan is spinning. |
Table 3: Contents of the demo
Even with utterances that don't include device names, such as "I'm home" or "It's dark," the agent can select and operate the appropriate device.
Future prospects
The demonstration showcased a configuration where all sensors and equipment are connected to an AI agent, which then assesses the situation, operates the equipment, and reflects the results in a digital twin. This configuration can be applied not only to homes but also to various other locations. One example of its application is a "factory manager agent" that manages factory sensors and equipment. NVIDIA has released the NVIDIA Factory Operations Blueprint (FOX). Examples of configuring a factory manager agent with NemoClaw, such as the one shown, have been published. Table 4 shows the correspondence with this demo.
|
element |
Examples of factory configurations |
This demo |
|---|---|---|
|
Manager Agent |
Factory Manager by NemoClaw |
House Manager by NemoClaw |
|
Target to connect to |
Sensor and device signals |
Microcontroller values (every second) and equipment |
|
Instruction methods and safety measures |
Natural language + control via OpenShell |
Natural language + communication policy |
|
Models and Computers |
Fine-tuned Nemotron, NVIDIA DGX Station™ (* 11) |
Nemotron 3.5 Lightning、DGX Spark |
|
digital twin |
Omniverse Digital Twin |
Omniverse Digital Twin |
Table 4: Dealing with Factory Manager Agents
*11 NVIDIA DGX Station: A desk-side AI supercomputer that supports training and inference of larger models than DGX Spark.
We believe this same configuration can be applied to other locations with many pieces of equipment and sensors, such as buildings, shops, and warehouses, in addition to factories.
Going forward, we plan to work on mechanisms to improve real-world systems based on test results on digital twins, as well as more advanced control using agents.
in conclusion
This article showcased a demonstration of a "smart home agent x digital twin" system created during the "Japan NVIDIA NPN Partner Agentic AI Hackathon," combining NVIDIA NemoClaw, Nemotron, and NVIDIA Omniverse on NVIDIA DGX Spark. We confirmed that it is possible to build a system where all processing is completed locally, all devices are connected to the AI agent, and their status can be monitored using a digital twin.
We hope this article will be helpful when considering systems that utilize AI agents and digital twins. As an authorized distributor of NVIDIA, Macnica provides comprehensive proposals and technical support for NVIDIA products and technologies, from edge computing to large-scale AI infrastructure. Please feel free to contact Macnica if you are considering using AI agents or digital twins.