Site Search

Introduction

AI Agents are now capable of more than just answering questions; they can understand the situation, call up the necessary tools, and execute tasks. NVIDIA The company uses these agents in a Box (*1 As a foundation for running within ) NVIDIA NemoClaw™ (*2) is offered. NVIDIA DGX Spark™ (*3) If you use this, LLM From inference of a large-scale language model to agent execution, on your local machine 1 It can all be done on one table.

*1 Box: A secure execution environment isolated from the outside world.
*2 NVIDIA NemoClaw: OpenClaw and Hermes Agent are open-source AI agents, and this is a secure execution environment for AI agents. NVIDIA OpenShell™ A stack made up of these elements.
*3
 NVIDIA DGX Spark: NVIDIA GB10 Grace Blackwell Superchip and 128GB Desktop size with integrated memory AI Supercomputer.

Macnica, NVIDIA Our partner companies (NPN: NVIDIA Partner Network) targeting Japan NVIDIA NPN Partner Agentic AI Hackathon Our company members participated in "[ ]" Smart Home Agent × Digital Twin on NVIDIA DGX Spark We developed and presented our results based on the theme of "[theme name]".

This article will describe the deliverables developed at this hackathon. AI The agent receives instructions in natural language and operates and checks the equipment and sensors in the miniature house, and then monitors its status. NVIDIA Omniverse™ (*4) Digital twin (*5 Regarding the system which will also be reflected in the following, the overall structure and the methods used NVIDIA We will introduce our company's technology and implementation innovations.

*4 NVIDIA Omniverse: NVIDIA It is provided as open source by the company. 3D A platform for simulation and digital twin construction.
*5
Digital twin: A virtual representation of a real-world object or space, like a twin.

Challenges and approaches taken

Whether in a home, office, or factory, each piece of equipment and sensor is manufactured and operated independently. It's primarily human responsibility to interpret situations like "it's hot," "it's dark," or "someone has arrived," and then translate that information into instructions for which equipment to operate and how. Even with automation, the only solution is to continuously add rules for each piece of equipment and each specific situation.

In other words, the ability to treat all devices as a single system and make flexible decisions based on the situation is still not fully developed. We thought that this problem could be solved by utilizing AI agents. Therefore, for this project, we chose a smart home as our theme, where an AI agent connects with all devices, assesses the situation, and controls them accordingly.

System configuration

The system consists of four layers: ① edge/device, ② agent + control system, ③ AI model, and ④ digital twin. Everything except the edge runs on two​ ​DGX Spark servers, one dedicated to LLM and the other to agents, database, voice input, and Omniverse (digital twin). It does not rely on the cloud, and all processing is completed locally.

Figure 1 System Configuration

For the edge microcontroller, we adopted the Texas Instruments (TI) CC3235MODSF wireless microcontroller (LaunchPad development kit). It controls sensors, fans, servo motors, LCDs, etc. via SPI, I2C, PWM, and GPIO, and also acts as its own Wi-Fi access point to communicate with the DGX Spark (Macnica also handles TI products).

The microcontroller transmits temperature, humidity, and the status of each device every second, and the DGX Spark transmission/reception system records this data in a database (DB: SQLite). The agent checks the status and operates the devices through the REST API exposed by this transmission/reception system. The main technologies used are shown in Table 1.

role

Technology used

point

Computing infrastructure

NVIDIA DGX Spark × 2

#1: Agent, DB, Omniverse; #2: LLM only. All processing runs locally.

inference

NVIDIA Nemotron™ 3.5 Lightning 30B-A3B (*6), vLLM + DSpark (speculative decoding) (*7)

Improved real-time response with high throughput

agent

NVIDIA NemoClaw(OpenShell + OpenClaw)

Control all devices from a single agent.

digital twin

NVIDIA Omniverse

Confirmation of the design and coordination with the actual device's operation.

Edge (microcontroller)

TI CC3235MODSF LaunchPad

Each device is controlled by a signal. It communicates with DGX Spark as an access point.

Database (DB)

FastAPI + SQLite

Record values every second and provide them via a REST API.​

Voice input

faster-whisper

The system detects when the speaker has finished speaking and automatically sends the transcript.

Design and Development

Claude Code(※8)、Blender

It can be used in a wide range of applications, from part modeling to coding.

Table 1 Main technologies used



*6
 NVIDIA Nemotron 3.5 Lightning 30B-A3B: Nemotron is an open LLM family released by NVIDIA.
"30B-A3B" represents a MoE (Mixture of Experts) structure in which approximately 3 billion out of a total of approximately 30 billion parameters are used during inference.
*7
 vLLM /Speculative Decoding: vLLM is an open-source LLM inference engine. Speculative decoding involves predicting and verifying subsequent tokens in advance.
A method to speed up production while maintaining quality.
*8
 Claude Code: An AI coding agent from Anthropic.

Implementation points

1. Use an AI agent to control devices throughout your home using natural language.

We used NVIDIA NemoClaw as the execution platform for the agents. A single agent handles everything from checking sensor values to operating equipment.

Figure 2: One agent controls all the devices in the house.

The devices in the house are exposed as a REST API for the transmission and reception system. The agent was given the "skill to operate the house (*9)".


*9
Skills: Instruction manuals that are given to agents, outlining how to perform specific tasks (such as available APIs and how to call them). In this case, the skills describe how to call the APIs of each device.

Usage

APIs

Fan rotation speed (10 to 100%)

POST /api/fan

Door opening and closing

POST /api/servo

Opening and closing curtains

POST /api/servo2

LED lights up

POST /api/led

Display on LCD​

POST /api/lcd

Current status (temperature, humidity, equipment status)

GET /api/telemetry/latest

Table 2 APIs called from the agent


The only skills I've written down are the bare minimum, like opening the door when someone says "I'm home." I leave decisions like turning on the LED lights when someone says "It's dark" to the LLM (Live Learner).
Even without covering all the detailed rules, the agent will understand your intentions and act accordingly. Furthermore, OpenShell 's communication policy (*10) restricts the agent's access to only the sending and receiving systems.

*10 OpenShell communication policy: A rule that restricts where the agent can communicate.

We also prepared a small program to monitor the room temperature. When the room temperature exceeds 30°C, this program requests the agent to "turn on the fan" in natural language. The equipment operates autonomously through the same agent, without requiring any human intervention.

2. Digital twin synchronized with the real thing

We assigned two roles to the digital twin: "to be used for design and operational verification" and "to reflect the actual state of the object."

Figure 3: Designed using a digital twin and synchronized with the 3D printed physical object.
1. Design and verify functionality using Claude Code.

We created 3D assets and a house model from parts photos and datasheets to build a digital twin of the smart home. We then created and refined the agent's skills while verifying their operation within that digital twin.

2. Create the actual object using 3D printing.

We printed the same model as the digital twin and then assembled and installed the equipment.

3. Synchronize the physical house with the digital twin.

The microcontroller sends the house status every second, and the transmission/reception system records this data in a database. The Omniverse then retrieves the latest status via an API and reflects it in each part of the digital twin. This synchronizes the status of elements such as fans, doors, LEDs, and LCDs with that of the actual house.

demo

During the presentation, we conducted a live demonstration, showing how a real house moved in approximately 5 to 8 seconds after the request was made, and how the same state was reflected in the digital twin.

request

Agent operation

The actual house and the digital twin

"Open the curtains."

Instructions to open the curtains

The curtains open

"I'm home"

Understand the intention and give the instruction to open the door.

The door opens

"It's so dark."

The instruction was to understand the intent and turn on the LED.​

LED lights up

(Autonomous operation) Room temperature is 30℃ or higher

The monitoring program instructed the fan to rotate.

The fan is spinning.

Table 3: Contents of the demo

Figure 4: Door open when saying "I'm home" (Left: Digital twin in Omniverse, Right: Actual house)


Even with utterances that don't include device names, such as "I'm home" or "It's dark," the agent can select and operate the appropriate device.

Future prospects

The demonstration showcased a configuration where all sensors and equipment are connected to an AI agent, which then assesses the situation, operates the equipment, and reflects the results in a digital twin. This configuration can be applied not only to homes but also to various other locations. One example of its application is a "factory manager agent" that manages factory sensors and equipment. NVIDIA has released the NVIDIA Factory Operations Blueprint (FOX).  Examples of configuring a factory manager agent with NemoClaw, such as the one shown, have been published. Table 4 shows the correspondence with this demo.

element

Examples of factory configurations

This demo

Manager Agent

Factory Manager by NemoClaw​

House Manager by NemoClaw​

Target to connect to

Sensor and device signals

Microcontroller values (every second) and equipment

Instruction methods and safety measures

Natural language + control via OpenShell​

Natural language + communication policy

Models and Computers

Fine-tuned Nemotron, NVIDIA DGX Station™ (* 11)

Nemotron 3.5 Lightning、DGX Spark

digital twin

Omniverse Digital Twin

Omniverse Digital Twin

Table 4: Dealing with Factory Manager Agents


*11
 NVIDIA DGX Station: A desk-side AI supercomputer that supports training and inference of larger models than DGX Spark.

We believe this same configuration can be applied to other locations with many pieces of equipment and sensors, such as buildings, shops, and warehouses, in addition to factories.

Going forward, we plan to work on mechanisms to improve real-world systems based on test results on digital twins, as well as more advanced control using agents.

in conclusion

This article showcased a demonstration of a "smart home agent x digital twin" system created during the "Japan NVIDIA NPN Partner Agentic AI Hackathon," combining NVIDIA NemoClaw, Nemotron, and NVIDIA Omniverse on NVIDIA DGX Spark. We confirmed that it is possible to build a system where all processing is completed locally, all devices are connected to the AI agent, and their status can be monitored using a digital twin.

We hope this article will be helpful when considering systems that utilize AI agents and digital twins. As an authorized distributor of NVIDIA, Macnica provides comprehensive proposals and technical support for NVIDIA products and technologies, from edge computing to large-scale AI infrastructure. Please feel free to contact Macnica if you are considering using AI agents or digital twins.

Inquiry