Introduction
Have you ever used an AI agent for work and experienced reaching its usage limit midway through a task, bringing your work to a halt? Cloud services are convenient, but each contract plan has usage limits, meaning you may not be able to use them fully when you need them. Another obstacle is security. For tasks involving confidential information that cannot be sent externally, such as customer data, source code, or unreleased deliverables, sending prompts to cloud services itself may be restricted. The more convenient a tool is, the more information you need to input initially, so the decision of "is it okay to share this information?" arises every time you use it, which can be a reason why its adoption doesn't progress. In addition, APIs have safeguards in place, so there are security-related tasks that cannot be performed.
On the other hand, OpenWaite's LLM has begun to show performance approaching that of Frontier models in public benchmarks. Many people are probably wondering if they can run the agent in their own environment. However, when it comes to actually considering implementation, there isn't much information to help make a decision.
Is it really usable in business?
How many people can use it simultaneously?
Can the investment be recouped?
Very little actual measurement data to answer these questions has yet been made public.
In this series, we will introduce our team's case study of migrating the use of AI agents for 30 employees within our company from the cloud to on-premises, along with actual measurement data.
The series consists of three parts, each covering the following topics:
◆Episode 1: The Transition Process and Structure
— To what extent can daily operations be handled with an open-weighted LLM? I will introduce the configuration of a hybrid inference environment and the results of my verification in actual work (402 out of 411 requests were completed on-premises).
◆Episode 2: Performance and Cost Verification
— We will quantitatively measure how many agents a single GPU server can support using public benchmarks. In addition, we will estimate, based on actual operational data, how many users are needed to recoup the initial investment of 100 million yen in the environment.
◆Episode 3: Technical Details
— This section explains the routing mechanism and configuration files, how to run benchmarks, the cost estimation process, and other details necessary to build the same environment yourself.
The figures in this article are based on actual measurements and operational data as of September 2026. Exchange rates, API unit prices, and benchmark results are subject to change, so please check the latest information when considering implementation.
Background: Cloud service usage limits
Prior to the migration, our company relied on cloud services such as Claude (Team plan), Microsoft 365, and GitHub Copilot for its internal agent usage. These services had limits on the number of seats and usage per contract plan, and once these limits were reached, work could not be continued. Furthermore, some services were not deployed to all employees, which was one reason why the use of generative AI was not progressing. The advantage of agents lies in the fact that other tasks can be carried out in parallel while the agent is handling a task. This advantage cannot be fully utilized when usage limits are a concern. In addition, there were concerns about increased costs if we switched to pay-as-you-go pricing and concerns about transmitting confidential information externally.
In addition to the above, there were two other concerns regarding reliance on cloud APIs, each of a different nature. One was the risk that business continuity would become difficult due to changes in the provider's specifications or service interruptions. This risk is troublesome because it is difficult to eliminate through contracts and operational adjustments alone. The other was that not all tasks could be resolved due to safeguards on the provider's side. The fact that some tasks could not be handled would also limit the scope of what could be utilized. Therefore, we considered placing the model that handles daily operations on-premises.
Reason for consideration: Improving the performance of open-weight LLM
A few years ago, local LLMs were not at a business-grade level in terms of both response speed and performance. However, the performance of open-weighted models (where weights are publicly available and downloadable by anyone) has been rapidly improving. The model adopted this time is Z.ai's zai-org/GLM-5.3-Flash, which has reached the same level as the previous generation Frontier model, Claude Opus 4.8. It is also the first model in the GLM-5 series to natively support multimodal input, and can handle not only text but also images and videos as input [1].
In actual operation, NVIDIA uses the nvidia/GLM-5.3-Flash-NVFP4, which is a quantized version of this model using 4 bits (NVFP4) [2]. Even after quantization, the accuracy remains nearly the same as the original model (0.9211 for GPQA Diamond compared to 0.9217 for the original model). It also directly supports multimodal input.
| model | Intelligence Index [3] | Classification |
| Claude Opus 5.5 | 58 | Latest Frontier (Closed, target for this escalation) |
| GPT-6 Astra | 53 | Latest Frontier (Closed) |
| Claude Opus 4.8 | 42 | Previous generation Frontier (closed) |
| GLM-5.3-Flash | 42 | Open wait (NVFP4 version used this time) |
*Artificial Analysis Intelligence Index v4.3.2. This is a comprehensive score that combines 10 types of evaluations, including coding and agent-based approaches. For models with inference settings, the maximum setting (max) is used. (As of September 25, 2026)
In a comprehensive score by a third-party organization, GLM-5.3-Flash received the same score as Opus 4.8. However, it still lags behind the latest Opus 5.5 and GPT-6. While it possesses frontier-level performance, it falls short of the most advanced models. This difference forms the basis for the policy, described later, of "leaving difficult tasks to frontier models." In terms of response speed, it achieved results equivalent to or better than the cloud API. In Artificial Analysis measurements, the output speed when using Opus 4.8 with the Anthropic API was approximately 55 tokens/second (confirmed September 15, 2026)[4]. In contrast, this on-premises environment maintained approximately 100 tokens/second per person even with 50 people using it simultaneously (Episode 2). Although this is not a strict comparison due to differences in models and measurement methods, the practical speed is sufficient.
In response to these performance improvements, we decided to set up an in-house environment to conduct testing.
Reasons for not making it cloud-only or on-premises only
There are three main approaches to addressing the challenges discussed in the previous sections: continuing to use only cloud APIs (managed services), switching to an on-premises + open model only, or combining both. The following table summarizes the characteristics of each approach from the perspective of the challenges.
|
perspective |
Cloud API only |
On-premise + open model only |
hybrid |
|
Usage limit |
× The process stops when the limit is reached. |
〇 Can be used without limit |
〇 No upper limit on daily work |
|
cost |
× Charges are based on usage and increase proportionally to usage. |
△ Mainly fixed costs, but initial investment is required. |
○ More advantageous than APIs for scales above a certain level (calculated in Episode 2) |
|
confidential information |
× The risk of external transmission remains. |
○ Data will not leave the company. |
〇 Confidential information is processed on-premises. |
|
Quality of difficult tasks |
〇 The latest Frontier model is available. |
× Does not reach the level of the latest frontiers |
〇 Only the difficult parts are sent to the cloud |
|
Service continuation |
× There is a risk of sudden stops. |
○ Controlled in-house |
○ Continue providing minimal services using only on-premises solutions. |
As this table shows, the challenges discussed in the previous section remain when using only the cloud, and the quality of difficult tasks falls short of the latest Frontier Model when using only on-premises solutions. Therefore, we adopted a hybrid configuration where daily operations are handled on-premises, and only the most difficult tasks are offloaded to the cloud's Frontier Model.
Configuration: Routing between on-premises environment and the cloud
We prepared one GPU server (NVIDIA DGX™ B200) and an open wait model. The GPU is designed to be shared between departments.
This environment has been tested in our AI TRY NOW PROGRAM [5].
The diagram above shows an overview of the department's testing environment. For this test, we are running the NVFP4 version of GLM-5.3-Flash (nvidia/GLM-5.3-Flash-NVFP4) on the DGX B200 and using it as the on-premise model.
In addition, we have created a channel to escalate issues to Claude (Opus 5.5) using NVIDIA NeMo™ Switchyard [6][7] if the on-premises model is insufficient. This approach of offloading only some of the difficult decisions to the frontier model has been shown to be effective in NVIDIA's technical blog [6] and LangChain's verification [8].
Clients can continue to use Claude Code, GitHub Copilot, and Codex as before. Users will hardly need to be aware that the connected model has changed.
Trying out a hybrid inference environment in daily operations: 402 out of 411 requests were completed on-premises.
Before rolling it out to the team, I first tested this environment myself in my daily tasks such as coding and document creation. Most of the requests during this trial were for development tasks such as creating and modifying code and investigating errors.
The image above shows the actual dashboard (Grafana) screen in operation. The left panel shows the number of requests by model, with [mcdr] representing the on-premises GLM-5.3-Flash and [claude-opus-5-5] representing Opus 5.5. Of the 411 requests, 402 were processed by GLM-5.3-Flash, and 9 (approximately 2%) were escalated to Opus 5.5. The right panel shows the breakdown of routing decisions. The 411 requests include auxiliary processing such as conversation title generation. These are sent directly to GLM without going through a decision. 129 requests were subject to decision, of which 120 were sent to GLM as the default action when a decision could not be made (fall_open). 9 requests were escalated to Opus 5.5, representing 7% of those subject to decision. The breakdown is as follows: 3 promotions based on scores indicating work stagnation (dimensions), and 6 instances where Opus was maintained in the turn immediately following a promotion (capable_hold).
At least in my work, there were only a few instances where the Frontier Model was necessary. Escalation wasn't determined by the difficulty of the task, but rather by whether an error requiring escalation occurred during tool execution. Based on these results, we determined that most tasks could be handled on-premises and decided to roll it out to the team.
Regarding the effectiveness of routing, there are also publicly available examples such as the following: LangChain evaluated Switchyard on 145 multi-turn agent tasks and found that the frontier model was needed for only 7% of all turns, with the remaining 93% handled by 30B class models [8]. Costs were reduced by 74%, while accuracy decreased by about 6 points. In such evaluations, it has also been reported that calls to the frontier model, although only 7% of the time, accounted for 68% of the cost [6][8]. This structure, where costs are concentrated on difficult parts with few calls, demonstrates that simply shifting daily tasks to less expensive models can result in significant cost savings. Additionally, an NVIDIA technical blog describes an example where Cognition integrated Switchyard routing into Devin Desktop, reducing average costs by 28% while maintaining nearly equivalent performance compared to when everything was handled by the frontier model [6].
Deployment to the team: Operation by approximately 30 people
We started with software engineers and then expanded it to the entire team (approximately 30 people), including members whose main job is document creation.
|
item |
Achievements |
|
Number of users |
Approximately 30 people |
|
Number of requests |
Approximately 4,300 requests per day (equivalent to approximately 1.027 million requests per year*) |
|
Document creation and brainstorming |
Approximately 30 people (the majority, including non-engineers, use it daily) |
|
Code generation and debugging |
Approximately 10 to 15 people |
*This calculation is based on the average of the past 8 business days (approximately 4,280 requests/day) multiplied by 240 business days per year.
The most common uses are not coding, but rather document creation and brainstorming. Almost everyone, including members who are not software engineers, uses it on a daily basis. Actual usage has confirmed that the AI agent's applications are not limited to software development.
Once approximately 30 people start using it on a daily basis, the next two concerns become whether the response speed can be maintained even when multiple people are using it simultaneously, and whether the investment in this environment can be recouped.
In Episode 2, we will verify these two points using public benchmarks and real-world operational data.
Summary
This article describes the process of building an on-premises environment within our company and deploying it to the team, following the performance improvements of OpenWaite's LLM. First, I tested it in my daily work, and out of 411 requests, 402 (approximately 98%) were processed by the on-premises GLM-5.3-Flash, with only 9 escalations to Opus 5.5. While there are still situations where the Frontier Model is necessary, its use is limited to a small portion. Based on these results, we deployed the on-premises environment to the entire team (approximately 30 people), and it is still in operation today.
In Episode 2, we will quantitatively measure how many agents this server can support using public benchmarks, and then, assuming an initial investment of approximately 100 million yen, we will estimate how many users are needed to break even based on actual operational data.
References
- [1] Z.ai, “zai-org/GLM-5.3-Flash” (Hugging Face Model Card)
- [2] NVIDIA, “nvidia/GLM-5.3-Flash-NVFP4” (Hugging Face model card)
- [3] Artificial Analysis, “LLM Leaderboard — Comparison of AI Models”(Intelligence Index v4.3.2)
- [4] Artificial Analysis, “Claude Opus 4.8: API Provider Performance Benchmarking & Price Analysis”
- [5] Macnica launches "AI TRY NOW PROGRAM" to provide AI agents and a physical AI verification environment using the "NVIDIA DGX B200 system" (September 3, 2025)
- [6] NVIDIA Technical Blog, “Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard” (August 11, 2026)
- [7] NVIDIA-NeMo, “Switchyard” (GitHub)
- [8] LangChain, “How many of your agent’s calls actually need a frontier model?”