Introduction

When using generative AI in financial operations, the key is not simply choosing a high-performance model, but rather translating it into a "comprehensible format for your own company," including how to handle data and the evaluation criteria. This article introduces an implementation loop for fine-tuning a Japanese model that can be handled in a local environment to suit financial tasks and continuously evaluating it with a fixed benchmark.

Specifically, we used "NVIDIA Nemotron-Nano-9B-v2-Japanese," a large-scale language model (LLM) trained from scratch by NVIDIA, designed as an integrated model that handles both inference and non-inference tasks and optimized specifically for Japanese. We performed fine-tuning on LoRA using JaFIn (Japanese Financial Instruction Dataset) and verified how far practical performance could be improved even with a small-scale model through evaluation using japanese-lm-fin-harness. As a result, we achieved a level equivalent to GPT-5 on JaFIn. This demonstrates that domain-specific SLMs can be competitive in practice if the evaluation design and data are appropriately narrowed.
This article summarizes the key design, implementation, and evaluation points that led to these results, and explains a concrete approach to "build small, measure, and improve" a Japanese LLM for the financial sector.

1. Why fine-tune Japanese LLMs for the financial sector?

The purpose of using a Japanese LLM in the financial sector is to adapt general conversational skills to the vocabulary, institutional knowledge, response formats, and evaluation criteria required in the job.

In internal FAQs, audit and accounting, tax, IR, credit, securities sales, and compliance, quality depends not only on natural Japanese but also on the use of system names, account titles, multiple-choice formats, supporting evidence, and handling of ambiguity. For financial models, it is important not only to be able to answer fluently but also to be able to provide consistent, verifiable answers in a business-like format.

The key point here is not only improving the accuracy of the model, but also being able to control the AI improvement cycle in-house. For financial institutions, customer information, internal regulations, audit documents, sales history, and risk assessment logs are directly linked to both competitiveness and accountability. While leveraging the high performance of external APIs is a valid option, not all financial data can be shared externally. Having a model that learns, evaluates, selects checkpoints, and logs within the company's own environment makes it easier to manage accuracy and data sovereignty within the same design.

The value of fine-tuning lies in its ability to build up the necessary level of accuracy in a manageable way. By tracking which data improved, which evaluation tasks saw progress, and which checkpoints should be adopted, model selection transforms from a subjective comparison into a reproducible improvement process. In financial AI, this reproducibility directly translates to auditability, operational quality, and ease of internal justification.

The implementation strategy described in this article is a minimal loop that uses a 9B-class Japanese model as a base, performs supervised fine-tuning with JaFin, and evaluates it on the fixed task of japanese-lm-fin-harness. By adopting LoRA, the burden of GPU memory, training time, and checkpoint management can be reduced, allowing us to focus on searching for target_modules, rank, learning rate, and checkpoints.

2. Technical configuration: model / dataset / benchmark / adapter

The minimum configuration consists of four layers: a base model, SFT data, a financial benchmark, and a LoRA adapter. While vLLM, NVIDIA TensorRT-LLM, and Transformers are potential inference platforms, a combination of Hugging Face Transformers, PEFT, and datasets is suitable for the initial training implementation. This allows for relatively easy management of tasks such as adapter swapping, merged export, and token length audits.

3. About NVIDIA Nemotron-Nano-9B-v2-Japanese

NVIDIA Nemotron-Nano-9B-v2-Japanese is a 9B-class reasoning model trained from scratch by NVIDIA and optimized specifically for the Japanese language. It supports context lengths up to 128K tokens. For local validation, it makes it easy to start experimental design even with a single GPU, and in environments with ample GPU memory, it can be expanded to long-text processing, batch inference, and comparative validation of merged checkpoints.

The architecture is a hybrid configuration combining Mamba-2, MLP, and a small number of attention layers. This is also important when selecting LoRA 's target_modules. For a standard Transformer, q_proj, k_proj, v_proj, and o_proj are the initial candidates, but for Nemotron-Hybrid, Mamba-based in_proj/out_proj and MLP-based up_proj/down_proj are also considered.

Furthermore, careful handling of reasoning traces is necessary. When using enable_thinking, ensure that the chat template and generation settings are consistent across training, evaluation, and inference. Including thinking only during training and removing it during evaluation can easily lead to discrepancies between the output format and the extracted answers. In financial benchmarks, many tasks require selecting the correct answer or providing a short explanation, so both reasoning ability and answer format should be checked in the logs.

4. Use JaFIn as supervised fine-tuning data.

JaFIn is a SFT (Software-Focused Format) dataset of Japanese financial Q&A in instruction/input/output format. It includes explanatory questions and answers frequently encountered in financial operations, covering topics such as public finance, government bonds, local government bonds, taxation, and financial systems. With a relatively small dataset of 1500 Q &A pairs, it is more suitable for modeling financial vocabulary, structuring answers, and aligning institutional explanations with existing models, rather than for massive knowledge injection.

In SFT, the instruction and input are combined into user content, and the output is used as the assistant response, serving as the training signal. Labels are applied only to the assistant-side tokens, while the prompt side is masked with -100. This ensures the model uses gradients in the direction of generating financial responses.

​Token length audits are effective for training with small-scale financial data. If some outputs are too long, the loss and GPU memory become unstable, making it difficult to directly improve short, selective benchmarks. Therefore, we standardize max_length, max_assistant_tokens, truncation rules, and the handling of empty inputs, and leave a dataset fingerprint and token statistics for each run to ensure reproducibility.

def build_messages(example):
    user_text = example["instruction"]
    if example.get("input"):
        user_text += " " + example["input"]
    return [
        {"role": "user", "content": user_text},
        {"role": "assistant", "content": example["output"]},
    ]

5. Financial benchmarks using japanese-lm-fin-harness

japanese-lm-fin-harness is an evaluation harness for comparing LLMs in the Japanese financial domain. This validation uses five benchmarks: chabsa, cma_basics, cpa_audit, fp2, and security_sales_1. By measuring the base model and LoRA model with the same task list, harness revision, and prompt convention, improvements can be compared for each experiment. In evaluating finance-specific LLMs, looking at only a single QA task makes it difficult to determine whether the model can generate "financially sensible text," whether it possesses the knowledge necessary for practical judgment, and whether it can consistently handle classification, reading comprehension, and multiple-choice questions. Therefore, evaluation is conducted across multiple tasks spanning financial text comprehension, management accounting, auditing, financial planning, and securities sales.

MODEL_PATH=$ROOT_DIR/outputs/iter16_attn_mamba_safe_ep2_lr5e-5_enable_thinking_ckpt100/merged
TASKS='chabsa-1.0-0.2,cma_basics-1.0-0.2.1,cpa_audit-1.0-0.2.1,fp2-1.0-0.2.1,security_sales_1-1.0-0.2.1'
RUN_NAME=iter16_attn_mamba_safe_ep2_lr5e-5_enable_thinking_ckpt100 BENCH_MODE=merged MODEL_PATH=$MODEL_PATH TASKS=$TASKS bash scripts/run_fin_benchmark.sh

6. Implementation points for LoRA fine-tuning

LoRA is a parameter-efficient fine-tuning method that adds low-rank matrix updates to linear layers while keeping the weights of a pre-trained model fixed. Key design variables include rank, alpha, dropout, target_modules, bias, quantization, and merge policy. In small-scale SFTs such as financial QA, narrowing the learnable parameters and comparing checkpoints more frequently tends to yield more stable evaluations.

RUN_NAME=scan01_attn_mamba_safe_ep2_lr5e-5_enable_thinking_steps20 
TARGET_PRESET=attn_mamba_safe FORCE_TORCH_MAMBA=0 NUM_EPOCHS=2 LEARNING_RATE=5e-5 
bash scripts/run_train_jafin.sh   --enable_thinking   --save_strategy steps   --eval_strategy steps   --save_steps 20   --eval_steps 20

7. How to interpret experimental results

In this version 5, the LoRA best checkpoint improved to 0.6718 compared to the base model's score_primary of 0.5747. The gain is +0.0971. When the GPT-5 Chat API is placed on the same benchmark contract, its score_primary is 0.6683, meaning this LoRA best is very close to the GPT-5 Chat level, and slightly surpasses it in primary score. The checkpoint used was iter16_attn_mamba_safe_ep2_lr5e-5_enable_thinking_ckpt100, with settings attn_mamba_safe, 2 epochs, learning rate 5e-5, and enable_thinking.

The main factors contributing to the score improvement were the growth in CMA Basics, FP2, and Security Sales. CHABSA was already high at the base level, leaving little room for further improvement. CPA Audit remained flat, and for audit-related multiple-choice questions, improvement may be limited if SFT, which focuses on explanatory QA, is the only method available.

​API models such as GPT-5 Chat are evaluated via response text matching, while local HF models are evaluated via likelihood scoring. Therefore, the result 0.6718 > 0.6683 should be considered a technical signal under these fixed benchmark conditions.

A broader ranking, including general chat capabilities, will be examined in future additional evaluations. Under the conditions limited to this financial task, local 9B + LoRA achieved a rating comparable to that of cutting-edge LLMs.

8. Points to note for practical implementation

- Data contamination: Check for overlapping paths in JaFIn, additional SFT data, few-shot examples, and evaluation data. Maintain stricter contamination checks for financial tasks.
- Evaluation benchmark: Fix the task list, harness revision, prompt template, number of few shots, answer extraction, seed, and checkpoint path in the run manifest.
• Output control: Financial QA should adhere to a "short conclusion, evidence, and points to note" format. RAG/Agent preprocessing should prioritize formats that allow downstream users to read the data, such as JSON or bullet points.
• Checkpoint selection: In small-scale SFTs, intermediate checkpoints often yield the best results. Save the best checkpoint, best eval loss, and best benchmark separately.
- Mamba fast path: Logs whether the module to which LoRA is applied is used in the actual forward path and whether it is receiving gradients.
• Deployment: Adapter operation is easy to swap out, and merged model operation is easy to serve. For internal production, add licensing, audit logs, RAG reference source display, and output verification.

Option: Use AutoResearch

AutoResearch is a workflow proposed by Andrej Karpathy, former AI director at Tesla and co-founder of OpenAI, that allows an AI agent to autonomously run small learning experiments. While this article focused on the four components of model, dataset, benchmark, and adapter, it's also possible to organize what we've done so far into short iterative loops of fine-tuning and benchmarking, and leverage the power of an AI agent to continuously explore parameters autonomously.

To begin experiment management, first define the save format for the mission, benchmark contract, acceptance rule, and run manifest in program.md. For each run, keep metrics, token_length_stats, sampling_stats, benchmark result, and generated sanity check, and manage candidates for the next exploration using keep/discard. In evaluation, in addition to the final score, it is important to check which tasks improved and which types of failures remained.

Summary

This time, we tried an approach that simultaneously improves accuracy, data management, and auditability with a small, localized model focused on the financial sector. Our best checkpoint achieved a score of 0.6718, which is on par with, and slightly surpasses, the GPT-5 Chat API (0.6683). In areas where the scope of work can be clearly defined, such as financial QA, auditing and accounting, and securities sales, domain-specific SLMs can deliver performance comparable to frontier LLMs.

In practice, what's crucial is not just the output quality of the model, but also the ability to manage the improvement process. Which data to use, which evaluations to make decisions based on, which checkpoints to adopt, and which logs to keep—if this entire process can be handled locally, it allows for the accumulation of evaluation results, predicting costs and latency, and updating the model to match business needs, all while keeping confidential data within the company. For financial institutions, this manageability is just as important as accuracy itself.

​Fine-tuning Japanese financial SLMs using LoRA is a practical way to start this approach on a small scale. There's no need to build a large-scale foundational model from the beginning. You can start with a 9B-class model, gather financial data, fixed benchmarks, adapter exploration, and evaluation logs, and build an improvement cycle that closely resembles real-world practice.

The next step is to narrow down the target business to one, fix the evaluation set, and continuously measure the difference between the base model and the LoRA model. Starting with specific business processes such as audit FAQs, securities sales QA, and financial document classification makes it easier to verify the balance between accuracy, cost, and data sovereignty. In the future, it may be possible to incorporate such local models into parts of RAG or Agent to improve the quality of internal knowledge retrieval, inquiry classification, and answer generation.

Reference material

• NVIDIA Nemotron-Nano-9B-v2-Japanese
• Sakaji-Lab/JaFIn
• pfnet-research/japanese-lm-fin-harness
• LoRA: Low-Rank Adaptation of Large Language Models
• Hugging Face PEFT LoRA documentation
• karpathy/autoresearch
• guchengwei / autoresearch-nemotron-jp-finance-lora

Related page