Generative AI App Governance Assessment

Risks in providing generative AI applications and services

Main risks

The development and use of AI is creating new types of risks on an unprecedented scale.
As shown below in the Ministry of Economy, Trade and Industry's "AI Business Guidelines (Version 1.1) Appendix (Summary)," there are a great many risks that must be taken into consideration.

Ministry of Economy, Trade and Industry "AI Business Guidelines (Version 1.1) Annex (Summary)"
https://www.meti.go.jp/shingikai/mono_info_service/ai_shakai_jisso/pdf/20250328_4.pdf

In light of these, AI developers and providers need to address both technical and social risks.
Among these, technical risks can be broadly divided into two categories: quality-related safety risks and security-related risks.

Safety-related risks and challenges

Incorrect output due to hallucination, etc.

Generative AI can output information that is plausible but not factual, and if used in customer interactions or decision-making, it could lead to a loss of corporate credibility and the need for compensation.

Biased output, discriminatory output, inconsistent output, etc.

If generative AI outputs inappropriate or discriminatory content, it could have a negative impact on a company's brand image and social responsibility. In particular, bias regarding gender, race, religion, etc. could lead to corporate compliance violations.

Changes in output during the upgrade process

Even if a safety / quality evaluation is conducted once, the output or operation may be affected in the following cases during long-term operation of the service.

・When the model you are using is updated

- When you add a new item to the database (assuming your application uses RAG)

Security risks and challenges

Attacks on AI systems, such as data contamination attacks

Attacks that exploit vulnerabilities in AI models (e.g., prompt injection, model extraction, meta-prompt tampering, etc.) could result in unintended output or falsification of information. There are also concerns about the use of AI to spread fraud and false information.

Information leaks and confidential data leaks

Inputting internal documents or personal information into generative AI can lead to unintentional information leaks. Cloud-based AI services, in particular, often have unclear mechanisms for re-training and saving data, putting a company's intellectual property and customer information at risk.

Unauthorized connection and use of external AI applications

When employees connect to AI-generated applications outside of their work, there is a high risk of internal information leaking to the outside. Shadow IT use in particular violates corporate security policies.

Generative AI application governance diagnostic service

Service contents

Taking the risks mentioned above into consideration, Macnica will conduct AI model testing and support customers in establishing governance systems.
We will provide you with the following report. * This is only a sample of a portion of the entire report.

Service flow

Safety Tests

We use Weights & Biases'Weave to log input and output to the application and provide advanced visualization of the evaluation results.
Then, Macnica technical members will carry out a third-party evaluation, analyze the results, and offer suggestions for improving the application.

Depending on the customer's request and situation, we can divide up the roles and carry out the tests.

Security Testing

We utilize the Red Teaming services of our partner companies to conduct application penetration tests using the partner's professional team.
Shadow IT detection is also available upon request.

Macnica is at the forefront of change, exploring the skills and knowledge that lie beyond the cutting edge, and we collaborate with partner companies around the world that have a proven track record.
A distinctive feature of this service is that it supports the implementation of AI risk assessments, model testing, and the establishment of governance systems based on cutting-edge overseas knowledge, while working directly with engineers at each manufacturer's overseas headquarters as needed.