Introduction

With the recent evolution of technology, generative AI has been attracting attention as a means of improving business efficiency and providing innovative solutions. In particular, the use of large language models (LLMs) has been adopted by many companies due to their infinite possibilities. However, many companies face several hurdles when introducing and implementing generative AI. These are not just technical challenges, but also related to business application challenges and organizational responses.

In this article, we will explain tips and specific methods for companies to properly introduce and make the most of generative AI, based on the Company own "failure cases." In particular, we will introduce common failure patterns and their solutions, based on the use of LLM in customer inquiry desks.

Failure Cases

When introducing generative AI, many companies face three common challenges:

• Excessive expectations and inappropriate job definition
• Integration with existing systems and poor quality
• Lack of collaboration within the organization

For example, the Company tried to use generative AI as a chatbot for customer support, but the actual usage rate was low, and as a result, we failed to achieve the expected results. The reasons for this include the aforementioned excessive expectations, a lack of proper task definition, and problems with the quality of the system itself.

Furthermore, after implementation, the accuracy of data searches was low and the AI generator was unable to output the answers that users expected, which led to a vicious cycle of poor user reputation and reduced usage.

Efforts towards LLM implementation

Detailed definition of the application

The most important step in implementing generative AI is to define the business process in detail, including identifying the inputs and outputs required at each step of the business process and planning how to translate them into LLM.

For example, in customer support operations, the process from confirming the inquiry content to providing a final response will be broken down into smaller steps, and the role that the generative AI will play at each stage will be clearly defined.

System improvements

The next important thing is to process the company's data so that it is easy to process with the LLM/RAG structure. To improve the quality of the data stored in the vector DB, we will unify the data source and organize the text structure. Furthermore, we will improve the UI/UX from the user's perspective to improve the accuracy of the answers output by the generation AI.

For example, you can improve the accuracy of responses by allowing users to select the question that most closely matches their answer.

Strengthening the organizational structure

And we will strengthen the collaboration system within the organization. It is important that the system development team and the work site work together as one and share information with each other while moving forward with the project. For this purpose, we will form a project team (e.g., chatbot promotion team) and hold regular meetings to check the progress and share and solve problems.

Measurement and feedback

It is also essential to measure the effectiveness after implementation. Specific KPIs are set and results are evaluated regularly. In addition to a simple numerical evaluation, feedback from the field is collected and reflected in improvements to the system and operational processes.

Summary

The introduction of generative AI (especially LLM) has common challenges that many companies face. However, with proper planning and cooperation within the organization, you can maximize its effectiveness. In this article, we will explain in detail the common mistakes that people make when introducing LLM and how to solve them.

We have presented an approach that will lead to success through detailed definition of applicable tasks, system improvements, strengthening of organizational structure, and continuous effectiveness measurement and feedback plan. By following these steps, generative AI implementation projects will achieve great results. We hope this will be helpful for improving business efficiency and providing innovative solutions.

Macnica, Inc.
Data & Application Division, 1st Technology Department, 1st Section
Kyoichi Sugimoto
Joined Macnica in 2021. He is in charge of technical support and customer training for integrated log management platform products. He currently coordinates the entire support team while also promoting customer and internal support work reform activities using generative AI.