About the VAST InsightEngine Kafka functionality

Kafka is a distributed messaging queue for processing streaming data. Kafka temporarily stores and manages real-time generated streaming data and its metadata. Data stored in Kafka is used by external systems such as data analytics platforms and web services.
The system is similar to a conveyor belt in a factory, managing the data that flows through it and linking it to external systems.
Kafka is becoming an essential feature for data pipelines that require real-time processing.
Kafka consists of three elements: a Producer that notifies when an event occurs, a Broker that stores the notified messages, and a Consumer that confirms receipt of the messages. Within the Broker, messages sent from the Producer are stored and managed in containers called topics.

Normally, using Kafka requires a dedicated server, but the server configuration had to be changed depending on the amount of stream data and the required throughput.
In contrast, VAST InsightEngine includes Kafka as a basic function, and when building a RAG pipeline using VAST Data, you can build the pipeline without preparing a dedicated server.

About VAST InsightEngine's Vector DB Functionality

A vector database, as the name suggests, is a database that stores vectors. Before introducing vector databases, let's first explain what a vector is.
The texts and images we see every day are in a format that humans can understand, but large-scale language models (LLMs) cannot directly understand them. Therefore, it is necessary to convert human-understandable data into a format that LLMs can understand. The format after such conversion is called a vector. A vector is a collection of floating-point numbers, and distances can be calculated between vectors. The closer the vectors are, the more similar their properties are.
A vector database stores vectors internally and searches its own vector data for the vector closest to a given vector. In other words, using a vector database makes it possible to extract vectors with similar properties. Thus, this vector database functionality is used in RAG to search for the closest answer (vector) to a question, and it is becoming an essential technology in modern AI applications.
Traditionally, vector databases, like Kafka, are built using dedicated servers and open-source software such as Milvus.
In contrast, VAST InsightEngine includes vector database functionality as a basic feature, making it possible to build a vector database without preparing a dedicated server.