Data Collection Challenges

The first challenge companies face is collecting the necessary data. Data sources are becoming more diverse, including some that are difficult to collect, such as physical data and human body data. This means that traditional data collection methods are increasingly unable to handle the demand. Specific challenges include:

• Data diversification: Not only IT assets but also physical data such as factory equipment and human bodies are increasing, making collection methods more complex.
• Limited network connectivity: Data sources that are not connected to the internet are difficult to collect.

Data Analysis Challenges

While increasing amounts of data may seem like a good thing, in practice it brings many problems, the main challenges being:

• Processing load: The load on the analysis platform increases as large amounts of data are processed, causing performance to degrade.
• Data quality: The presence of noisy or poor quality data reduces the accuracy of analysis results.

To overcome these challenges, we need to rethink how we collect and analyze data.

When considering the use of AI, learning from poor-quality information or information that contains noise can reduce the quality of the answers given, and processing large amounts of data can take time, potentially compromising real-time performance.

Data Collection at the Edge

Traditionally, data has been collected primarily in central analytics platforms.

However, by providing data collection functionality to the edge, it becomes possible to efficiently collect data that was previously very difficult to collect. Specific solution examples include the following:

• IoT Gateway: Collects data from sensors and cameras and sends it to an analysis platform via the Internet.
• Wearable devices: Collect physical data using devices such as smartwatches.

For example, Macnica is solving data collection issues by using a solution called Splunk Edge Hub. This solution is palm-sized and easy to install, and supports multiple protocols, especially OT protocols, which are considered difficult to collect.

Data Analytics at the Edge

It is also very important to perform data analysis at the edge. By eliminating unnecessary data and performing preprocessing such as labeling before sending the data to the central analysis platform, it is expected that the accuracy of analysis will improve and the processing load will be reduced.

For example, a solution called Cribl optimizes the analysis process by preprocessing data collected from data sources at the edge and providing high-quality data to the analysis platform.