Manage internet traffic with gateway equipment

MITSUBISHI ELECTRIC Corporation (hereinafter referred to as MITSUBISHI ELECTRIC Corporation) has cultivated a wide range of technological assets such as control technology and power electronics in its nearly 100-year history. We have strengthened our business base, which has continued to improve, and have increased our solid reliability. Currently, we have positioned eight fields of power systems, transportation systems, building systems, factory automation systems, automotive equipment, space systems, power devices, and air-conditioning systems as growth drivers. We are expanding our business globally to emerging markets.

The company uses the machine data analysis platform "Splunk Enterprise" (hereinafter referred to as Splunk) provided by Macnica to extract logs of internal email sending and receiving, web access, etc. from various systems and make it possible to perform cross-sectional searches. In addition to building a "Log Search System," we also took on the challenge of developing a "Network Traffic Visualization System," believing that it could be applied to visualizing network application traffic volume during the construction process. In this case study, we will introduce the process of implementing both of these areas, which Splunk excels at.

まず、ログ検索システムが必要になった背景について、両プロジェクトを主導した三菱電機 IT戦略室 システム基盤部 ネットワークシステムグループマネージャーの柳瀬賢治氏は次のように語る。「私たちIT戦略室では、三菱電機インフォメーションネットワーク株式会社（MI ND）の協力を得ながら、全社のインターネットトラフィックをまとめるゲートウェイ設備の管理・運用をしています。従来は、このゲートウェイ設備のログを一切公開していなかったので、メール送受信やWebアクセスに関する各拠点からの問い合わせに関してはIT戦略室とMINDが全て引き受け、調べた上で回答するという運用になっていました。しかし、近年はメール送受信やWebアクセスに関する問い合わせが急増する傾向にあり、対応業務が大きな負担となるほか、集中する時などには回答時間も長くなるなどの課題を抱えていました」

Aiming to reduce the workload of the IT strategy office by releasing the log search function to the IT department of the base

MITSUBISHI ELECTRIC Corporation MITSUBISHI ELECTRIC Corporation approximately 40 bases in Japan, including manufacturing plants, research laboratories, technology centers, and development centers. It extends to bases/company. Each base and affiliated company has its own IT department, but the causes of problems such as “emails not being delivered” and “web access not being possible” are recorded in the log of the gateway equipment, so the IT Strategy Office and MIND was flooded with log investigation requests.

Mr. Yanase says, ``We didn't use a dedicated product to search logs, and instead used an open-source environment to archive text and search with our own script, but the size of the log gradually increased. It is true that it became difficult to respond with a script as the investigation requirements became bloated and complicated."

In addition, Mr. Toshiya Seyama, who is in charge of network system group, IT Strategy Office, System Infrastructure Department, MITSUBISHI ELECTRIC Corporation, which is actually responsible for log management of gateway equipment, points out that the content of inquiries has also changed. “In the past, we received many pinpoint inquiries, such as the month and day on which this email did not arrive, but recently, we have received long-term inquiries such as which employees have accessed this suspicious site in the past six months. Investigation requests for logs have increased.The content of the log investigation is not uniform, and the investigation procedure has not been established or defined, so if the person in charge changes, the procedure will change, and the investigation time will vary. We needed some way to enable stable operations.”

In the event of an emergency or an inquiry from an overseas base, it would sometimes be late at night, and how to respond to that was also a problem.

Therefore, the IT Strategy Office aimed to reduce the workload and improve services by appropriately disclosing the logs of the gateway equipment to the IT departments of each base/affiliated company. To that end, it is essential to introduce a system that can control viewing privileges so that the range of logs that can be viewed at each site can be limited to the scope of work at each site. It was also necessary to prevent it from being done, and it was also necessary to leave a log of what kind of operation was being done.

The number of inquiries has decreased significantly and the governance of viewing authority has been secured.

Mr. Seyama was researching various log management software and learned about Splunk when he visited Macnica booth at InteropTokyo 2013 in June 2013. "Splunk met the selection criteria because it not only allows cross-searching of various logs, but also allows controlling log viewing privileges and visualizing operation logs. Also, the response is very fast, and the graphical and unified I was also impressed by the user-friendly interface.While it provides an easy-to-use screen for beginners, I was also attracted by the depth of the search command that allows users to dig deeper if they want to find out more information.'' (Mr. Seyama)

There are some differences in the skills of the IT departments of each base/affiliated company. Not everyone can type commands and search logs. The condition was to provide an interface that allows the necessary searches without any skill. The IT Strategy Office used the 60-day free evaluation version download service, and as a result of testing various usability, decided to start building a log search system using Splunk.

In the log search system, logs from mail servers, proxy servers, firewalls, etc. were imported into Splunk and made search targets. However, logs of email sending and receiving have multiple lines per event, and the location cannot be identified simply by an email address. Therefore, we used Splunk's construction support service to set up and automate lookups (a function that refers to an external file to capture the field values needed for a search), a function that combines multiple lines into a single event to determine authorization, and response tuning, all with the help of Macnica," explains Seyama.

Operation of the log search system started in September 2014, and the IT departments of each base/affiliated company also started using it. After the start of operation, the number of inquiries to the IT Strategy Office and MIND has decreased significantly, and the nighttime response has almost disappeared. Since the log search system can be used without special skills, it has become possible to obtain results quickly. Governance is firmly secured.

Mr. Seyama said, "In the past, it took a lot of time and effort for the IT Strategy Office to investigate the logs of the gateway equipment on the same day. We are now able to respond immediately,” he evaluates.

Determined that visualization of network traffic could be achieved with Splunk

After the log search system went live, the IT Strategy Office was faced with another issue. It was network traffic analysis. Regarding the network traffic situation, even if we can grasp the total traffic volume of the head office and each base/affiliated company, we cannot grasp the breakdown in detail, such as how much mail traffic is in it, how much traffic is in the internal business system, etc. was. For this reason, it was difficult to estimate the impact on the network if, for example, the scale of use of a system within the company was doubled.

According to Mr. Seyama, the goal was to effectively utilize multiple line services. We were looking for ways to create an efficient network by assigning priorities to the network. It was necessary to visualize the breakdown of traffic, but in order to understand the traffic situation, it was necessary to capture the network traffic of each base and then investigate it separately, which was both laborious and time-consuming. I had half given up,” he confesses.

Mr. Yanase also said, "Although it is necessary to give priority to system traffic that employees use for their daily work, we can tolerate a slight drop in line quality for batch system traffic that occurs at night. The goal was to maximize the effective use of bandwidth while suppressing noise.”

Seyama visited the "Splunk Worldwide User Conference" (.conf2014) held in Las Vegas in October 2014 to find ideas for solving SDN-related issues. At the partner booth, Netflow Logic released the idea of visualizing traffic details based on network device flow information in the form of Splunk Apps. brought back the information. “As a result of our internal discussion, we realized that the original purpose of effectively utilizing multiple line services through SDN was unfortunately not so easy, and that it was premature. I decided that it was possible, and decided to turn it into a project,” Yanase confesses.

In addition, Mr. Seyama said that it was difficult to find the optimum sampling amount when capturing flow information. “You can collect complete information by getting 1/1 flow information from the router, but that would put a lot of load on the router and consume traffic, so you should carefully tune the sampling interval to 1/10 or 1/100. I went

Significantly reduces the burden of investigation man-hours and greatly shortens the time to solve problems

In November 2015, the development of the "network traffic visualization system" was completed. Initially, we had specific departments try it out and fine-tuned it, and during that time we also reviewed the dashboard and released it to the entire company in April 2016.

Up until now, when we received an inquiry from an overseas office that it was slow even though we had secured bandwidth for a specific system, we spent a lot of time investigating whether the bandwidth limit had been reached or whether the system itself was slow. However, now it is possible to easily check the availability of secured bandwidth even from overseas bases. If the system being used is slow, even though there is free bandwidth, it becomes immediately clear that the problem is not on the network side, but on the system itself. “In the past, the IT Strategy Office investigated in detail, and there were times when we were unable to respond in a timely manner due to the time difference, but now the IT departments at the bases are able to investigate on their own, so it takes only a few hours to respond. We were able to grasp the cause of the problem within a short period of time, which greatly reduced the workload of the IT Strategy Office and greatly shortened the time required to solve the problem. It seems that the stress in the IT department has also been considerably reduced." (Mr. Yanase)

In addition, for visualization of network traffic, "Splunk Add-on for NetFlow" (a template for normalization during data collection) is used. “If we didn’t have this, we would have had to combine open-source software, convert the binary information of the flow information into text through trial and error, and input it to Splunk. We were able to easily integrate it with Splunk,” says Mr. Seyama.

In the future, the IT Strategy Office plans to expand the target range of the network traffic visualization system to overseas base lines. Since overseas line costs are high, visualization, analysis, and improvement are highly effective.

Looking back on this project, Mr. Seyama said, ``Based on my experience of building two systems using Macnica, I am confident that I made the right choice. It has been very helpful to me, and I look forward to continued support in the future."

Mr. Yanase also said, ``Splunk's strength is that it can import any kind of log, but its versatility also increases complexity, so Macnica proposed a concrete means of realizing it.'' It was a great help. The project is not over, and different requirements may arise, so I look forward to continued flexible proposals."

As Splunk's domestic distributor, Macnica will continue to support MITSUBISHI ELECTRIC Corporation 's use of Splunk with all its might.

User Profile

045-476-2010
月～金 8：45～17：30

inquiry
Please feel free to contact us

Free seminar
Click here for various seminars

Document download
Click here for various materials