Popular Terms

Let’s look closely at the contents of the accepted papers in the conference. Using the titles of the accepted papers in the conference, we generated a word cloud and top keywords figure below.

Word cloud and top keywords in NeurIPS 2020 papers

Note that in creating the figures above, we removed stop words and frequently used machine learning terms like “neural”, “learning”, “network”, “algorithm”, etc.

Last year, the popular terms were “Optimization”, “Reinforcement”, “Efficient”, “Bayesian” and “Graph” See our explanation of the popular terms here. Most of the top keywords from last year, are still popular in this year’s conference.  This year, the popular terms are “Graph”, “Optimization”, “Reinforcement”, “Adversarial”, and “Robust”.

Let’s discuss few popular terms from this year’s conference. For more information about the last year’s popular terms, visit this link here.

Adversarial

In recent years, ML and AI in general have made great strides in various fields and industries. Many ML models are getting better compared to human performance in terms of speed and accuracy. However, these state-of-the-art models still have their weaknesses. A model with high accuracy can easily be fooled by adversarial attacks. From its root word, “adversary”, adversarial describes opposition or conflict between different parties. Adversarial attacks are made to cause models to not work properly.

Let’s look at the image below:

The images on the left are predicted labels made by an image classification model. As you can see, the labels are correct, and the model is confident in its predictions. The dark images in the middle are perturbations or adversarial attacks that wants to confuse the model. When the perturbations are added to the original images (images on the left), the model predicted very wrong labels. We humans would still be able to identify the images correctly, but the model was fooled when a generated noise was added to the image.

This is just an example of an adversarial attack. Some models are specifically trained to fool other models. Adversarial attacks can be used in other AI use cases like face recognition, spam filtering, and more. While this may show weaknesses in current models, adversarial attacks can be exploited to create more robust models.

Robust

As discussed above, modern ML models still have their weaknesses. High accuracy models can still fail when given out-of-distribution inputs. Robustness is the ability of a model to correctly handle different patterns of input data, attacks, and drastic data changes.

One major problem in creating a robust model is defending against adversarial attacks that are specifically made to fool the model. One way to handle this is to learn from these attacks. Recently, there’s been a great interest in making models robust against attacks and unseen data changes.

Word Co-occurrence

For a deeper analysis on the submitted paper, we generated a co-occurrence network of important terms extracted from the abstract of the accepted papers. This visualization reveals keywords that are usually used together within a document. By analyzing a co-occurrence network, we’ll be able to see general overview of keyword clusters, which keywords are related to each other, and which keywords were more often used.

VOSviewer was used to generate this figure. Abstract text of accepted papers in NeurIPS 2020 was given as input. Clustering and natural language processing techniques were automatically applied.

The size of a circle and word shows how frequently the words were used. The bigger they are, the more frequent it was used. The connections between the circles mean that these words co-occur on the same document. The colors are the computed clusters of keywords.

From the co-occurrence figure, we can observe 4 main clusters. The following are our observations in the different clusters.

In the green cluster, the keywords are related to computer vision and natural language processing. General terms like “image”, “text”, “language”, “object” shown to be to most used keywords in the green cluster. However, popular methods like “BERT” and “GANs” also occurred frequently.

The blue cluster mostly features terms about graph, adversarial, and general deep learning terms. Terms such as robustness, defense, attack are related to adversarial learning. Then, terms like “GNN”, “flow”, “classifier”, “layer”, “DNN” are related to graph and deep learning.

The most obvious cluster is the yellow cluster. Most of the terms are about reinforcement learning. The circles for reinforcement learning related terms like “agent”, “action”, and “reward” are big, which means there is great interest in reinforcement learning.

Lastly, the red cluster contains terms about the design of ML and neural network design. “function space”, “rate”, “iteration”, “SGD”, “Adam”, and “ReLU” are related to NN design. These words are usually used when explaining about model design or architecture.

論文を活用した事例はこちら

アイシン・エィ・ダブリュ株式会社様事例

アイシン・エィ・ダブリュ工業株式会社様事例