Data augmentation is a technique with the potential to improve model generalization performance. Automatic Augmentation, which autonomously designs suitable data augmentation strategies, has significantly improved performance. One of the effective Automatic Augmentation is RandAugment (RA), which randomly selects methods to apply from the set of available augmentations. However, its selection with equal probabilities for all available augmentations is not ideal. In this study, we propose Affinity-Weighted RandAugment which assigns weights to the selection probabilities for the available augmentations in RA. The weights are calculated based on the value of Affinity, which represents the distributional similarity between original and augmented data. Affinity provides the prediction of the effect of each augmentation according to both the model and the dataset in advance. This enables the assignment of weights to the selection probabilities with low computational cost. As an additional effort, whether Affinity could be obtained at a much lower computational cost was considered. Based on these insights we implemented Single-Pass Affinity-Weighted RA, which can achieve effective weighting for RA without additional or separate learning. Experiments in image classification using CIFAR-10/100 and Describable Textures Dataset (DTD) demonstrated that Affinity-Weighted RandAugment achieves better test accuracy in comparison with the other known RA methods.

Gradient regularization (GR) is a method that penalizes the gradient norm of the training loss during training. While some studies have reported that GR can improve generalization performance, little attention has been paid to it from the algorithmic perspective, that is, the algorithms of GR that efficiently improve the performance. In this study, we first reveal that a specific finite-difference computation, composed of both gradient ascent and descent steps, reduces the computational cost of GR. Next, we show that the finite-difference computation also works better in the sense of generalization performance. We theoretically analyze a solvable model, a diagonal linear network, and clarify that GR has a desirable implicit bias to so-called rich regime and finite-difference computation strengthens this bias. Furthermore, finite-difference GR is closely related to some other algorithms based on iterative ascent and descent steps for exploring flat minima. In particular, we reveal that the flooding method can perform finite-difference GR in an implicit way. Thus, this work broadens our understanding of GR for both practice and theory.

Data analysis education plays an important role in accelerating the efficient use of data analysis technologies in various domains. Not only the knowledge of statistics and machine learning, but also practical skills of deploying machine learning and data analysis techniques, are required for conducting data analysis projects in the real world. Data analysis competitions, such as Kaggle, have been considered as an efficient system for learning such skills by addressing real data analysis problems. However, current data analysis competitions are not designed for educational purposes and it is not well studied how data analysis competition platforms should be designed for enhancing educational effectiveness. To answer this research question, we built, and subsequently operated an educational data analysis competition platform called University of Big Data for several years. In this paper, we present our approaches for supporting and motivating learners and the results of our case studies. We found that providing a tutorial article is beneficial for encouraging active participation of learners, and a leaderboard system allowing an unlimited number of submissions can motivate the efforts of learners. We further discuss future directions of educational data analysis competitions.

A method is presented for predicting the weights used in neural networks that shortens the training time by omitting some weight updates. In general neural network training, convergence slows as the dimensions of the input data, the amount of data, and/or the complexity of the model increases. The proposed method advances convergence and facilitates the implementations of neural networks on low-performance hardware such as mobile devices. Experiments using the MNIST, CIFAR-10, and SVHN datasets demonstrated that the proposed weight prediction method shortens the training time.

We present a comprehensive framework of search methods, such as simulated annealing and batch training, for solving nonconvex optimization problems. These methods search a wider range by gradually decreasing the randomness added to the standard gradient descent method. The formulation that we define on the basis of this framework can be directly applied to neural network training. This produces an effective approach that gradually increases batch size during training. We also explain why large batch training degrades generalization performance, which previous studies have not clarified.