Recent studies show that an ensemble of deep networks can have better adversarial robustness by increasing the deep feature learning diversity of base models to limit the adversarial transferability. However, existing schemes mostly rely on a second-order method for gradient regularization which usually involves a heavy computation overhead. In this paper, we propose a simple yet effective method which eliminates the use of a second-order optimization and significantly reduces the computation complexity of regularized simultaneous training of deep ensemble networks. For the first time, we show analytically that stochastic regularization by the proposed approach can promote both model smoothness and feature diversity of representation learning in the deep space. We also show that the proposed method is able to achieve a better gain of certified robustness. This is due to the effect of a prioritized feature selection enabled by an adaptive and continuous sampling of neuron activation among the base networks. Experimental results show that our method can improve adversarial robustness significantly comparing with the existing ensemble models on several image benchmark datasets. The ensemble performance can be further boosted by complementing the stochastic regularization approach with other defense paradigms such as adversarial training.

Learning-based classifiers are susceptible to adversarial examples. Existing defence methods are mostly devised on individual classifiers. Recent studies showed that it is viable to increase adversarial robustness by promoting diversity over an ensemble of models. In this paper, we propose adversarial defence by encouraging ensemble diversity on learning high-level feature representations and gradient dispersion in simultaneous training of deep ensemble networks. We perform extensive evaluations under white-box and black-box attacks including transferred examples and adaptive attacks. Our approach achieves a significant gain of up to 52% in adversarial robustness, compared with the baseline and the state-of-the-art method on image benchmarks with complex data scenes. The proposed approach complements the defence paradigm of adversarial training, and can further boost the performance. The source code is available at https://github.com/ALIS-Lab/AAAI2021-PDD.

Dropout regularization has been widely used in various deep neural networks to combat overfitting. It works by training a network to be more robust on information-degraded data points for better generalization. Conventional dropout and variants are often applied to individual hidden units in a layer to break up co-adaptations of feature detectors. In this paper, we propose an adaptive dropout to reduce the co-adaptations in a group-wise manner by coarse semantic information to improve feature discriminability. In particular, we showed that adjusting the dropout probability based on local feature densities can not only improve the classification performance significantly but also enhance the network robustness against adversarial examples in some cases. The proposed approach was evaluated in comparison with the baseline and several state-of-the-art adaptive dropouts over four public datasets of Fashion-MNIST, CIFAR-10, CIFAR-100 and SVHN.