Deep visual graph matching (GM) is a challenging combinatorial task that involves finding a permutation matrix that indicates the correspondence between keypoints from a pair of images. Like many learning systems, empirical studies have shown that visual GM is susceptible to adversarial attacks, with reliability issues in downstream applications. To the best of our knowledge, certifying robustness for deep visual GM remains an open challenge with two main difficulties: how to handle the paired inputs together with the heavily non-linear permutation output space (especially at large scale), and how to balance the trade-off between certified robustness and matching performance.

Many real-world machine learning problems involve structured prediction beyond categorical labels. However, most existing robustness certification works are devoted to the classification case. It remains open for robustness certification for more general outputs. In this paper, we propose a novel framework of robustness certification for structured prediction problems, where the output space is modeled as a semimetric space with a distance function that satisfies non-negativity and symmetry but not necessarily the triangle inequality. We further develop our tailored certification methods for binary, numerical, and hybrid inputs in structured prediction. Experiment results show that our method achieves tighter robustness guarantees than the SOTA structured certification baseline for numerical inputs (for which it only supports) with ℓ2 norm perturbation when outputs are measured by intersection over union (IoU) similarity, total variation distance, and perceptual distance. Moreover, we achieve good robustness certification for binary inputs with ℓ0 norm perturbation and hybrid inputs with corresponding perturbation when outputs are measured by Manhattan distance.

Recently, deep learning methods have proved exceptional performance in video action recognition. However, it remains a challenging problem to extract discriminative features from videos effectively. Most existing methods mainly focus on spatial-temporal information separately. In this paper, we propose a novel Stacked Diverse Attention Network (SDAN). It uses a Multi-dimensional Attention Module to emphasize informative maps along the channel and spatial-temporal dimension. Besides, a Supervised Attention Module is designed to generate a weighted feature map in a class-supervised way. Compared with previous methods, the proposed method has the following advantages: (1) The Multi-dimensional Attention Module can exploit effective combinations and correlation of attention mechanisms along different dimensions. (2) The Supervised Attention Module improves the network capability supervised directly by action labels which reveal the class-related object and limb information. Extensive experimental evaluation demonstrates the effectiveness of the proposed approach and establishes significant results on Kinetics400, UCF101 and HMDB51 Datasets. Codes are available on https://github.com/jeff62802217/SDAN-Pytorch.

This paper introduces a system integrating speech interaction and visual sensing, which aims at the smart home scenario. It is divided into three modules: data transmission and distribution, data reception and processing, monitoring and management. Simultaneously, the system employs a centralized hardware architecture with wired connection, ensuring the stability of duplex communication. It is proved that the system has powerful extensibility and favorable maintainability. In order to make speech interaction algorithms perform better in this system, a blind soure beamforming algorithm for speech enhancement is proposed. Meanwhile, a lot of considerations are used to improve the usability of visual sensing in smart home scenarios. Precisely speaking, the blind soure beamforming algorithm improves the poor performance of speech interaction for far-field and multi-angle, and the consideration improves the accuracy and reduces the time-delay of visual sensing.