Recently, action recognition has achieved impressive performance, mainly due to the aid of deep convolutional neural networks and large datasets. Traditionally, most efforts in action recognition have focused on capturing motion information by dense optical flow, but optical flow extraction is very time-consuming. Moreover, prior arts seek to improve accuracy but neglect the part-whole relationship between objects in videos, which may be self-defeating and even deteriorate the performance of methods. To circumvent the above challenges, we present a novel collaborative multipath capsule network (CMCN) for action recognition. In particular, we propose a plug-and-play collaborative multipath block containing spatiotemporal, channel, and motion units, which are complementary and crucial information for action recognition. We exploit the interaction of these three units and selectively emphasize informative spatial-temporal motion to reduce the expensive computational costs. Subsequently, we explore a new capsule voting procedure to reduce the computation used in the capsule dynamic routing mechanism. The critical insight is that the same type of capsules simulates the same entity in different positions, and their voting results should be consistent. This strategy lessens the number of learning parameters that backward pass in the training process, and thus strengthens part-whole relationships in a video. Extensive experiments on multiple real-world datasets for action recognition demonstrate that our model significantly outperforms state-of-the-art models.

Weakly Supervised Semantic Segmentation with image-level annotation uses localization maps from the classifier to generate pseudo labels. However, such localization maps focus only on sparse salient object regions, it is difficult to generate high-quality segmentation labels, which deviates from the requirement of semantic segmentation. To address this issue, we propose a dual-aware domain mining and cross-aware supervision (DDMCAS) method for weakly-supervised semantic segmentation. Specifically, we propose a dual-aware domain mining (DDM) module consisting of graph-based global reasoning unit and salient-region extension controller, which produces dense localization maps by exploring object features in salient regions and adjacent non-salient regions simultaneously. In order to further bridge the gap between salient regions and adjacent non-salient regions to generate more refined localization maps, we propose a cross-aware supervision (CAS) strategy to recover missing parts of the target objects and enhance weak attention in adjacent non-salient regions, leading to pseudo labels of higher quality for training the segmentation network. Based on the generated pseudo-labels, extensive experiments on PASCAL VOC 2012 dataset demonstrate that our method outperforms state-of-the-art methods using image-level labels for weakly supervised semantic segmentation.

As the scope of receptive field and the depth of Graph Neural Networks (GNNs) are two completely orthogonal aspects for graph learning, existing GNNs often have shallow layers with truncated-receptive field and far from achieving satisfactory performance. In this article, we follow the idea of decoupling graph convolution into propagation and transformation processes, which generates representations over a sequence of increasingly larger neighborhoods. Though this manner can enlarge the receptive field, it has two critical problems unsolved: how to find the suitable receptive field to avoid under-smoothing or over-smoothing? and how to balance different diffusion operators for better capturing the local and global dependencies? We tackle these challenges and propose a S calable, A daptive G raph C onvolutional N etworks ( SAGCN ) with Transformer architecture. Concretely, we propose a novel non-heuristic metric method that quickly finds the suitable number of diffusing iterations and produces smoothed local embeddings that enable the truncated receptive field to become scalable and independent of prior experience. Furthermore, we devise smooth2seq and diffusion-based position schemes introduced into Transformer architecture for better capturing local and global information among embeddings. Experimental results show that SAGCN enjoys high accuracy, scalability and efficiency on various open benchmarks and is competitive with other state-of-the-art competitors.

Weakly supervised semantic segmentation with image-level labels is a challenging problem that typically relies on the initial responses generated by the classification network to locate object regions. However, such initial responses only cover the most discriminative parts of the object and may incorrectly activate in the background regions. To address this problem, we propose a Cross-pixel Dependency with Boundary-feature Transformation (CDBT) method for weakly supervised semantic segmentation. Specifically, we develop a boundary-feature transformation mechanism, to build strong connections among pixels belonging to the same object but weak connections among different objects. Moreover, we design a cross-pixel dependency module to enhance the initial responses, which exploits context appearance information and refines the prediction of current pixels by the relations of global channel pixels, thus generating pseudo labels of higher quality for training the semantic segmentation network. Extensive experiments on the PASCAL VOC 2012 segmentation benchmark demonstrate that our method outperforms state-of-the-art methods using image-level labels as weak supervision.

While Graph Convolutional Networks (GCNs) have been extended to various fields of artificial intelligence with their powerful representation capabilities, recent studies have revealed that their ability to capture the part-whole structure of the graph is limited. Furthermore, though many GCNs variants have been proposed and obtained state-of-the-art results, they face the situation that much early information may be lost during the graph convolution step. To this end, we innovatively present an Graph Capsule Network with a Dual Adaptive Mechanism (DA-GCN) to tackle the above challenges. Specifically, this powerful mechanism is a dual-adaptive mechanism to capture the part-whole structure of the graph. One is an adaptive node interaction module to explore the potential relationship between interactive nodes. The other is an adaptive attention-based graph dynamic routing to select appropriate graph capsules, so that only favorable graph capsules are gathered and redundant graph capsules are restrained for better capturing the whole structure between graphs. Experiments demonstrate that our proposed algorithm has achieved the most advanced or competitive results on all datasets.

Scene recognition is challenging due to intra-class diversity and inter-class similarity. Previous works recognize scenes either with global representations or with intermediate representations of objects. By contrast, we investigate more discriminative sequential representation of object-to-scene relations (SOSRs) for scene recognition. Particularly, we develop an Attention-Preserving Memory-Learning (APML) model, which enforces the Memory Network of the semantic domain to guide the Learning Network of the appearance domain in the learning procedure. Accordingly, we allocate semantics-preserving attention to different objects, which is more effective to seek the key encoded SOSR and discard the misleading encoded SOSR between objects and scene without requiring extra labeled data. Based on the proposed APML networks, we obtain the state-of-the-art results of RGB-D scene recognition on SUN RGB-D and NYUD2 datasets.