Video Moment Retrieval (VMR) involves locating specific moments within a video based on natural language queries. However, existing VMR methods that employ various strategies for cross-modal alignment still face challenges such as limited understanding of fine-grained semantics, semantic overlap, and sparse constraints. To address these limitations, we propose a novel Concept Decomposition Transformer (CDTR) model for VMR. CDTR introduces a semantic concept decomposition module that disentangles video moments and sentence queries into concept representations, reflecting the relevance between various concepts and capturing fine-grained semantics which is crucial for cross-modal matching. These decomposed concept representations are then used as pseudo-labels, determined as positive or negative samples by adaptive concept-specific thresholds. Subsequently, fine-grained concept alignment is performed in video intra-modal and textual-visual cross-modal, aligning different conceptual components within features, enhancing the model's ability to distinguish fine-grained semantics, and alleviating issues related to semantic overlap and sparse constraints. Comprehensive experiments demonstrate the effectiveness of the CDTR, outperforming state-of-the-art methods on three widely used datasets: QVHighlights, Charades-STA, and TACoS.

Temporal Sentence Grounding (TSG) requires a thorough understanding of the complex cross-modal semantic relationships between videos and text. However, existing methods fail to accurately capture content at diverse granularity levels with distinct semantics, making it difficult to achieve fine alignment of visuals and text. To overcome this issue, we attempt to mine for rich semantic clues by utilizing the hierarchical correspondence structure and multi-granularity visual-to-text reconstruction, achieving fine-grained reasoning. Specifically, for the TSG task, we propose a novel Hierarchical Cross-modal Fine-grained Mining Network (HCFMN), which utilizes an attention mechanism based on temporal hierarchical relationships to extract temporal features corresponding to the text of different granularities. We leverage the reconstructability of visual-to-text, recovering multi-granularity textual content from coarse to fine by focusing on temporal features at different layers, hierarchically extracting temporal features and the dependencies related to the text, and implementing fine-grained cross-modal semantic alignment. Furthermore, HCFMN introduces a novel partitioned efficient attention mechanism, which significantly enhances the model’s efficiency through a two-stage attention based on sequence and channel compression. Extensive experimental results on three public datasets (ActivityNet-Captions, TACoS, and Charades-STA) demonstrate that the proposed method achieves state-of-the-art performance.

Deep Neural Networks (DNNs) have primarily been demonstrated to be successful when large-scale labeled data are available. However, DNNs usually fail when tasked in few-sample learning scenarios, and the results will be much worse when the limited data show large intra-class variation and inter-class similarity (a.k.a fine-grained classification). To solve this challenging task, the idea of carrying out feature augmentation is visited and better achieved by exploring the merit of the forward Euler method in solving ordinary differential equations (ODEs), and a novel high-order feature augmentation (HFA) model with ResNet is proposed. Specifically, the proposed method leverages the stacked residual structure to model the direction of feature change over the initial state, and uses the triplet loss as constraint to model the step size of change in an adaptive manner. As a result, the initial features can then be augmented by a residual structure with a forward Eulerian form to generate features of the same subcategory with a similar representation as the input image. Furthermore, the proposed augmentation mechanism enjoys two additional benefits: a) it can help avoid the over-fitting issue when learned with insufficient training data; b) it can be used seamlessly with any residual structure-based classification network, and the ResNet used in this paper remains unchanged during testing. Extensive experiments are carried out on fine-grained visual categorization benchmarks, and the results demonstrate that our approach can significantly improve the categorization performance when the training data is highly insufficient.

Fine-grained visual classification (FGVC) is challenging due to the interclass similarity and intraclass variation in datasets. In this work, we explore the great merit of complex values in introducing an imaginary part for modeling data uncertainty (e.g., different points on the complex plane can describe the same state) and graph convolutional networks (GCNs) in learning interdependently among classes to simultaneously tackle the above two major challenges. To the end, we propose a novel approach, termed text-assisted complex-valued fusion network (TA-CFN). Specifically, we expand each feature from 1-D real values to 2-D complex value by disassembling feature maps, thereby enabling the extension of traditional deep convolutional neural networks over the complex domain. Then, we fuse the real and imaginary parts of complex features through complex projection and modulus operation. Finally, we build an undirected graph over the object labels with the assistance of a text corpus, and a GCN is learned to map this graph into a set of classifiers. The benefits are in two folds: 1) complex features allow for a richer algebraic structure to better model the large variation within the same category and 2) leveraging the interclass dependencies brought by the GCN to capture key factors of the slight variation among different categories. We conduct extensive experiments to verify that our proposed model can achieve the state-of-the-art performance on two widely used FGVC datasets.

Recurrent neural networks (RNNs) have gained tremendous popularity in almost every sequence modeling task. Despite the effort, these kinds of discrete unstructured data, such as texts, audio, and videos, are still difficult to be embedded in the feature space. Studies in improving the neural networks have accelerated since the introduction of more complex or deeper architectures. The improvements of previous methods are highly dependent on the model at the expense of huge computational sources. However, few of them pay attention to the algorithm. In this article, we bridge the Taylor series with the construction of RNN. Training RNN can be considered as a parameter estimate for the Taylor series. However, we found that there is a discrete term called the remainder in the finite Taylor series that cannot be optimized using gradient descent, which is part of the reason for the truncation error and the model falling into the local optimal solution. To address this, we propose a training algorithm that estimates the range of remainder and introduces the remainder obtained by sampling in this continuous space into the RNN to assist in optimizing the parameters. Notably, the performance of RNN can be improved without changing the RNN architecture in the testing phase. We demonstrate that our approach is able to achieve state-of-the-art performance in action recognition and cross-modal retrieval tasks.

Learning delicate feature representation of object parts plays a critical role in fine-grained visual classification tasks. However, advanced deep convolutional neural networks trained for general visual classification tasks usually tend to focus on the coarse-grained information while ignoring the fine-grained one, which is of great significance for learning discriminative representation. In this work, we explore the great merit of multi-modal data in introducing semantic knowledge and sequential analysis techniques in learning hierarchical feature representation for generating discriminative fine-grained features. To this end, we propose a novel approach, termed Channel Cusum Attention ResNet (CCA-ResNet ), for multi-modal joint learning of fine-grained representation. Specifically, we use feature-level multi-modal alignment to connect image and text classification models for joint multi-modal training. Through joint training, image classification models trained with semantic level labels tend to focus on the most discriminative parts, which enhances the cognitive ability of the model. Then, we propose a Channel Cusum Attention (CCA ) mechanism to equip feature maps with hierarchical properties through unsupervised reconstruction of local and global features. The benefits brought by the CCA are in two folds: a) allowing fine-grained features from early layers to be preserved in the forward propagation of deep networks; b) leveraging the hierarchical properties to facilitate multi-modal feature alignment. We conduct extensive experiments to verify that our proposed model can achieve state-of-the-art performance on a series of fine-grained visual classification benchmarks.

Adversarial attacks are very successful on image classification, but there are few researches on vision-language systems, such as image captioning. In this paper, we study the robustness of a CNN+RNN based image captioning system being subjected to adversarial noises in complex domain. In particular, we propose Fooled-by-Imagination, a novel algorithm for crafting adversarial examples with semantic embedding of targeted caption as perturbation in complex domain. The proposed algorithm explores the great merit of complex values in introducing imaginary part for modeling adversarial perturbation, and maintains the similarity of the image in real part. Our approach provides two evaluation approaches, which check whether neural image captioning systems can be fooled to output some randomly chosen captions or keywords. Besides, our method has good transferability under black-box setting. At last, our extensive experiments show that our algorithm can successfully craft visually-similar adversarial examples with randomly targeted captions or keywords at a higher success rate.

Fine-grained visual recognition, which aims to identify subcategories of the same base-level category, is a challenging task because of its large intra-class variances and small inter-class variances. Human beings can perform object recognition task based on not only the visual appearance but also the knowledge from texts, as texts can point out the discriminative parts or characteristics which are always the key to distinguishing different subcategories. This is an involuntary transfer from human textual attention to visual attention, suggesting that texts are able to assist fine-grained recognition. In this paper, we propose a Text-Embedded Bilinear (TEB) model which incorporates texts as extra guidance for fine-grained recognition. Specially, we first conduct a text-embedded network to embed text feature into the discriminative image feature learning to get a embedded feature. In addition, since the cross-layer part feature interaction and fine-grained feature learning are mutually correlated and can reinforce each other, we also extract a candidate feature from the text encoder and embed it into the inter-layer feature of the image encoder to get an embedded candidate feature. At last we utilize a cross-layer bilinear network to fuse the two embedded features. Comparing with state-of-the-art methods on the widely used CUB-200-2011 dataset and Oxford Flowers-102 dataset for fine-grained image recognition, the experimental results demonstrate our TEB model achieves the best performance.

Small data challenges have emerged in many learning problems, since the success of deep neural networks often relies on the availability of a huge number of labeled data that is expensive to collect. We explore a highly challenging task, few-sample training, which uses a small number of labeled images of each category and corresponding textual descriptions to train a model for fine-grained visual categorization. In order to tackle overfitting caused by small data, in this paper, we propose two novel feature augmentation approaches, Semantic Gate Feature Augmentation (SGFA) and Semantic Boundary Feature Augmentation (SBFA). Instead of generating a new image instance, we propose to directly synthesize instance features by leveraging semantic information, and its main novelties are: (1) The SGFA method is proposed to reduce the overfitting of small data by adding random noise to different regions of the image's feature maps through a gating mechanism. (2) The SBFA approach is proposed to optimize the decision boundary of the classifier. Technically, the decision boundary of the image feature is estimated through the assistance of semantic information, and then feature augmentation is performed by sampling in this region. Experiments in fine-grained visual categorization benchmark demonstrate that our proposed approach can significantly improve the categorization performance.

Due to the popularity of soccer and the purposeful use of the head during play, traumatic brain injury to soccer players has been a concern for decades. However, there is a sense of urgency now in understanding and preventing concussions better, due to raising public awareness. Towards that end, intra-oral devices such as Vector MouthGuards are being studied [3] to measure the athlete's head's linear and rotational accelerations from impacts experienced in practices and games. But given the players' natural distaste for such intra-oral devices, more palatable alternatives for head impact monitoring are being developed [2]. X2 Biosystems xPatchis an electronic skin patch thatis worn behind the ear. Reebok Checklight embeds the impact sensor in the back of a skullcap which can be worn with or without a helmet. Triax SIM-P is placed inside a headband for non-helmeted sports and a skullcap for helmeted sports. While all these devices are much more convenient to wear than intra-oral devices, it is yet to be seen whether they gain wider acceptance, particularly by the millions of amateur soccer players all over the world.

Silicon offers a new set of possibilities and challenges for RF, microwave, and millimeter-wave applications. While the high cutoff frequencies of the SiGe heterojunction bipolar transistors and the ever-shrinking feature sizes of MOSFETs hold a lot of promise, new design techniques need to be devised to deal with the realities of these technologies, such as low breakdown voltages, lossy substrates, low-Q passives, long interconnect parasitics, and high-frequency coupling issues. As an example of complete system integration in silicon, this paper presents the first fully integrated 24-GHz eight-element phased array receiver in 0.18-/spl mu/m silicon-germanium and the first fully integrated 24-GHz four-element phased array transmitter with integrated power amplifiers in 0.18-/spl mu/m CMOS. The transmitter and receiver are capable of beam forming and can be used for communication, ranging, positioning, and sensing applications.