Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.

ABSTRACT Library inventory is vital for collection management and reader satisfaction. Conventional manual methods cannot support real‐time updates, while existing automated solutions relying on centralized cloud computing suffer from bandwidth and latency limitations. To address these issues, we propose an edge‐cloud collaborative real‐time book inventory system. Spine detection and text recognition are executed on embedded edge devices, while the cloud handles rapid data retrieval to balance timeliness and accuracy. We design lightweight models for edge deployment, including the Library You Only Look Once (Lib‐YOLO) detector with a StarNet backbone, shared convolutional head, and dual‐scale hierarchical detection, supporting rotated objects for robust spine extraction. The optimized paddle practical optical character recognition (PP‐OCR) pipeline removes text rectification and integrates a filtering algorithm to reduce redundant computation and improve efficiency. Deployed on an NVIDIA Jetson Nano, the system achieves 73 ms spine detection latency, 191 ms text recognition latency, and 97.1% overall accuracy under simulated library conditions. The Lib‐YOLO model contains only 1.39 M parameters with 99% mean average precision (mAP), demonstrating the feasibility of precise, real‐time inventorying in resource‐constrained embedded environments.

Edge computing extends computing power down to the network edge near the sensors and provides promising solutions to address high bandwidth requirements and latency-sensitive problems in smart video surveillance. Deep learning-based object detection has made significant progress in accuracy and is playing an important role in video surveillance. However, the computation and storage resources are still far below the advances reached by cloud servers, making it difficult to migrate traditional object detection models to the resource-constrained edge. Thus, in this paper, we propose a new lightweight object detection model, called AT-YOLO to detect key targets in the video stream. The efficient single compressed dense block (SCDB) is introduced to build the backbone network of AT-YOLO, which improves the computational efficiency of the network. Besides, the Attention Guide Task-aligned Head (AGT-head), which enhances the classification and the localization features, and encourages the interaction and the alignment of both, is proposed to improve the detection accuracy. Considering the redundancy in video streams and the improvement of detection efficiency, a Convolutional Neural Network (CNN)-based keyframe algorithm is considered as it has only 0.32 M parameters and can accurately and quickly filter out keyframes. We deployed the CNN-based keyframe algorithm and AT-YOLO on the edge devices of the system to implement an intelligent surveillance system based on edge computing. Experiments and evaluations on COCO demonstrate that AT-YOLO achieves 29.9% mAP at 3.47 M parameters. Moreover, in experiments with real devices, our approach can reduce the bandwidth requirement to 467.29 Kbps. And it achieves a processing speed of 13.69 FPS on the Raspberry Pi 4B, which is a single-board computer. With the more powerful device selected, the better real-time performance it can get.

Few-shot semantic segmentation aims to extract information from few annotated support images to segment unknown class objects in the query image. Traditional algorithms may produce errors and insufficient feature extraction using multi-layer cosine similarity to extract correlation information, due to the large differences in appearance and posture between novel class objects, as well as the similarity in texture and shape among different categories. To address the above issue, we propose a Complementary Feature-Enhanced Network (CFENet). Specifically, we propose a correlation complementary extraction module (CCEM) to facilitate long-range information interaction between query features and support features in the intermediate layer, which contains detailed information. The generated multi-channel correlation information complements the prior information obtained through cosine similarity comparison. In addition, we propose a multi-branch feature enhancement module to capture long-range dependencies in aggregated features which are composed of prior correlation information and query features. The module effectively suppresses noise in the aggregated features and enhances the query target object feature from both global and local perspectives in a complementary way. Experiments of the network on PASCAL-$5^{i}$and COCO-$20^{i}$datasets validate the effectiveness of our proposed method.

Abstract With constant advances in deep learning methods as applied to image processing, deep convolutional neural networks (CNNs) have been widely explored in single‐image super‐resolution (SISR) problems and have attained significant success. These CNN‐based methods cannot fully use the internal and external information of the image. The authors add a lightweight Transformer structure to capture this information. Specifically, the authors apply a dense block structure and residual connection to build a residual dense convolution block (RDCB) that reduces the parameters somewhat and extracts shallow features. The lightweight transformer block (LTB) further extracts features and learns the texture details between the patches through the self‐attention mechanism. The LTB comprises an efficient multi‐head transformer (EMT) with small graphics processing unit (GPU) memory footprint, and benefits from feature preprocessing by multi‐head attention (MA), reduction, and expansion. The EMT significantly reduces the use of GPU resources. In addition, a detail‐purifying attention block (DAB) is proposed to explore the context information in the high‐resolution (HR) space to recover more details. Extensive evaluations of four benchmark datasets demonstrate the effectiveness of the authors’ proposed model in terms of quantitative metrics and visual effects. The proposed EMT only uses about 40% as much GPU memory as other methods, with better performance.

The understanding of subcellular localization (SCL) of proteins and proteome variation in the different tissues and organs of the human body are two crucial aspects for increasing our knowledge of the dynamic rules of proteins, the cell biology, and the mechanism of diseases. Although there have been tremendous contributions to these two fields independently, the lack of knowledge of the variation of spatial distribution of proteins in the different tissues still exists. Here, we proposed an approach that allows predicting protein SCL on tissue specificity through the use of tissue-specific functional associations and physical protein-protein interactions (PPIs). We applied our previously developed Bayesian collective Markov random fields (BCMRFs) on tissue-specific protein-protein interaction network (PPI network) for nine types of tissues focusing on eight high-level SCL. The evaluated results demonstrate the strength of our approach in predicting tissue-specific SCL. We identified 1,314 proteins that their SCL were previously proven cell line dependent. We predicted 549 novel tissue-specific localized candidate proteins while some of them were validated via text-mining.