The aim of audio-visual segmentation (AVS) is to precisely differentiate audible objects within videos down to the pixel level. Traditional approaches often tackle this challenge by combining information from various modalities, where the contribution of each modality is implicitly or explicitly modeled. Nevertheless, the interconnections between different modalities tend to be overlooked in audio-visual modeling. In this paper, inspired by the human ability to mentally simulate the sound of an object and its visual appearance, we introduce a bidirectional generation framework. This framework establishes robust correlations between an object's visual characteristics and its associated sound, thereby enhancing the performance of AVS. To achieve this, we employ a visual-to-audio projection component that reconstructs audio features from object segmentation masks and minimizes reconstruction errors. Moreover, recognizing that many sounds are linked to object movements, we introduce an implicit volumetric motion estimation module to handle temporal dynamics that may be challenging to capture using conventional optical flow methods. To showcase the effectiveness of our approach, we conduct comprehensive experiments and analyses on the widely recognized AVSBench benchmark. As a result, we establish a new state-of-the-art performance level in the AVS benchmark, particularly excelling in the challenging MS3 subset which involves segmenting multiple sound sources. Code is released in: https://github.com/OpenNLPLab/AVS-bidirectional.

Futuristic Space Networks (SN) present unprece-dented prospects for ubiquitous, low-latency Internet services. Yet, these networks also encounter unique challenges arising from the dynamic nature of satellites on a global scale. To comprehensively address emerging issues in SNs, researchers require the capability to conduct a diverse array of experiments. However, existing experimental approaches either implement visualization functionality but lack network functionality (e.g., the space simulator), or implement network functionality but lack visualization (e.g., the networking simulator). In this paper, we present SNK, a novel simulation platform with visualization and networking capabilities for evaluating the space network performance of global Internet services. SNK offers real-time communication visualization and supports the simulation of routing between edge node of network. The platform enables the evaluation of routing and network performance metrics such as latency, stretch, network capacity, and throughput under different network structures and density. The effectiveness of SNK is demonstrated through various simulation cases, including the routing between fixed edge stations or mobile edge stations and analysis of snace network structures.

Advancements in multimodal learning, particularly in video understanding and generation, require high-quality video-text datasets for improved model performance. Vript addresses this issue with a meticulously annotated corpus of 12K high-resolution videos, offering detailed, dense, and script-like captions for over 420K clips. Each clip has a caption of ~145 words, which is over 10x longer than most video-text datasets. Unlike captions only documenting static content in previous datasets, we enhance video captioning to video scripting by documenting not just the content, but also the camera operations, which include the shot types (medium shot, close-up, etc) and camera movements (panning, tilting, etc). By utilizing the Vript, we explore three training paradigms of aligning more text with the video modality rather than clip-caption pairs. This results in Vriptor, a top-performing video captioning model among open-source models, comparable to GPT-4V in performance. Vriptor is also a powerful model capable of end-to-end generation of dense and detailed captions for long videos. Moreover, we introduce Vript-Hard, a benchmark consisting of three video understanding tasks that are more challenging than existing benchmarks: Vript-HAL is the first benchmark evaluating action and object hallucinations in video LLMs, Vript-RR combines reasoning with retrieval resolving question ambiguity in long-video QAs, and Vript-ERO is a new task to evaluate the temporal understanding of events in long videos rather than actions in short videos in previous works. All code, models, and datasets are available in https://github.com/mutonix/Vript.

Aiming for efficient coordination mechanisms for the distributed weighted set cover problem, we study from ordinal potential game theoretic learning and propose a Nash equilibrium selection algorithm (NESA). An ordinal potential game model is established, where the local utility function is designed by incorporating a greedy heuristic. To distinguish Nash equilibria of different global fitness, we further classify them into the inferior Nash equilibrium (INE) and the superior Nash equilibrium (SNE), and show that the optimal solution must be an SNE. High-quality SNE solutions are obtained by assigning each player a local stochastic rule based on its category and a finite memory. By demonstrating the existence of a finite improvement path from each INE to an SNE, we prove finite-time convergence of the NESA. Numerical experiments are carried out and comparisons against representative methods are presented, which demonstrate the effectiveness as well as the superiority of our methodology to the state-of-the-art.

We explore a new task for audio-visual-language modeling called fine-grained audible video description (FAVD). It aims to provide detailed textual descriptions for the given audible videos, including the appearance and spatial locations of each object, the actions of moving objects, and the sounds in videos. Existing visual-language modeling tasks often concentrate on visual cues in videos while undervaluing the language and audio modalities. On the other hand, FAVD requires not only audio-visual-language modeling skills but also paragraph-level language generation abilities. We construct the first fine-grained audible video description benchmark (FAVDBench) to facilitate this research. For each video clip, we first provide a one-sentence summary of the video, i.e., the caption, followed by 4–6 sentences describing the visual details and 1–2 audio-related descriptions at the end. The descriptions are provided in both English and Chinese. We create two new metrics for this task: an EntityScore to gauge the completeness of entities in the visual descriptions, and an AudioScore to assess the audio descriptions. As a preliminary approach to this task, we propose an audio-visual-language transformer that extends existing video captioning model with an additional audio branch. We combine the masked language modeling and auto-regressive language modeling losses to optimize our model so that it can produce paragraph-level descriptions. We illustrate the efficiency of our model in audio-visual-language modeling by evaluating it against the proposed benchmark using both conventional captioning metrics and our proposed metrics. We further put our benchmark to the test in video generation models, demonstrating that employing fine-grained video descriptions can create more intricate videos than using captions. Code and dataset are available at https://github.com/OpenNLPLab/FAVDBench. Our online benchmark is available at www.avlbench.opennlplab.cn.

Acute respiratory distress syndrome (ARDS) prognosis has become integral to modern critical care models aimed at determining expected patient outcomes, optimizing clinical pathways, and improving resource allocation. However, existing clinical studies of ARDS prognosis face severe challenges due to the redundancy of multiple sources of heterogeneous data in current healthcare datasets, the high dimensionality of patient features, and the wide variation in the importance of each feature. In this paper, we propose a holistic ARDS prognostic framework for assessing ARDS prognosis through a standardized data governance process and an integrated feature selection approach based on embedded algorithms. Specifically, we extract ARDS patient data from medical data sources by various criteria and normalize the features. Then we apply multiple embedded feature selection methods to obtain a decision matrix based on the data-governed dataset. We conducted extensive experiments to demonstrate the efficiency and superiority of our proposed framework. The experimental results show that our proposed framework performs well in both data governance and feature selection and has wide clinical application potential.

Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers.However, they usually suffer from degraded performances on various tasks and corpora.In this paper, we examine existing kernel-based linear transformers and identify two key issues that lead to such performance gaps: 1) unbounded gradients in the attention computation adversely impact the convergence of linear transformer models; 2) attention dilution which trivially distributes attention scores over long sequences while neglecting neighbouring structures.To address these issues, we first identify that the scaling of attention matrices is the devil in unbounded gradients, which turns out unnecessary in linear attention as we show theoretically and empirically.To this end, we propose a new linear attention that replaces the scaling operation with a normalization to stabilize gradients.For the issue of attention dilution, we leverage a diagonal attention to confine attention to only neighbouring tokens in early layers.Benefiting from the stable gradients and improved attention, our new linear transformer model, TRANSNORMER, demonstrates superior performance on text classification and language modeling tasks, as well as on the challenging Long-Range Arena benchmark, surpassing vanilla transformer and existing linear variants by a clear margin while being significantly more space-time efficient.The code is available at TRANSNORMER.

To achieve the theoretical performance gains of massive multiple-input multiple-output (MIMO) systems, the base station (BS) must acquire the downlink channel state information (CSI). In frequency division duplexing (FDD) massive MIMO systems, downlink CSI is estimated at user terminals with the pilot symbols transmitted by the BS at the first step and then user terminals feed it back to the BS. However, the huge number of antennas at the BS will result in heavy feedback overhead. Meanwhile, CSI acquisition is very challenging because of the high mobility of user terminals which causes the priori channel knowledge of the channels to change from one slot to another. In order to solve these problems, we propose a joint channel training and feedback scheme based on compressive sensing (CS) and deep learning (DL). Specifically, with the CS-based algorithm, named AS-JOMP, the sparse channel in time-delay domain can be adaptively reconstructed firstly. Then the DL-based network, named DnLSTM, is utilized to estimate the CSI. Simulation results demonstrate that the proposed method can reduce the training and feedback overhead and outperforms the existing classical algorithms at time-varying channel estimation.

Abstract There exists periodic modulation problem in radar echoes due to the main rotor blades periodic blockage in helicopter‐borne fire‐control radar which is mounted atop the main rotor mast of the helicopter. Such modulation echo induces a set of ghosts in synthetic aperture radar (SAR) image, further resulting in a poor performance of subsequent tracking and striking. To address this problem, this article proposes a method on rotor blades blockage modulation suppression for helicopter‐borne SAR. By decomposing the blocked echo into the form of Fourier series in the azimuth direction, a reference function could be constructed to suppress the modulation directly by adopting an iterative approximation strategy. This method effectively avoids the complex blocked data recovery methods, and thus can be used to suppress various kinds of periodic modulation components without requiring certain distribution models. Both simulated and real‐measured data are processed to demonstrate the effectiveness of the proposed algorithm.

This paper proposes a framework called Episodic memory-driving Markov decision processes (EM-MDPs) for incremental self-learning of robotic experience and cognitive behavior control under uncertainty. The framework simulates the organization process of episodic memory by introducing the neuron stimulation mechanism. Firstly, episode model is built, and the activation and stimulation mechanism of state neurons is proposed based on cognitive neuroscience. Secondly, episodic self-learning is also proposed by utilizing sparse distributed memory (SDM) through Hebbian rules, to realize memory real-time storage, incremental accumulation and integration. Finally, a robotic cognitive behavior control approach is established. Neuron synaptic potential is introduced for event localization. Robot can evaluate the past events sequence, predict the current state and plan the desired behavior. Two main challenges in robot behavior control under uncertainty are addressed in the paper: high computational complexity and perceptual aliasing. The proposed system is evaluated in several real life environments for mobile robot. The applicability and the usefulness of the developed method are validated by the results obtained.