In traditional sound event localization and detection (SELD) tasks, the focus is typically on sound event detection (SED) and direction-of-arrival (DOA) estimation, but they fall short of providing full spatial information about the sound source. The 3D SELD task addresses this limitation by integrating source distance estimation (SDE), allowing for complete spatial localization. We propose three approaches to tackle this challenge: a novel method with independent training and joint prediction, which firstly treats DOA and distance estimation as separate tasks and then combines them to solve 3D SELD; a dual-branch representation with source Cartesian coordinate used for simultaneous DOA and distance estimation; and a three-branch structure that jointly models SED, DOA, and SDE within a unified framework. Our proposed method ranked first in the DCASE 2024 Challenge Task 3, demonstrating the effectiveness of joint modeling for addressing the 3D SELD task. The relevant code is available now on https://github.com/yxdong0320/Solution_on_3D_SELD.

In the MMCSG task of the CHiME-8 Challenge, achieving real-time speaker-attributed transcriptions with limited multi-modal data presents significant challenges. To cope with the problem, we propose a novel ASR framework that leverages both audio-only and multi-modal inputs in a streaming fashion. For the audio-only modality, analyzing and emulating the characteristics of real audio, we utilize a multi-channel simulation to generate the augmented dataset, which efficiently reduces the model training deviation between real and simulated data. Additionally, we integrate the IMU data with audio data in the network structure, demonstrating that the functional filtered and encoded IMU data can assist audio information in achieving better real-time speech recognition performance with ablation experiments. Notably, our explorations based on the above schemes not only secured first place in the MMCSG sub-track but also represented the first investigation into the effectiveness of leveraging IMU data for this task.

Watermarking for large language models (LLMs) offers a promising approach to identifying AI-generated text. Existing approaches, however, either compromise the distribution of original generated text by LLMs or are limited to embedding zero-bit information that only allows for watermark detection but ignores identification. We present StealthInk, a stealthy multi-bit watermarking scheme that preserves the original text distribution while enabling the embedding of provenance data, such as userID, TimeStamp, and modelID, within LLM-generated text. This enhances fast traceability without requiring access to the language model’s API or prompts. We derive a lower bound on the number of tokens necessary for watermark detection at a fixed equal error rate, which provides insights on how to enhance the capacity. Comprehensive empirical evaluations across diverse tasks highlight the stealthiness, detectability, and resilience of StealthInk, establishing it as an effective solution for LLM watermarking applications.

Abstract—Fine-grained object detection (FOD) is essential in many remote sensing image interpretation tasks. Existing FOD methods have achieved remarkable progress in modeling discriminative features for FOD in remote sensing images. However, they receive unsatisfactory recognition accuracy due to the curse of dimensionality (CoD) problem. In this paper, we propose an orthogonal constraint-based discriminative latent-space learning (DLL) method to address the CoD problem. We first optimize a sparse optimization paradigm with convex relaxation to extract shared features between fine-grained objects into a low-dimensional latent-space. Then, we solve the orthogonal space of the latent-space for extracting irrelevant features related to common features from input features, i.e., discriminative features. The sparse optimization paradigm reprojects the underlying trends hidden in high-dimensional feature space into a low-dimensional latent space and hence addresses the CoD problem. We use neural parameters with latent-space and orthogonal constraints to approximately solve the proposed DLL, which can be efficiently optimized under convex programming tools. We theoretically prove the effectiveness of the proposed DLL. By adding our DLL to the existing deep learning-based object detection method, extensive experiments conducted on two datasets demonstrate that our method achieves superior performance compared with other state-of-the-art methods.

This study presents an audio-visual information fusion approach to sound event localization and detection (SELD) in low-resource scenarios. We aim at utilizing audio and video modality information through cross-modal learning and multi-modal fusion. First, we propose a cross-modal teacher-student learning (TSL) framework to transfer information from an audio-only teacher model, trained on a rich collection of audio data with multiple data augmentation techniques, to an audiovisual student model trained with only a limited set of multimodal data. Next, we propose a two-stage audio-visual fusion strategy, consisting of an early feature fusion and a late video-guided decision fusion to exploit synergies between audio and video modalities. Finally, we introduce an innovative video pixel swapping (VPS) technique to extend an audio channel swapping (ACS) method to an audio-visual joint augmentation. Evaluation results on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge data set demonstrate significant improvements in SELD performances. Furthermore, our submission to the SELD task of the DCASE 2023 Challenge ranks first place by effectively integrating the proposed techniques into a model ensemble.

The audio-visual direction of arrival (DOA) estimation has demonstrated superior performance recently. In this paper, we present a novel audio-visual multi-speaker DOA estimation network, which for the first time incorporates multi-speaker lip features to adapt the complex overlapping and noisy scenarios. Firstly, we encode the multi-channel audio features, the reference angles and the lip Regions of Interest (RoIs) detected from the video respectively to acquire high-level representations. Then the multi-modal embeddings of audio, speaker angles and lips are fused by a tri-modal gated fusion module to balance their contributions to the output. The fused embedding is sent to the backend network to obtain the accurate DOA estimation with the combination of the predicted speaker angular vectors and the speaker activities. Experimental results show that our proposed approach can reduce the localization error by 73.48% compared to the previous work on the 2021 Multi-modal Information based Speech Processing (MISP) Challenge corpus. Meanwhile, the high accuracy and stability of localization results demonstrate the robustness of the proposed model in multi-speaker scenarios.

In this paper, we propose a novel framework for recognizing both discrete and dimensional emotions. In our framework, deep features extracted from foundation models are used as robust acoustic and visual representations of raw video. Three different structures based on attention-guided feature gathering (AFG) are designed for deep feature fusion. Then, we introduce a joint decoding structure for emotion classification and valence regression in the decoding stage. A multi-task loss based on uncertainty is also designed to optimize the whole process. Finally, by combining three different structures on the posterior probability level, we obtain the final predictions of discrete and dimensional emotions. When tested on the dataset of multimodal emotion recognition challenge (MER 2023), the proposed framework yields consistent improvements in both emotion classification and valence regression. Our final system achieves state-of-the-art performance and ranks third on the leaderboard on MER-MULTI sub-challenge.

The complicated multi-step attacks, such as Advanced Persistent Threats (APTs), have brought considerable threats to cybersecurity because they are naturally varied and complex. Therefore, studying the strategies of adversaries and making predictions are still significant challenges for attack prevention. To address these problems, we proposeDeepAG, a framework utilizing system logs to detect threats and predict the attack paths.DeepAGleverages transformer models to novelly detect APT attack sequences by modeling semantic information of system logs. On the other hand,DeepAGutilizes Long Short-Term Memory (LSTM) network to propose bi-directional prediction for attack paths, which achieves higher performance than traditional BiLSTM. In addition, with previously detected attack sequences and predicted paths,DeepAGconstructs the attack graphs that attackers may follow to compromise the network. Furthermore,DeepAGoffers the mechanisms of Out-Of-Vocabulary (OOV) word processor and online update respectively to adapt new attack patterns that show up during detection and prediction stages. The experiments on open-source data sets show that more than 99% of over 15000 sequences can be detected accurately byDeepAG. Moreover,DeepAGcan improve the baseline by 11.166% of accuracy in terms of prediction.