Knowledge graphs (KGs) play a central role in enabling structured reasoning for AI applications. Recent research has advanced two prominent extensions of KGs: multimodal KGs, which integrate diverse data sources such as text, images, audio, and video; and temporal KGs, which capture the dynamic evolution of knowledge over time. However, these two paradigms remain largely disjoint—multimodal KGs typically ignore temporal dynamics, while temporal KGs overlook rich multimodal context. To bridge this gap, we introduceFocusKG, the first discrete-time multimodal knowledge graph that unifies four modalities (text, image, audio, and video) with temporal annotations. Built from the Focus news program, FocusKG captures real-world events as temporally grounded multimodal facts, offering new expressiveness for time-sensitive tasks. We further propose aDiscrete-TimeMultimodal Knowledge GraphEmbedding (DTME) method, a novel framework tailored for learning representations over temporal multimodal KGs. DTME is composed of three key modules: (1) Multimodal Preprocessing, which encodes raw inputs from each modality into vector representations using modality-specific encoders; (2) Cross-Modal Graph Enhancer, which captures semantic interplay between entity and relation modalities through role-aware subgraph construction and a modality interaction network; and (3) Multi-Branch Time Aggregation, which injects temporal signals by modeling interactions between time and modality-aware features. Extensive experiments demonstrate FocusKG's value and DTME's efficacy. On link prediction, DTME outperforms state-of-the-art multimodal KG models and temporal KG models. We release FocusKG as a benchmark to foster research in multimodal temporal reasoning.

Single-channel speech separation remains one of the most challenging tasks in the field of speech signal processing. In many situations, such as during epidemics that involve respiratory diseases (e.g., COVID-19 or influenza A), individuals are required to wear masks while communicating. Is it possible to address the challenge of speech separation when the target speaker is wearing a mask? Can audio–visual approaches achieve better speech separation performance than that of audio-only approaches in scenarios where speakers are wearing masks? To address the aforementioned questions, we first construct a large-scale multimodal dataset, termed Speech Separation while Wearing a Mask (SSWM), which includes both the audio modality and the visual modality with masked faces. We explore two strategies for addressing the problem of facial occlusion. One strategy involves utilizing occluded faces—which lack critical visual cues such as mouth movements—directly as supervisory information for self-supervised speech separation; the other strategy involves the use of Wav2Lip to first generate visual information, which is then used as supervisory guidance for self-supervised speech separation. Building upon these two strategies, we propose the SSWM network (SSWMNet), which can flexibly choose to either utilize occluded facial images directly or employ Wav2Lip to generate visual information. The experimental results demonstrate that the proposed speech separation method in which Wav2Lip is used for visual information generation outperforms the approach of utilizing occluded faces directly for self-supervised speech separation. Both proposed audio–visual methods outperform the audio-only speech separation approach, which operates without the aid of visual information. Availability—SSWMNet is available at https://github.com/fanmanqian/SSWMNetwork .

National audience

Video question answering (VideoQA) is the challenging task of accurately responding to natural language questions based on a given video. Most previous methods focus on designing complex cross-modal interactions to perform question-oriented video scene mining and semantic reasoning, and utilize straightforward classification and matching strategies with different decoders to forcibly associate the predicted representation with ground-truth answer. However, the limitations of question-oriented reasoning and the overlapping semantic co-occurrences between questions and candidates may cause them to fall into spurious correlation reasoning. In this paper, we propose a Collaborative aware Bidirectional Semantic Reasoning (CBSR) model to alleviate this challenging problem. Specifically, we first propose a collaborative aware adaptive correlation reasoning module to collaboratively mine multi-granularity text-aware critical video scenes and reason about the complex intrinsic correlations between them via bottom-up cross-granularity adaptive aggregation. By progressively performing video reasoning from object-level to frame-level, we can obtain a set of semantically rich critical video representations. Then, we collaboratively decode it together with question and knowledge semantics into an implicit representation through the proposed unified answer semantic collaborated decoding module. Finally, a novel bidirectional semantic reasoning learning strategy is proposed to bridge and strengthen the unique positive semantic correlation between the learned implicit representation and the ground-truth answer, and explicitly alleviate the challenge of overlapping semantic co-occurrence. Benefiting from the same model structure and learning strategy, our method can achieve seamless transfer between Open-Ended and Multi-Choice tasks. Extensive experimental results on seven commonly tested datasets (i.e. MSVD-QA, MSRVTT-QA, NExT-QA, Causal-VidQA, NExT-OOD, ActivityNet-QA and EgoSchema) verify the superior performance of our method and the effectiveness of each reasoning module. We provide our source codes and experimental datasets athttps://github.com/XizeWu/CBSR.

Recent advances in Multimodal Entity Linking leverage multimodal information to link target mentions to corresponding entities. However, existing methods uniformly adopt a “one-size-fits-all” approach, which overlooks the unique requirements of individual samples and fails to adequately balance modality-assisted disambiguation and modality-induced noise. Also, the commonly used separate large-scale visual and text pretrained models for feature extraction do not address inter-modal heterogeneity and the high computational cost of fine-tuning. To resolve these two issues, we introduce a novel approach named Multimodal Entity Linking with Dynamic Modality Selection and Interactive Prompt Learning (DSMIP). First, we design three expert networks that utilize different subsets of modalities tailored to the task and train them individually. Specifically, for the multimodal expert network, we enhance entity and mention feature extraction by updating multimodal prompts and setting up a coupling function to realize the interaction of prompts between modalities. Subsequently, to select the best-suited expert network for each specific sample, we devise a Modality Selection Gating Network to gain the optimal one-hot selection vector by applying a specialized reparameterization technique and a two-stage training process. Experimental results on three public benchmark datasets demonstrate that the proposed DSMIP outperforms all state-of-the-art baselines. The code is released on https://github.com/mayy-seu/DSMIP-code.

In image processing, wavelet transform (WT) offers multiscale image decomposition, generating a blend of low-resolution approximation images and high-resolution detail components. Drawing parallels to this concept, we view feature maps in convolutional neural networks (CNNs) as a similar mix, but uniquely within the channel domain. Inspired by multitask learning (MTL) principles, we propose a wavelet-based dual-task (WDT) framework. This novel framework employs WT in the channel domain to split a single task into two parallel tasks, thereby reforming traditional single-task CNNs into dynamic dual-task networks. Our WDT framework integrates seamlessly with various popular network architectures, enhancing their versatility and efficiency. It offers a more rational approach to resource allocation in CNNs, balancing between low-frequency and high-frequency information. Rigorous experiments on Cifar10, ImageNet, HMDB51, and UCF101 validate our approach's effectiveness. Results reveal significant improvements in the performance of traditional CNNs on classification tasks, and notably, these enhancements are achieved with fewer parameters and computations. In summary, our work presents a pioneering step toward redefining the performance and efficiency of CNN-based tasks through WT.

Deep learning and Convolutional Neural Networks (CNNs) have driven major transformations in diverse research areas. However, their limitations in handling low-frequency in-formation present obstacles in certain tasks like interpreting global structures or managing smooth transition images. Despite the promising performance of transformer struc-tures in numerous tasks, their intricate optimization com-plexities highlight the persistent need for refined CNN en-hancements using limited resources. Responding to these complexities, we introduce a novel framework, the Mul-tiscale Low-Frequency Memory (MLFM) Network, with the goal to harness the full potential of CNNs while keep-ing their complexity unchanged. The MLFM efficiently preserves low-frequency information, enhancing perfor-mance in targeted computer vision tasks. Central to our MLFM is the Low-Frequency Memory Unit (LFMU), which stores various low-frequency data and forms a parallel channel to the core network. A key advantage of MLFM is its seamless compatibility with various prevalent networks, requiring no alterations to their original core structure. Testing on ImageNet demonstrated substantial accuracy improvements in multiple 2D CNNs, including ResNet, MobileNet, EfficientNet, and ConvNeXt. Furthermore, we showcase MLFM's versatility beyond traditional image classification by successfully integrating it into image-to-image translation tasks, specifically in semantic segmenta-tion networks like FCN and U-Net. In conclusion, our work signifies a pivotal stride in the journey of optimizing the ef-ficacy and efficiency of CNNs with limited resources. This research builds upon the existing CNN foundations and paves the way for future advancements in computer vision. Our codes are available at https://github.com/AlphaWuSeu/MLFM.

Cardiac vibration signals provide critical insights into the heart's mechanical function, making them essential for diagnosing and monitoring heart failure. An implantable cardiac device equipped with a 3D accelerometer has been recently developed in our research group to capture these signals. It enables continuous long-term monitoring of heart conditions. However, noise and artifacts can significantly degrade the quality of these signals, obscuring important cardiac events and diminishing the clinical utility of the implant. To improve signal quality, a graph-based method is introduced in this paper to denoise cardiac vibration signals captured by the 3D accelerometer which is embedded in an implantable device located in the gastric fundus. The proposed method leverages the intrinsic pseudo-periodicity of heart vibration signals across cardiac cycles. More precisely, it reformulates the denoising problem as the inference of a low-rank matrix along with target signals being assumed smooth on a graph structure which is optimally learned from the observed data. The efficacy of the proposed approach compared to standard denoising methods is confirmed using real recordings from a cohort of pigs, encompassing both those with and without heart failure.

Federated learning (FL) for histopathology image segmentation involving multiple medical sites plays a crucial role in advancing the field of accurate disease diagnosis and treatment. However, it is still a task of great challenges due to the sample imbalance across clients and large data heterogeneity from disparate organs, variable segmentation tasks, and diverse distribution. Thus, we propose a novel FL approach for histopathology nuclei and tissue segmentation, FedSODA, via synthetic-driven cross-assessment operation (SO) and dynamic stratified-layer aggregation (DA). Our SO constructs a cross-assessment strategy to connect clients and mitigate the representation bias under sample imbalance. Our DA utilizes layer-wise interaction and dynamic aggregation to diminish heterogeneity and enhance generalization. The effectiveness of our FedSODA has been evaluated on the most extensive histopathology image segmentation dataset from 7 independent datasets. The code is available at https://github.com/yuanzhang7/FedSODA.

Cardiac vibration signals provide insights into the mechanical function of the heart, making them potentially useful for the diagnosis and follow-up of heart failure. These signals can be captured using implantable cardiac devices incorporating a 3D accelerometer, as recently proposed in our group, enabling continuous longitudinal monitoring. However, the quality of these signals is highly affected by the presence of noise and artefacts originating from several physiological and non-physiological sources, thereby reducing the clinical effectiveness. To address this issue, a graph-based approach to improve the quality of cardiac vibration signals captured through 3D accelerometer recordings from an implantable device located in the gastric fundus, is proposed in this paper. By harnessing the inherent repetitive nature of these heart vibration signals across recorded heart cycles, the denoising problem is reformulated as the fact of inferring a target signal matrix incorporating both the smoothness on graph and low-rank constraints. The effectiveness of the proposed approach is shown through experiments conducted on real recordings acquired from a group of pigs, both with and without heart failure. The proposed approach shows superior performance compared to traditional de noising techniques.

By seamlessly integrating wavelet transforms into the image patching stage of ViT, we leverage the power of multi-level wavelet transforms to decompose images into a diverse array of frequency-domain features. These features, integrated with spatial characteristics at equivalent scales, enrich image details, enhancing ViT's proficiency in delineating intricate textures and distinct edges. Consequently, we registered a notable 2.7% accuracy enhancement on the ImageNet100 dataset in ViT. Our wavelet patching module, designed for versatility, seamlessly fits into various ViT derivatives without necessitating architecture modifications. This advancement has uplifted the performance of several leading vision transformers by 0.46–4.3%, preserving parameter efficiency without notable FLOPs increment.

The aim of end-to-end sign language translation (SLT) is to interpret continuous sign language (SL) video sequences into coherent natural language sentences without any intermediary annotations, i.e., glosses. However, end-to-end SLT suffers several intractable issues: (i) the temporal correspondence constraint loss problem between SL videos and glosses, and (ii) the weakly supervised sequence labeling problem between long SL videos and sentences. To address these issues, we propose an adaptive video representation enhanced Transformer (AVRET), with three extra modules: adaptive masking (AM), local clip self-attention (LCSA) and adaptive fusion (AF). Specifically, we utilize the first AM module to generate a special mask that adaptively drops out temporally important SL video frame representations to enhance the SL video features. Then, we pass the masked video feature to the Transformer encoder consisting of LCSA and masked self-attention to learn clip-level and continuous video-level feature information. Finally, the output feature of encoder is fused with the temporal feature of AM module via the AF module and use the second AM module to generate more robust feature representations. Besides, we add weakly supervised loss terms to constrain these two AM modules. To promote the Chinese SLT research, we further construct CSL-FocusOn, a Chinese continuous SLT dataset, and share its collection method. It involves many common scenarios, and provides SL sentence annotations and multi-cue images of signers. Our experiments on the CSL-FocusOn, PHOENIX14T, and CSL-Daily datasets show that the proposed method achieves the competitive performance on the end-to-end SLT task without using glosses in training. The code is available at https://github.com/LzDddd/AVRET.

In this paper, we propose a new simple and learning-free deep learning network named MomentsNet, whose convolution layer, nonlinear processing layer and pooling layer are constructed by Moments kernels, binary hashing and block-wise histogram, respectively. Twelve typical moments (including geometrical moment, Zernike moment, Tchebichef moment, etc.) are used to construct the MomentsNet whose recognition performance for binary image is studied. The results reveal that MomentsNet has better recognition performance than its corresponding moments in almost all cases and ZernikeNet achieves the best recognition performance among MomentsNet constructed by twelve moments. ZernikeNet also shows better recognition performance on a binary image database than that of PCANet, which is a learning-based deep learning network.

This paper proposes an multilinear discriminant analysis network (MLDANet) for the recognition of multidimensional objects, knows as tensor objects. The MLDANet is a variation of linear discriminant analysis network (LDANet) and principal component analysis network (PCANet), both of which are the recently proposed deep learning algorithms. The MLDANet consists of three parts: 1) The encoder learned by MLDA from tensor data. 2) Features maps obtained from decoder. 3) The use of binary hashing and histogram for feature pooling. A learning algorithm for MLDANet is described. Evaluations on UCF11 database indicate that the proposed MLDANet outperforms the PCANet, LDANet, MPCA+LDA, and MLDA in terms of classification for tensor objects.

The understanding of dose/side-effects relationships in prostate cancer radiotherapy is crucial to define appropriate individual's constraints for the therapy planning. Most of the existing methods to predict side-effects do not fully exploit the rich spatial information conveyed by the three-dimensional planned dose distributions. We propose a new classification method for three-dimensional individuals' doses, based on a new semi-nonnegative ICA algorithm to identify patients at risk of presenting rectal bleeding from a population treated for prostate cancer. The method first determines two bases of vectors from the population data: the two bases span vector subspaces, which characterize patients with and without rectal bleeding, respectively. The classification is then achieved by calculating the distance of a given patient to the two subspaces. The results, obtained on a cohort of 87 patients (at two year follow-up) treated with radiotherapy, showed high performance in terms of sensitivity and specificity.

In this letter, a new algorithm for joint diagonalization of a set of matrices by congruence is proposed to compute the nonnegative joint diagonalizer. The nonnegativity constraint is imposed by means of a square change of variables. Then we formulate the high-dimensional optimization problem into several sequential polynomial subproblems using LU matrix factorization. Numerical experiments on simulated matrices emphasize the advantages of the proposed method, especially in the case of degeneracies such as for low SNR values and a small number of matrices. An illustration of blind separation of nuclear magnetic resonance spectroscopy confirms the validity and improvement of the proposed method.

We present a new fast radix-3 algorithm for the computation of the length-Ngeneralized discrete Hartley transform of type-II (GDHT-II), whereN= 3m,m≥ 2. Then we apply this algorithm to the direct computation of length-NGDHT-II coefficients when given three adjacent length-N/3 GDHT-II coefficients. The computational complexity of the proposed method is lower than that of the traditional approach for lengthN≥ 9. The arithmetic operations can be saved from 19% to 29% forN= 3mvarying from 9 to 243 and from 17% to 29% forN= 3×2mvarying from 12 to 384. Furthermore, the new approach can be easily implemented.

A novel algorithm for fast computation of the modulated complex lapped transform (MCLT) with sine windowing function is presented. For the MCLT of length-2Minput data sequence, the proposed algorithm is based on computing a length-2Mtype-II generalized discrete Hartley transform. Comparison with existing algorithms shows that the proposed method achieves the minimal number of arithmetic operations.

In this paper, the objective was to investigate the heart rate variability in two selected groups of premature infants (sepsis vs non-sepsis). We studied the RR interval series not only by linear methods - time domain and frequency domain, but also by non-linear methods - chaos theory and information theory, in order to find the optimal parameters to distinguish sepsis premature infants from non-sepsis ones. The results show that indexes of information theory are useful parameters for the diagnosis of late neonatal infection in premature infants with recurrent apnea-bradycardia.

We present in this letter an efficient direct method for the computation of a length-N type-II generalized discrete Hartley transform (GDHT) when given two adjacent length-N/2 GDHT coefficients. The computational complexity of the proposed method is lower than that of the traditional approach for length Nges8. The arithmetic operations can be saved from 16% to 24% for N varying from 16 to 64. Furthermore, the new approach can be easily implemented

Knowledge of the q-th (q > 2) order spectrum of a linear non-Gaussian process allows to reconstruct both the magnitude and the phase of the corresponding input sequence. We propose in this paper a new family of phase retrieval algorithms, based on higher order spectra, named PEP (phase estimation using polyspectra) These new algorithms are easier to implement and use. Moreover, computer simulations show that among them, the 4-PEP and the (3.4)-PEP algorithms exhibit good performances facing classical methods especially for bandlimited systems

This paper deals with the conception of a new system for sleep staging in ambulatory conditions. Sleep recording is performed by means of five electrodes: two temporal, two frontal and a reference. This configuration enables to avoid the chin area to enhance the quality of the muscular signal and the hair region for patient convenience. The electroencephalopgram (EEG), eletromyogram (EMG), and electrooculogram (EOG) signals are separated using the Independent Component Analysis approach. The system is compared to a standard sleep analysis system using polysomnographic recordings of 14 patients. The overall concordance of 67.2% is achieved between the two systems. Based on the validation results and the computational efficiency we recommend the clinical use of the proposed system in a commercial sleep analysis platform.