In real-world scenarios, multimodal information continuously evolves, with new entity and relation types emerging, necessitating timely updates to multimodal knowledge graphs for supporting downstream tasks.However, existing methods struggle to balance real-time adaptability and computational efficiency in continual learning scenarios.To this end, this paper proposes the Continual Multimodal Entity and Relation Joint Extraction (CMERJE) task and a Multimodal Prompt-based Boundaryenhanced Continual (MPBoCo) framework.Specifically, MPBoCo incrementally stores task-specific knowledge via learnable multimodal prompts, dynamically matches relevant prompts for each instance, and fuses them into a frozen backbone model for task-specific reasoning.Subsequently, the boundary-enhanced dual-branch module leverages the auxiliary branch to preserve local syntactic continuity and provide boundary guidance.Experimental results demonstrate that MPBoCo achieves superior performance in real-world scenarios, significantly outperforming baseline methods by 5.5% and 7.2% in 10-task and 5-task settings, respectively.

Video question answering (VideoQA), a critical task in vision-language understanding and reasoning, encounters significant challenges in integrating visual concepts for compositional spatio-temporal reasoning, due to the inherent complexity of video data. Existing methods struggle to effectively capture both static and dynamic visual concepts and establish practical symbolic reasoning, which are essential for answering compositional questions. To address these challenges, we propose a neural-symbolic framework called Progressive Symbolic Reasoning based on Static-Dynamic Scene Parser Network (PSR-SDSP) for VideoQA tasks. PSR-SDSP features two key innovations: (1) the Scene Parser Network, which converts unstructured video data into static-dynamic symbolic representations, extracting structured representations of persons, objects, relationships, and action chronologies. (2) the Symbolic Reasoning Machine (SRM), which conducts top-down question decomposition and bottom-up compositional reasoning. Specifically, the SRM applies a program execution algorithm that integrates static-dynamic visual concepts with predicted programs to deduce the answers. Our results on AGQA Decomp show that PSR-SDSP outperforms existing models in compositional spatio-temporal reasoning. Experiments on the Novel Compositions and More Compositional Steps tasks highlight its potential generalization. Evaluations on datasets with varying degrees of domain differences, including STAR, ActivityNet-QA, and TVQA+, confirm PSR-SDSP's scalability.

Landscape painting is a gem of cultural and artistic heritage that showcases the splendor of nature through the deep observations and imaginations of its painters. Limited by traditional techniques, these artworks were confined to static imagery in ancient times, leaving the dynamism of landscapes and the subtleties of artistic sentiment to the viewer's imagination. Recently, emerging text-to-video (T2V) diffusion methods have shown significant promise in video generation, providing hope for the creation of dynamic landscape paintings. However, current T2V methods focus on generating natural videos, emphasizing the capture of details and the authenticity of physical laws. In contrast, landscape painting videos emphasize the overall dynamic aesthetic. Besides, challenges, such as the lack of specific datasets, the intricacy of artistic styles, and the creation of extensive, high-quality videos pose difficulties for these models in generating landscape painting videos. In this article, we propose landscape painting videos-high definition (LPV-HD), a novel T2V dataset for landscape painting videos, and noise-aware diffusion model (NADM), a T2V model that utilizes Stable Diffusion. Specifically, we present a motion module featuring a dual attention mechanism to capture the dynamic transformations of landscape imageries, alongside a noise adapter to leverage unsupervised contrastive learning in the latent space to ensure the overall beauty of the landscape painting video. Following the generation of keyframes, we employ optical flow for frame interpolation to enhance video smoothness. Our method not only retains the essence of the landscape painting imageries but also achieves dynamic transitions, significantly advancing the field of artistic video generation. Source code and dataset are available at https://github.com/llzlh21/NADM.

Abstract Image prior plays a decisive role in the performance of widely studied model‐based restoration methods. To further improve restoration performance, this paper proposes an exponential scale mixture‐based simultaneous sparse prior (ESM‐SSP) to accurately characterize image prior information. Specifically, first, two structured dictionaries are adaptively learned to explore the local and non‐local sparsity of similar patch groups simultaneously. Then, the exponential scale mixture (ESM) is employed to model simultaneous sparse coefficients. The adoption of ESM enables us to accurately estimate simultaneous sparse coefficients by adaptively adjusting the regularization parameters. With the aid of ESM‐SSP, an effective image restoration algorithm is developed to preserve more image details. Extensive experimental results on image denoising and deblocking demonstrate that compared with many state‐of‐the‐art model‐based methods, the proposed ESM‐SSP‐based restoration algorithm not only has competitive peak signal‐to‐noise ratio, but also produces higher structural similarity index and better visuals. More importantly, the proposed method can also compete favourably with the superior deep learning‐based restoration methods.

Modulation recognition has been an essential part in non-cooperative communications. Without knowing any prior knowledge of the transmitted data, the received signals may be affected by intersymbol interference (ISI), carrier frequency offset, phase offset and timing delay. ISI and these unknown parameters would degrade the performance of many recognition methods. There have been many robust estimators of carrier frequency offset, phase offset and timing delay. However, the robust recognition methods combating ISI have not been well designed. Most blind channel equalization methods, which are designed to overcome the effects of ISI, have poor robustness to noise and carrier frequency offset. In this paper, a robust recognition scheme is proposed to recognize phase-shift keying (PSK) and quadrature amplitude modulation (QAM) signals in non-cooperative communication with ISI environment. The envelope spectrum of PSK and QAM signals are analyzed. Then the recognition scheme based on the ratio of spectral peak to spectral centroid (RSPSC) of envelope spectrum is presented. Such scheme achieves a good recognition performance in the presence of ISI. Spectral centroid is utilized to enhance the robustness to noises. By extracting the envelope spectrum, in addition, the method avoids being affected by carrier frequency offset, phase offset and timing delay. The feasibility of the proposed method has been proved in the respects of theory and simulations.

Dual-tree transforms have recently received much attention for the properties of shift-invariance and directional-selectivity. However, their designs generally encounter fractional-delay constraints, and become more complicated for providing linear-phase (LP) individual filters and flexible directional-selectivity, two important properties in image processing. In this paper, we propose an alternative shift-invariant and directional-selective transform-the dual-tree cosine-modulated filter bank (DTCMFB). In the proposed DTCMFB, its primal and dual filter banks are derived by cosine-modulating one LP prototype filter, and thus its design involves no fractional-delay constraints. Meanwhile, the derived modulation technique guarantees each individual filter to be LP and the LP condition is satisfied without any constraint on the prototype filter. By separable operations, the DTCMFB is extended to two-dimensions. The resulting 2D DTCMFB can provide much more flexible directional-selectivity. Finally, several simulations are given to verify the proposed DTCMFB, and the experiments on nonlinear approximation and image denoising are presented to demonstrate its potential in image processing.

Directional filter banks (DFBs) are highly desired in directional representation of images. In this correspondence, we propose a 2-D nonsubsampled nonuniform directional filter bank (NUDFB) and its design method. The proposed NUDFB has nonuniform wedge-shaped subbands and allows arbitrary frequency partitioning schemes. It can extract directional information according to the directional distribution of images. This attractive advantage cannot be achieved by the existing directional transforms. The design method of the proposed NUDFB is based upon the pseudopolar Fourier transform. By utilizing the geometry property of the pseudopolar grid, we employ a 1-D nonsubsampled nonuniform filter bank to obtain a set of nonuniform wedge-shaped subbands. During the design process, only 1-D operations are involved and, thus, the difficulty encountered in the design of 2-D fan filters is avoided. To demonstrate the potential of the proposed NUDFB, an example on image directional decomposition is given.

Adaptive Directional Lifting (ADL) has been successfully implemented in image compression and denoising due to the feature of simple structure and flexible directional selectivity. However, image denoising by means of ADL introduces many visual artifacts caused by Gibbs phenomena due to the lack of translation invariance. In this paper, we propose a translation invariant directional lifting (TI-DL) by employing the cycle-spinning based technique to reduce artifacts in denoising results. Moreover, the inefficiency and high computational complexity of the orientation estimation technique in ADL strongly influences the performance. In order to achieve better denoising results, in this paper, 2-D Gabor filters are adopted for orientation estimation to achieve better orientation estimation results with lower complexity. Experimental results demonstrate that the proposed method achieves state-of-art denoising performance in terms of both objective (PSNR) and subjective (SSIM) evaluation.