To effectively support lifelong learning, personalized e-learning systems aim to provide customized educational services tailored to individual learners’ needs. A crucial aspect for enabling these intelligent services is knowledge tracing, which involves dynamically modeling learners’ knowledge acquisition based on their past learning records. Traditional deep knowledge tracing models primarily focus on learners’ responses and the knowledge components covered in exercises, often neglecting other valuable multimodal learning features. In response to this limitation, our contribution is the introduction of the multimodal knowledge tracing (MMKT) model, which leverages four distinct learning modalities: textual, image, cognitive parametric, and knowledge association modalities. MMKT initially employs a combination of pretrained models and fine-tuning to obtain representations for text and image modalities. Using the EM algorithm, it acquires the cognitive parameter modality for exercises and learners. Subsequently, a multimodal fusion is applied to obtain a comprehensive exercise representation. In the next step, the model explores knowledge impact weights through the knowledge association modality. This information is combined with the cognitive parameter modality of the learner and the hidden states tracked by the gated recurrent unit, resulting in a multimodal representation of the learner’s knowledge state. Experiments conducted on publicly available datasets demonstrate the significant improvement of the proposed MMKT model. The proposed model outperforms the widely used baselines, deep knowledge tracing and dynamic key–value memory networks, achieving relative AUC gains of 3.57% and 6.37%, respectively, in predicting student learning performance. This study provides valuable insights that assist learners in understanding their knowledge mastery and contribute to the realization of lifelong learning from a technical standpoint.

Intelligent learning diagnosis is a critical engine of intelligent tutoring systems, which aims to estimate learners’ current knowledge mastery status and predict their future learning performance. The significant challenge with traditional learning diagnosis methods is the inability to balance diagnostic accuracy and interpretability. Although the existing psychometric‐based learning diagnosis methods provide some domain interpretation through cognitive parameters, they have insufficient modeling capability with a shallow structure for large‐scale learning data. While the deep learning‐based learning diagnosis methods have improved the accuracy of learning performance prediction, their inherent black‐box properties lead to a lack of interpretability, making their results untrustworthy for educational applications. To settle the abovementioned problem, the proposed unified interpretable intelligent learning diagnosis framework, which benefits from the powerful representation learning ability of deep learning and the interpretability of psychometrics, achieves a better performance of learning prediction and provides interpretability from three aspects: cognitive parameters, learner‐resource response network, and weights of self‐attention mechanism. Within the proposed framework, this paper presents a two‐channel learning diagnosis mechanism LDM‐ID as well as a three‐channel learning diagnosis mechanism LDM‐HMI. Experiments on two real‐world datasets and a simulation dataset show that our method has higher accuracy in predicting learners’ performances compared with the state‐of‐the‐art models and can provide valuable educational interpretability for applications such as precise learning resource recommendation and personalized learning tutoring in intelligent tutoring systems.

End-to-end Long Short-Term Memory (LSTM) has been successfully applied to video summarization. However, the weakness of the LSTM model, poor generalization with inefficient representation learning for inputted nodes, limits its capability to efficiently carry out node classification within user-created videos. Given the power of Graph Neural Networks (GNNs) in representation learning, we adopted the Graph Information Bottle (GIB) to develop a Contextual Feature Transformation (CFT) mechanism that refines the temporal dual-feature, yielding a semantic representation with attention alignment. Furthermore, a novel Salient-Area-Size-based spatial attention model is presented to extract frame-wise visual features based on the observation that humans tend to focus on sizable and moving objects. Lastly, semantic representation is embedded within attention alignment under the end-to-end LSTM framework to differentiate indistinguishable images. Extensive experiments demonstrate that the proposed method outperforms State-Of-The-Art (SOTA) methods.

This paper presents a compression technique to improve the quality of three-dimensional (3D) audio produced by multiple loudspeaker channels or by headphone. The approach is based on extracting the side information of spatial sound sources within the three-dimensional space when capturing the sound sources. Different from other compression technique, the distances of sound sources are included in the side information. The separated signals of different sound sources are downmixed into one mono or stereo audio signal with the side information. The resulting downmixed signal is then compressed with traditional audio coder, resulting in a better perceptual quality of 3D audio by adding the distance parameter in the side information, and maintaining a low bit rates comparable with directional audio coding (DirAC).

Three dimensional (3D) audio technologies are booming with the success of 3D video technology. The sharply increased audio channels make its huge data unacceptable for transmitting bandwidth and storage media. This paper investigates the conventional Mid/Side (M/S) coding method, and expands it to a Three-channel Dependent M/S coding (3D-M/S) method. 3D-M/S perform sum and difference coding based on three channels instead of conventional two channels, and corresponding transform matrixes are presented. Furthermore, a framework is proposed to enable 3D-M/S compress any number of audio channels. Experiment shows proposed method obtains 25.4% objective quality improvement comparing with independent channel coding, and only increases 11.3% complexity comparing with the 29.9% of PCA method.

In this paper, we present a novel single image super-resolution method. To simultaneously improve the resolution and perceptual image quality, we bring forward a practical solution combining manifold regularization and sparse support regression. The main contribution of this paper is twofold. Firstly, a mapping function from low resolution (LR) patches to high-resolution (HR) patches will be learned by a local regression algorithm called sparse support regression, which can be constructed from the support bases of the LR-HR dictionary. Secondly, we propose to preserve the geometrical structure of the image patch dictionary, which is critical for reducing the artifacts and obtaining better visual quality. Experimental results demonstrate that the proposed method produces high quality results both quantitatively and perceptually.

High efficiency audio compression is the basic technology in audio involved multimedia application. Down mixing and parametric coding are efficient coding scheme with widely applications in some up to date audio codecs such as PS in EAAC+ and MPEG-Surround, and PCA stereo coding followed this idea to map two channels to one channel with maximum energy and parameterize the secondary channel. This paper investigates the conventional PCA method performance under general stereo model with multiple sound sources and different directions, and then proposes a Polar Coordinate based PCA (PC-PCA) stereo coding method. It has been proved that when multiple sound sources exist with different directions, proposed method is better than the conventional PCA method in certain conditions. A stereo codec based on PC-PCA has also been proposed to validate the performance improvement of proposed method.

This paper proposes a bottom-up audio attention model based on spatial audio cues and sub band energy change for unsupervised event detection in stereo audio surveillance. Firstly, the spatial audio parameter Interaural Level Difference (ILD) is extracted to calculate and represent the attention events, which are caused by rapid moving sound source. Then the sub band energy change is computed to present the salient energy distribution change in frequency domain. At last, an environment adaptive normalization is used to assess the normalized attention level. Experimental results demonstrate that the proposed audio attention model is effective for audio surveillance event detection.