Multimodal personality analysis aims to identify and express human personality traits in videos. However, RNN and its variants have a limited ability to learn long-term temporal dependencies and existing methods neglect bimodal association features. Based on the fact that visual modalities play a dominant role in this task. Therefore, we propose a personality prediction method catering to learning and fusing intra-modal and intermodal feature dynamics. We first utilize pretrained models’ encoders to extract unimodal spatial scene features from videos. Then, we use xLSTM to capture sequence dependencies between different scene frames used as scene features. Meanwhile, we design a graph relational transformer network to learn longer intra-modal temporal interaction in three unimodal spatial features. Then, we calculate the similarity scores between visual and audio or text features as bimodal association features. Second, we design a multimodal attention feature fusion module to determine the contribution of each feature and aggregate these features. Finally, the MLP model is trained and used to predict scores for personality traits. Experiments on two benchmark datasets demonstrate that our method outperforms the existing methods and achieves state-of-the-art performance. Our code is available at https://github.com/RongquanWang/MP-PMGRT.

Abstract Multimodal personality analysis targets accurately detecting personality traits by incorporating related multimodal information. However, existing methods focus on unimodal features while overlooking the bimodal association features crucial for this interdisciplinary task. Therefore, we propose a multimodal personality prediction framework based on an adaptive graph transformer network and multi‐task learning. Firstly, we utilize pre‐trained models to learn specific representations from different modalities. Here, we employ pre‐trained multimodal models' encoders as the backbones of the modality‐specific extraction methods to mine unimodal features. Specifically, we introduce a novel adaptive graph transformer network to mine personality‐related bimodal association features. This network effectively learns higher‐order temporal dependencies based on relational graphs and emphasizes more significant features. Furthermore, we utilize a multimodal channel attention residual fusion module to obtain the fused features, and we propose a multimodal and unimodal joint learning regression head to learn and predict scores for personality traits. We design a multi‐task loss function to enhance the robustness and accuracy of personality prediction. Experimental results on the two benchmark datasets demonstrate the effectiveness of our framework, which outperforms the state‐of‐the‐art methods. The code is available at https://github.com/RongquanWang/PPF-AGTNMTL .

MOTIVATION: A critical issue in drug benefit-risk assessment is to determine the frequency of side effects, which is performed by randomized controlled trails. Computationally predicted frequencies of drug side effects can be used to effectively guide the randomized controlled trails. However, it is more challenging to predict drug side effect frequencies, and thus only a few studies cope with this problem. RESULTS: In this work, we propose a neighborhood-regularization method (NRFSE) that leverages multiview data on drugs and side effects to predict the frequency of side effects. First, we adopt a class-weighted non-negative matrix factorization to decompose the drug-side effect frequency matrix, in which Gaussian likelihood is used to model unknown drug-side effect pairs. Second, we design a multiview neighborhood regularization to integrate three drug attributes and two side effect attributes, respectively, which makes most similar drugs and most similar side effects have similar latent signatures. The regularization can adaptively determine the weights of different attributes. We conduct extensive experiments on one benchmark dataset, and NRFSE improves the prediction performance compared with five state-of-the-art approaches. Independent test set of post-marketing side effects further validate the effectiveness of NRFSE. AVAILABILITY AND IMPLEMENTATION: Source code and datasets are available at https://github.com/linwang1982/NRFSE or https://codeocean.com/capsule/4741497/tree/v1.

A critical issue of drug risk-benefit evaluation is to determine the frequencies of drug side effects. Randomized controlled trail is the conventional method for obtaining the frequencies of side effects, while it is laborious and slow. Therefore, it is necessary to guide the trail by computational methods. Existing methods for predicting the frequencies of drug side effects focus on modeling drug-side effect interaction graph. The inherent disadvantage of these approaches is that their performance is closely linked to the density of interactions but which is highly sparse. More importantly, for a cold start drug that does not appear in the training data, such methods cannot learn the preference embedding of the drug because there is no link to the drug in the interaction graph. In this work, we propose a new method for predicting the frequencies of drug side effects, DSGAT, by using the drug molecular graph instead of the commonly used interaction graph. This leads to the ability to learn embeddings for cold start drugs with graph attention networks. The proposed novel loss function, i.e. weighted $\varepsilon$-insensitive loss function, could alleviate the sparsity problem. Experimental results on one benchmark dataset demonstrate that DSGAT yields significant improvement for cold start drugs and outperforms the state-of-the-art performance in the warm start scenario. Source code and datasets are available at https://github.com/xxy45/DSGAT.

Compression of encrypted data draws much attention in recent years due to the security concerns in a service oriented environment such as cloud computing. We propose a scalable lossy compression scheme for images having their pixel value encrypted with a standard stream cipher. The encrypted data are simply compressed by transmitting a uniformly sub sampled portion of the encrypted data and some bit-planes of another uniformly sub sampled portion of the encrypted data. With a proposed content adaptive interpolation prediction method with side information, at the receiver side, a decoder performs content adaptive interpolation based on the decrypted partial information, where the received bit-plane information serves as the side information that reflects the image edge information, making the image reconstruction more precise. When more bit-planes are transmitted, higher quality of the decompressed image can be achieved. The experimental results show that our proposed scheme achieves much better performance than the existing lossy compression scheme for pixel value encrypted images, and also similar performance as the state-of-the-art lossy compression for pixel permutation based encrypted images. In addition, our proposed scheme has the following advantages: at the decoder side, no computationally intensive iteration and no additional public orthogonal matrix is needed. It works well for both smooth and texture-rich images.