Leveraging deep learning to explore the associations between voices and faces has attracted extensive research interest. Usually, this research is formalized by cross-modal verification, matching, and retrieval tasks. Most works rely on a single local or global optimization objective for learning, ignoring that testing scenarios may prefer different optimization objectives. For example, local objectives are more helpful for verification and matching tasks, while global objectives contribute more to retrieval. In this study, we proposed a learning framework based on local and global objectives to improve the generalizability of the learned representations. Firstly, we explored two ways of applying supervised contrastive loss (SCL) to learn voice-face representations. Secondly, we designed a contrastive-form global optimization objective, which shows better performance and training efficiency. Experiments on the VoxCeleb dataset demonstrate the effectiveness of our framework.

Learning associations between voices and faces have recently attracted much research interest. Previous studies mainly focused on improving loss function design for learning better representations, neglecting that informative inputs are the prerequisite of association findings and effective representation alignments. Based on this motivation, we proposed an unsupervised transformer-based learning framework, which fuses the knowledge of readily available single-modal expert models to learn better representations. Benefiting from higher-quality inputs, a simple noise contrastive estimation (NCE) loss is used for training. In addition, we proposed a statistical batch construction (SBC) strategy, which obtains high-quality negative samples during the unsupervised learning process. Experiments on the VoxCeleb1 dataset demonstrate the effectiveness of our framework. It yields SOTA results in voice-face verification, matching, and retrieval tasks. In the challenging gender-constrained matching tasks, it achieves over 81% (ACC), which is 5% higher than the previous.

Voice-face association learning (VFAL) aims to tap into the potential connections between voices and faces. Most studies currently address this problem in a supervised manner, which cannot exploit the wealth of unlabeled video data. To solve this problem, we propose an unsupervised learning framework: Self-Lifting (SL), which can use unlabeled video data for learning. This framework includes two iterative steps of "clustering" and "metric learning". In the first step, unlabeled video data is mapped into the feature space by a coarse model. Then unsupervised clustering is leveraged to allocate pseudo-label to each video. In the second step, the pseudo-label is used as supervisory information to guide the metric learning process, which produces the refined model. These two steps are performed alternately to lift the model's performance. Experiments show that our framework can effectively use unlabeled video data for learning. On the VoxCeleb dataset, our approach achieves SOTA results among the unsupervised methods and has competitive performance compared with the supervised competitors. Our code is released on Github.

This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus, and on this basis, explores a specified adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents. Our investigation begins with the thorough discus- sions upon the objective function of general Generative Adversarial Nets (GAN) architectures, and the training instability problem is proved to be highly relative with the special local distributions of conversational corpora. Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space, and the mathematical approximation of this energy-based distribution is finally found. Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN. Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.

Sonoporation, which typically employs acoustic cavitation microbubbles, can enhance the permeability of the cell membrane, allowing foreign matter to enter cells across the natural barriers. However, the diameter nonuniformity and random distribution of microbubbles make it difficult to achieve controllable and high-efficiency sonoporation, while complex extern acoustic driving system also limits its applicability. Herein, we demonstrate a low-cost, expandable, and portable acoustofluidic device for cell sonoporation using acoustic streaming generated by oscillating sharp edges. The streaming-induced high shear forces can (i) quickly trap target cells at the tip of sharp edges and (ii) transiently modulate the permeability of the cell membrane, which is utilized to perform cell sonoporation events. Using our device, sonoporation is successfully achieved in a microbubble-free manner, with a sonoporation efficiency of more than 90%. Furthermore, our acoustic driving system is designed around the open-source Arduino prototyping platform due to its extendibility and portability. In addition to these benefits, our acoustofluidic device is simple to fabricate and operate, and it can work at relatively low frequency (4.6 kHz). All these advantages make our novel cell sonoporation device invaluable for many biological and biomedical applications such as drug delivery and gene transfection.

MOTIVATION: Owing to its importance in both basic research (such as molecular evolution and protein attribute prediction) and practical application (such as timely modeling the 3D structures of proteins targeted for drug development), protein remote homology detection has attracted a great deal of interest. It is intriguing to note that the profile-based approach is promising and holds high potential in this regard. To further improve protein remote homology detection, a key step is how to find an optimal means to extract the evolutionary information into the profiles. RESULTS: Here, we propose a novel approach, the so-called profile-based protein representation, to extract the evolutionary information via the frequency profiles. The latter can be calculated from the multiple sequence alignments generated by PSI-BLAST. Three top performing sequence-based kernels (SVM-Ngram, SVM-pairwise and SVM-LA) were combined with the profile-based protein representation. Various tests were conducted on a SCOP benchmark dataset that contains 54 families and 23 superfamilies. The results showed that the new approach is promising, and can obviously improve the performance of the three kernels. Furthermore, our approach can also provide useful insights for studying the features of proteins in various families. It has not escaped our notice that the current approach can be easily combined with the existing sequence-based methods so as to improve their performance as well. AVAILABILITY AND IMPLEMENTATION: For users' convenience, the source code of generating the profile-based proteins and the multiple kernel learning was also provided at http://bioinformatics.hitsz.edu.cn/main/~binliu/remote/

The human-generated question-answer pairs in the Web social communities are of great value for the research of automatic question-answering technique. Due to the large amount of noise information involved in such corpora, it is still a problem to detect the answers even though the questions are exactly located. Quantifying the semantic relevance between questions and their candidate answers is essential to answer detection in social media corpora. Since both the questions and their answers usually contain a small number of sentences, the relevance modeling methods have to overcome the problem of word feature sparsity. In this article, the deep learning principle is introduced to address the semantic relevance modeling task. Two deep belief networks with different architectures are proposed by us to model the semantic relevance for the question-answer pairs. According to the investigation of the textual similarity between the community-driven question-answering (cQA) dataset and the forum dataset, a learning strategy is adopted to promote our models’ performance on the social community corpora without hand-annotating work. The experimental results show that our method outperforms the traditional approaches on both the cQA and the forum corpora.

Although Support Vector Machines(SVM) succeed in classifying several image databases using image descriptors proposed in the literature, no single descriptor can be optimal for general object categorization. This paper describes a novel framework to learn the optimal combination of kernels corresponding to multiple image descriptors before SVM training, leading to solve a quadratic programming problem efficiently. Our framework takes into account the variation of kernel matrix and imbalanced dataset, which are common in real world image categorization tasks. Experimental results on Graz-01 and Caltech-101 image databases show the effectiveness and robustness of our algorithm.

A Chinese sign language synthesis system is proposed in this paper. H-Anim standard of VRML is adopted to describe the avatar. The traditional reverse kinematics resolution algorithm is employed and constraints of the elbow position are considered to make the editing of hand shapes and arm gestures efficient. Various interpolation methods are used for the motion of joints that have different degrees-of-freedom. The Chinese sign language synthesis system is implemented as a Web service which can be accessed by 3rd-party applications. It has been applied in sign language teaching and sign language translation.