Existing user simulators based on prompting to role-play or SFT are generally confined to imitating users' textual utterances, without adequately considering the multi-faceted cognitive processes that underlie human decision-making during interactions.To facilitate better alignment with real human thinking patterns, we construct the LMSYS-UserThinking dataset, in which we augment 51k human-LLM conversations by reconstructing the user's inner reasoning both during and at the end of each dialogue.Furthermore, to enhance controllability and situational coherence, we introduce scenario settings that describe the global context and user goals throughout multi-turn conversations.Using this dataset, we train user simulators called ThinkingUS on different base models.We evaluate our approach from both offline and online user simulation perspectives, ultimately demonstrating its effectiveness.

Facial Kinship Verification (FKV) aims at automatically determining whether two subjects have a kinship relation based on human faces. It has potential applications in finding missing children and social media analysis. Traditional FKV faces challenges as it is vulnerable to spoof attacks and raises privacy issues. In this paper, we explore for the first time the FKV by analyzing cardiac activity through physiological signals, with a specific focus on remote Photoplethysmography (rPPG). rPPG signals are extracted from facial videos, resulting in a one-dimensional signal that measures the changes in visible light reflection emitted to and detected from the skin caused by the heartbeat. Specifically, in this paper, we employed a straightforward one-dimensional Convolutional Neural Network (1DCNN) with a 1DCNN-Attention module and kinship contrastive loss to learn the kin similarity from rPPGs. The network takes multiple rPPG signals extracted from various facial Regions of Interest (ROIs) as inputs. Additionally, the 1DCNN attention module is designed to learn and capture the discriminative kin features from feature embeddings. Finally, we demonstrate the feasibility of rPPG to detect kinship with the experiment evaluation on the UvANEMO Smile Database from different kin relations.

Facial kinship verification refers to automatically determining whether two people have a kin relation from their faces. It has become a popular research topic due to potential practical applications. Over the past decade, many efforts have been devoted to improving the verification performance from human faces only while lacking other biometric information, for example, speaking voice. In this article, to interpret and benefit from multiple modalities, we propose for the first time to combine human faces and voices to verify kinship, which we refer it as the audio-visual kinship verification study. We first establish a comprehensive audio-visual kinship dataset that consists of familial talking facial videos under various scenarios, called TALKIN-Family. Based on the dataset, we present the extensive evaluation of kinship verification from faces and voices. In particular, we propose a deep-learning-based fusion method, called unified adaptive adversarial multimodal learning (UAAML). It consists of the adversarial network and the attention module on the basis of unified multimodal features. Experiments show that audio (voice) information is complementary to facial features and useful for the kinship verification problem. Furthermore, the proposed fusion method outperforms baseline methods. In addition, we also evaluate the human verification ability on a subset of TALKIN-Family. It indicates that humans have higher accuracy when they have access to both faces and voices. The machine-learning methods could effectively and efficiently outperform the human ability. Finally, we include the future work and research opportunities with the TALKIN-Family dataset.

The goal of Facial Kinship Verification (FKV) is to automatically determine whether two individuals have a kin relationship or not from their given facial images or videos. It is an emerging and challenging problem that has attracted increasing attention due to its practical applications. Over the past decade, significant progress has been achieved in this new field. Handcrafted features and deep learning techniques have been widely studied in FKV. The goal of this paper is to conduct a comprehensive review of the problem of FKV. We cover different aspects of the research, including problem definition, challenges, applications, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. In retrospect of what has been achieved so far, we identify gaps in current research and discuss potential future research directions.

Images captured behind glass are often contaminated by the undesired reflection that hinders other computer vision tasks. Due to ubiquitous glasses, the removal of undesired reflection becomes more important. Reflection is determined by imaging system parameters such as the transparency, color of the glasses, the position of the camera. In this letter, we propose a novel generative model based on a generative adversarial network (GAN), where an Encoder extracts system parameters from input images and encodes them succinctly as a distribution and a Generator emulates the optical process of reflection. Separating the optical process of the system parameters allows our model to cope with diversified real-life scenarios and avoid the mode collapse phenomenon. Based on the generative model, we introduce a reflection removal model that simultaneously extracts the original image, the reflection layer, and the encoded parameters from single image input. Computational results of real data show that our approach outperforms existing approaches.

A Dialogue State Tracker (DST) is a core component of a modular task-oriented dialogue system.Tremendous progress has been made in recent years.However, the major challenges remain.The state-of-the-art accuracy for DST is below 50% for a multi-domain dialogue task.A learnable DST for any new domain requires a large amount of labeled indomain data and training from scratch.In this paper, we propose a Meta-Reinforced Multi-Domain State Generator (MERET).Our first contribution is to improve the DST accuracy.We enhance a neural model based DST generator with a reward manager, which is built on policy gradient reinforcement learning (R-L) to fine-tune the generator.With this change, we are able to improve the joint accuracy of DST from 48.79% to 50.91% on the Multi-WOZ corpus.Second, we explore to train a DST meta-learning model with a few domains as source domains and a new domain as target domain.We apply the model-agnostic metalearning (MAML) algorithm to DST and the obtained meta-learning model is used for new domain adaptation.Our experimental results show this solution is able to outperform the traditional training approach with extremely less training data in target domain.