Lizi Liao

dblp:149/1249 · DBLP profile ↗
← Back
27ranked-venue papers in the field
7as first author
22since 2021 · last 2026
0000-0002-9973-3305ORCID · verified

Domains — venue-derived; a paper can count in several

Information Retrieval & Web Search · 19 (3 first)Database Systems & Data Management · 5 (3 first)Data Mining & Knowledge Discovery · 3 (1 first)
YearPublicationVenuePosition
2026 VideoCreator: An Agentic System for Multi-turn Video Production
abstract
Recent advances in video generation models enable visually compelling single clips. However, real-world video creation is inherently continuous and iterative: creators refine content over multiple rounds while maintaining narrative, style, and entity consistency. Existing standalone generators are largely stateless and lack memory of previously generated segments, making it difficult to produce a coherent and consistent video project. To address this gap, we present VideoCreator, a unified video agent that integrates generation and understanding with a project-level memory system. VideoCreator leverages understanding capabilities to perform fine-grained analysis of newly produced content and uses persistent memory to retain and reuse prior context across turns, enabling continuous multi-round creation with consistency throughout the project. The code is in https://github.com/chrisx599/VideoCreator.
Zhengyang Liang, Cathal Gurrin, Nicu Sebe, Lizi Liao
ICMR5
2026 Generation-Augmented Video Corpus Moment Retrieval
abstract
Video Corpus Moment Retrieval (VCMR) requires models to efficiently retrieve and precisely locate specific moments relevant to natural language queries within a massive, untrimmed video corpus. However, existing discriminative approaches typically rely on shallow visual-textual feature matching mechanisms, which often struggle to capture fine-grained semantic differences. To address this limitation, we propose Video-GAR, a novel framework that reframes the conventional retrieval task from superficial matching to generative understanding, positing that the capability for query reconstruction evidences deep semantic comprehension. Specifically, Video-GAR orchestrates three synergistic components: To overcome the computational efficiency bottleneck, we construct a Bi-Mamba backbone that leverages the linear complexity of state-space models for efficient global context modeling. Building on these representations, we introduce a generation-augmented fusion module, in which a training-only decoder acts as a semantic regularizer to implicitly calibrate cross-modal attention without increasing inference overhead. Finally, to ensure fine-grained precision, we propose a boundary-aware localization strategy that integrates boundary modeling with categorical supervision. Experiments on two benchmark datasets demonstrate that Video-GAR significantly improves retrieval and localization accuracy while maintaining outstanding inference speed.
Mingjin Kuai, Qianyin Xiao, Juncheng Li 0006, Lizi Liao, Wei Ji 0008
SIGIR5
2026 Subgraph-Centric Multi-Agent Reinforcement Learning for Multi-Hop Knowledge Graph Reasoning
abstract
Multi-hop Knowledge Graph Reasoning (KGR) seeks to identify accurate answers within Knowledge Graphs (KGs) via multi-step reasoning, predominantly utilizing reinforcement learning (RL) to enhance the efficiency of the reasoning process. Unlike traditional Knowledge Graph Embedding (KGE) methods, RL-based approaches offer superior interpretability. However, these methods often underperform due to two critical limitations: (1) their over-reliance on Horn rules for reasoning paths, which restricts their expressive power; and (2) inadequate utilization of reasoning states during the process. To address these issues, we propose a novel RL-based framework, RAR, which shifts focus from individual paths to subgraph structures for more robust predictions. RAR frames the retrieval of reasoning subgraphs from the KG as a Markov Decision Process (MDP) and incorporates a subgraph retriever. To efficiently explore the extensive subgraph space, we integrate multi-agent RL to enhance the retriever's capabilities. Additionally, RAR features an advanced analyst module that meticulously examines reasoning states. These modules function iteratively: the retriever expands the subgraph, followed by the analyst module's in-depth analysis. The insights gained are then used to inform subsequent retrieval steps. Ultimately, the predicted scores from both modules are synthesized to produce more precise posterior scores. Experimental results across multiple datasets demonstrate RAR's efficacy, showcasing a notable improvement over existing state-of-the-art RL-based KGR methods.
Tao He 0014, Zerui Chen, Lizi Liao, Yixin Cao 0002, Yuanxing Liu 0001, Wei Tang 0015, Xun Mao, Ming Liu 0004, Bing Qin 0001
IEEE Trans. Knowl. Data Eng.3
2025 DistillCaps: Enhancing Audio-Language Alignment in Captioning via Retrieval-Augmented Knowledge Distillation
abstract
Automated audio captioning (AAC) benefits from incorporating external context to interpret complex sounds, but doing so with retrieval-augmented generation (RAG) at inference is sometimes infeasible due to data availability or incurs significant latency and complexity. We propose DistillCaps, a novel training-time framework that leverages RAG to guide knowledge distillation for improved audio-language alignment, while lessening the reliance on retrieval during inference. In our framework, a RAG-equipped teacher model retrieves relevant textual information (e.g., similar captions) for each audio clip and uses it for training to generate context-enriched captions. Simultaneously, a student model is trained to imitate this teacher, learning to produce high-quality captions from audio alone. We further introduce a Fast Fourier Transform (FFT) adapter in the audio encoder to inject frequency-domain features, enhancing the quality of audio representations before feeding them into the language model. The result is an efficient captioning model that retains RAG's contextual benefits without its deployment overhead. On standard AAC benchmarks (AudioCaps and Clotho), DistillCaps achieves performance competitive with or exceeding prior RAG-based systems despite using no retrieval at test time. Notably, our distilled model matches state-of-the-art captioning results under real-time settings, and when optionally allowing retrieval, it even outperforms previous models by up to 4% on the Clotho benchmark on the in-distribution setting, demonstrating the effectiveness of RAG-guided distillation for audio-language alignment. Code and dataset are available here. https://github.com/pgthinh/DistillCaps.
Thinh Pham, Nghiem Tuong Diep, Lizi Liao, Binh T. Nguyen 0001
CIKM3
2025 Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planning
abstract
Recent advancements in dialogue policy planning have focused on optimizing system agent policies to achieve predefined goals, emphasizing strategy design, trajectory acquisition, and training efficiency.However, these approaches often overlook the critical role of user characteristics, which are essential in real-world scenarios like conversational search and recommendation, where interactions must adapt to individual user traits such as personality, preferences, and goals.To address this gap, we conduct a comprehensive study using task-specific user personas to evaluate dialogue policy planning under diverse user behaviors.Our analysis, based on these user profiles, reveals significant shortcomings in existing approaches, underscoring the necessity for user-tailored dialogue policies.Building on these insights, we propose the User-Tailored Dialogue Policy Planning (UDP) framework, which integrates an Intrinsic User World Model to capture user traits and feedback.UDP operates in three stages: (1) User Persona Portraying, employing a diffusion model to dynamically infer user profiles; (2) User Feedback Anticipating, using a Brownian Bridge-inspired mechanism to predict user reactions; and (3) User-Tailored Policy Planning, synthesizing these elements to optimize response strategies.To enhance robustness, we introduce an active learning approach that prioritizes challenging user personas during training.Extensive experiments across benchmarks, including both collaborative and non-collaborative settings, demonstrate UDP's effectiveness in learning user-specific dialogue strategies.Results confirm the framework's utility, highlighting its robustness, adaptability, and potential to advance user-centric dialogue systems.
Tao He 0014, Lizi Liao, Ming Liu 0004, Bing Qin 0001
SIGIR2
2025 Retrieval Augmented Generation for Dynamic Graph Modeling
abstract
Modeling dynamic graphs, such as those found in social networks, recommendation systems, and e-commerce platforms, is crucial for capturing evolving relationships and delivering relevant insights over time. Traditional approaches primarily rely on graph neural networks with temporal components or sequence generation models, which often focus narrowly on the historical context of target nodes. This limitation restricts the ability to adapt to new and emerging patterns in dynamic graphs. To address this challenge, we propose a novel framework, Retrieval-Augmented Generation for Dy namic Graph modeling (RAG4DyG ), which enhances dynamic graph predictions by incorporating contextually and temporally relevant examples from broader graph structures. Our approach includes a time- and context-aware contrastive learning module to identify high-quality demonstrations and a graph fusion strategy to effectively integrate these examples with historical contexts. The proposed framework is designed to be effective in both transductive and inductive scenarios, ensuring adaptability to previously unseen nodes and evolving graph structures. Extensive experiments across multiple real-world datasets demonstrate the effectiveness of RAG4DyG in improving predictive accuracy and adaptability for dynamic graph modeling. The code and datasets are publicly available at https://github.com/YuxiaWu/RAG4DyG.
Yuxia Wu, Lizi Liao, Yuan Fang 0001
SIGIR2
2025 Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark
abstract
Empathetic Response Generation (ERG) is one of the key tasks of the affective computing area, which aims to produce emotionally nuanced and compassionate responses to user's queries. However, existing ERG research is predominantly confined to the singleton text modality, limiting its effectiveness since human emotions are inherently conveyed through multiple modalities. To combat this, we introduce an avatar-based Multimodal ERG (MERG) task, entailing rich text, speech, and facial vision information. We first present a large-scale high-quality benchmark dataset, AvaMERG, which extends traditional text ERG by incorporating authentic human speech audio and dynamic talking-face avatar videos, encompassing a diverse range of avatar profiles and broadly covering various topics of real-world scenarios. Further, we deliberately tailor a system, named Empatheia, for MERG. Built upon a Multimodal Large Language Model (MLLM) with multimodal encoder, speech and avatar generators, Empatheia performs end-to-end MERG, with Chain-of-Empathetic reasoning mechanism integrated for enhanced empathy understanding and reasoning.Finally, we devise a list of empathetic-enhanced tuning strategies, strengthening the capabilities of emotional accuracy and content, avatar-profile consistency across modalities. Experimental results on AvaMERG data demonstrate that Empatheia consistently shows superior performance than baseline methods on both textual ERG and MERG. All data and code are open at https://AvaMERG.github.io/.
Han Zhang 0035, Zixiang Meng, Meng Luo 0010, Hong Han 0001, Lizi Liao, Erik Cambria, Hao Fei 0001
WWW5
2025 Enriching Responses with Crowd-Sourced Knowledge for Task-Oriented Conversational Agents
abstract
Task-oriented conversational agents strive to aid users across various tasks by concentrating on generating suitable responses to guarantee successful task accomplishment. Nonetheless, several factors have a substantial influence on user contentment beyond task fulfillment, requiring further investigation. Within this work, we aim to analyze diverse behavioral patterns of conversational agents with the goal of enhancing user satisfaction. Our findings lead to the exploration of three different enriched response generation schemes: EnRG-ATT, EnRG-TIP, and EnRG-SIM. Specifically, EnRG-ATT is designed to integrate the model's capabilities with a dual attention mechanism across two distinct modalities of external resources. It employs a pair of gates to regulate the utilization of such sources efficiently. More elegantly, we introduce EnRG-TIP, which simplifies response enrichment as a sequence prediction problem and exploits the pre-trained language model to capture user tips related to the conversation. Moreover, building on the efficiency of grounding on similar responses, EnRG-SIM further enhances response generation by inserting similar responses into the training sequences, to direct the pre-trained model's attention towards this additional knowledge. Our comprehensive experiments demonstrate that our three proposed methods not only achieve good task completion but also generate responses that yield higher user satisfaction.
Zhaohui Wei, Lizi Liao, Xinguang Xiang, Xiaoyu Du 0002
ACM Trans. Intell. Syst. Technol.2
2025 Proactive Conversational AI: A Comprehensive Survey of Advancements and Opportunities
abstract
Dialogue systems are designed to offer human users social support or functional services through natural language interactions. Traditional conversation research has put significant emphasis on a system’s response-ability, including its capacity to understand dialogue context and generate appropriate responses. However, the key element of proactive behavior—a crucial aspect of intelligent conversations—is often overlooked in these studies. Proactivity empowers conversational agents to lead conversations towards achieving pre-defined targets or fulfilling specific goals on the system side. Proactive dialogue systems are equipped with advanced techniques to handle complex tasks, requiring strategic and motivational interactions, thus representing a significant step towards artificial general intelligence. Motivated by the necessity and challenges of building proactive dialogue systems, we provide a comprehensive review of various prominent problems and advanced designs for implementing proactivity into different types of dialogue systems, including open-domain dialogues, task-oriented dialogues, and information-seeking dialogues. We also discuss real-world challenges that require further research attention to meet application needs in the future, such as proactivity in dialogue systems that are based on large language models, proactivity in hybrid dialogues, evaluation protocols and ethical considerations for proactive dialogue systems. By providing a quick access and overall picture of the proactive dialogue systems domain, we aim to inspire new research directions and stimulate further advancements towards achieving the next level of conversational AI capabilities, paving the way for more dynamic and intelligent interactions within various application domains.
Yang Deng 0002, Lizi Liao, Wenqiang Lei, Grace Hui Yang, Wai Lam, Tat-Seng Chua
ACM Trans. Inf. Syst.2
2025 Revisiting Conversation Discourse for Dialogue Disentanglement
abstract
Dialogue disentanglement aims to detach the chronologically ordered utterances into several independent sessions. Conversation utterances are essentially organized and described by the underlying discourse, and thus dialogue disentanglement requires the full understanding and harnessing of the intrinsic discourse attribute. In this article, we propose enhancing dialogue disentanglement by taking full advantage of the dialogue discourse characteristics. First of all, in feature encoding stage , we construct the heterogeneous graph representations to model the various dialogue-specific discourse structural features, including the static speaker-role structures (i.e., speaker-utterance and speaker-mentioning structure) and the dynamic contextual structures (i.e., the utterance-distance and partial-replying structure). We then develop a structure-aware framework to integrate the rich structural features for better modeling the conversational semantic context. Second, in model learning stage , we perform optimization with a hierarchical ranking loss mechanism, which groups dialogue utterances into different discourse levels and carries training covering pairwise and session-wise levels hierarchically. Third, in inference stage , we devise an easy-first decoding algorithm, which performs utterance pairing under the easy-to-hard manner with a global context, breaking the constraint of traditional sequential decoding order. On two benchmark datasets, our overall system achieves new state-of-the-art performances on all evaluations. In-depth analyses further demonstrate the efficacy of each proposed idea and also reveal how our methods help advance the task. Our work has great potential to facilitate broader multi-party multi-thread dialogue applications.
Bobo Li 0001, Hao Fei 0001, Fei Li 0021, Shengqiong Wu, Lizi Liao, Yinwei Wei, Tat-Seng Chua, Donghong Ji
ACM Trans. Inf. Syst.5
2024 Broadening the View: Demonstration-augmented Prompt Learning for Conversational Recommendation
abstract
Conversational Recommender Systems (CRSs) leverage natural language dialogues to provide tailored recommendations.Traditional methods in this field primarily focus on extracting user preferences from isolated dialogues.It often yields responses with a limited perspective, confined to the scope of individual conversations.Recognizing the potential in collective dialogue examples, our research proposes an expanded approach for CRS models, utilizing selective analogues from dialogue histories and responses to enrich both generation and recommendation processes.This introduces significant research challenges, including: (1) How to secure high-quality collections of recommendation dialogue exemplars?(2) How to effectively leverage these exemplars to enhance CRS models?To tackle these challenges, we introduce a novel Demonstrationenhanced Conversational Recommender System (DCRS), which aims to strengthen its understanding on the given dialogue contexts by retrieving and learning from demonstrations.In particular, we first propose a knowledge-aware contrastive learning method that adeptly taps into the mentioned entities and the dialogue's contextual essence for pretraining the demonstration retriever.Subsequently, we further develop two adaptive demonstrationaugmented prompt learning approaches, involving contextualized prompt learning and knowledge-enriched prompt learning, to bridge the gap between the retrieved demonstrations and the two end tasks of CRS, i.e., response generation and item recommendation, respectively.Rigorous evaluations on two established benchmark datasets underscore DCRS's superior performance over existing CRS methods in both item recommendation and response generation 1 .
Huy Dao, Yang Deng 0002, Dung D. Le, Lizi Liao
SIGIR4
2024 Towards Human-centered Proactive Conversational Agents
abstract
Recent research on proactive conversational agents (PCAs) mainly focuses on improving the system's capabilities in anticipating and planning action sequences to accomplish tasks and achieve goals before users articulate their requests. This perspectives paper highlights the importance of moving towards building human-centered PCAs that emphasize human needs and expectations, and that considers ethical and social implications of these agents, rather than solely focusing on technological capabilities. The distinction between a proactive and a reactive system lies in the proactive system's initiative-taking nature. Without thoughtful design, proactive systems risk being perceived as intrusive by human users. We address the issue by establishing a new taxonomy concerning three key dimensions of human-centered PCAs, namely Intelligence, Adaptivity, and Civility. We discuss potential research opportunities and challenges based on this new taxonomy upon the five stages of PCA system construction. This perspectives paper lays a foundation for the emerging area of conversational information retrieval research and paves the way towards advancing human-centered proactive conversational systems.
Yang Deng 0002, Lizi Liao, Zhonghua Zheng, Grace Hui Yang, Tat-Seng Chua
SIGIR2
2024 On the Feasibility of Simple Transformer for Dynamic Graph Modeling
abstract
Dynamic graph modeling is crucial for understanding complex structures in web graphs, spanning applications in social networks, recommender systems, and more. Most existing methods primarily emphasize structural dependencies and their temporal changes. However, these approaches often overlook detailed temporal aspects or struggle with long-term dependencies. Furthermore, many solutions overly complicate the process by emphasizing intricate module designs to capture dynamic evolutions. In this work, we harness the strength of the Transformer's self-attention mechanism, known for adeptly handling long-range dependencies in sequence modeling. Our approach offers a simple Transformer model, called SimpleDyG, tailored for dynamic graph modeling without complex modifications. We re-conceptualize dynamic graphs as a sequence modeling challenge and introduce a novel temporal alignment technique. This technique not only captures the inherent temporal evolution patterns within dynamic graphs but also streamlines the modeling process of their evolution. To evaluate the efficacy of SimpleDyG, we conduct extensive experiments on four real-world datasets from various domains. The results demonstrate the competitive performance of SimpleDyG in comparison to a series of state-of-the-art approaches despite its simple design.
Yuxia Wu, Yuan Fang 0001, Lizi Liao
WWW3
2023 Real: A Representative Error-Driven Approach for Active Learning
Cheng Chen 0050, Yong Wang 0021, Lizi Liao, Yueguo Chen, Xiaoyong Du 0001
ECML/PKDD (1)3
2023 Proactive Conversational Agents in the Post-ChatGPT World
abstract
ChatGPT and similar large language model (LLM) based conversational agents have brought shock waves to the research world. Although astonished by their human-like performance, we find they share a significant weakness with many other existing conversational agents in that they all take a passive approach in responding to user queries. This limits their capacity to understand the users and the task better and to offer recommendations based on a broader context than a given conversation. Proactiveness is still missing in these agents, including their ability to initiate a conversation, shift topics, or offer recommendations that take into account a more extensive context. To address this limitation, this tutorial reviews methods for equipping conversational agents with proactive interaction abilities.
Lizi Liao, Grace Hui Yang, Chirag Shah 0001
SIGIR1
2023 Proactive Conversational Agents
abstract
Conversational agents, or commonly known as dialogue systems, have gained escalating popularity in recent years. Their widespread applications support conversational interactions with users and accomplishing various tasks as personal assistants. However, one key weakness in existing conversational agents is that they only learn to passively answer user queries via training on pre-collected and manually-labeled data. Such passiveness makes the interaction modeling and system-building process relatively easier, but it largely hinders the possibility of being human-like hence lowering the user engagement level. In this tutorial, we introduce and discuss methods to equip conversational agents with the ability to interact with end users in a more proactive way. This three-hour tutorial is divided into three parts and includes two interactive exercises. It reviews and presents recent advancements on the topic, focusing on automatically expanding ontology space, actively driving conversation by asking questions or strategically shifting topics, and retrospectively conducting response quality control.
Lizi Liao, Grace Hui Yang, Chirag Shah 0001
WSDM1
2022 Structured and Natural Responses Co-generation for Conversational Search
abstract
Generating fluent and informative natural responses while main- taining representative internal states for search optimization is critical for conversational search systems. Existing approaches ei- ther 1) predict structured dialog acts first and then generate natural response; or 2) map conversation context to natural responses di- rectly in an end-to-end manner. Both kinds of approaches have shortcomings. The former suffers from error accumulation while the semantic associations between structured acts and natural re- sponses are confined in single direction. The latter emphasizes generating natural responses but fails to predict structured acts. Therefore, we propose a neural co-generation model that gener- ates the two concurrently. The key lies in a shared latent space shaped by two informed priors. Specifically, we design structured dialog acts and natural response auto-encoding as two auxiliary tasks in an interconnected network architecture. It allows for the concurrent generation and bidirectional semantic associations. The shared latent space also enables asynchronous reinforcement learn- ing for further joint optimization. Experiments show that our model achieves significant performance improvements.
Chenchen Ye 0001, Lizi Liao, Fuli Feng, Wei Ji 0008, Tat-Seng Chua
SIGIR2
2022 Topic-Guided Conversational Recommender in Multiple Domains
abstract
Conversational systems have recently attracted significant attention. Both the research community and industry believe that it will exert huge impact on human-computer interaction, and specifically, the IR/RecSys community has begun to explore Conversational Recommendation. In real-life scenarios, such systems are often urgently needed in helping users accomplishing different tasks under various situations. However, existing works still face several shortcomings: (1) Most efforts are largely confined in single task setting. They fall short of hands in handling tasks across domains. (2) Aside from soliciting user preference from dialogue history, a conversational recommender naturally has access to the back-end data structure which should be fully leveraged to yield good recommendations. In this paper, we thus present a Topic-guided Conversational Recommender (TCR) which is specifically designed for the multi-domain setting. It augments the sequence-to-sequence (seq2seq) models with a neural latent topic component to better guide the response generation. To better leverage the dialogue history and the back-end data structure, we adopt a graph convolutional network (GCN) to model the relationships between different recommendation candidates while also capture the match between candidates and the dialogue history. We then seamlessly combine these two parts with the idea of pointer networks. We perform extensive evaluation on a large-scale task-oriented multi-domain dialogue dataset and the results show that our method achieves superior performance as compared to a wide range of baselines.
Lizi Liao, Ryuichi Takanobu, Yunshan Ma 0002, Xun Yang 0001, Minlie Huang, Tat-Seng Chua
IEEE Trans. Knowl. Data Eng.1
2022 A Patience-Aware Recommendation Scheme for Shared Accounts on Mobile Devices
abstract
As sharing of accounts is quite common among family members or roommates, the design of efficient recommender schemes for shared accounts has raised much attention recently. Generally speaking, after each login, it is essential for a recommender system to identify the current user behind and leverage this information to make recommendations. One naive approach is first to identify the identity of the current user and then make recommendations. However, this two-stage based approach may not achieve satisfactory performance. The key is that the recommended items favoring identifying users in the first stage may not be interesting to the users, which can deplete the user's patience quickly and cause early termination of users. To address the problem, we propose a novel recommendation scheme that makes a tradeoff between recommending discriminating items (helpful for identifying the user) and recommending interesting ones to the user (helpful for increasing the number of clicks). Under this scheme, we develop a patience model to capture the user's dynamic patience level during the recommendation process. Moreover, considering the increasing popularity of mobile devices, we also incorporate mobile sensor data (i.e., angle, accelerometer, gyroscope, etc.) into our approach to further improve the performance of the system. We implemented the above system in an App on mobile devices and carried out extensive experiments. The results demonstrate that our proposed scheme significantly outperforms the existing state-of-the-art approaches.
Kaili Mao, Jianwei Niu 0002, Xuefeng Liu 0001, Shaojie Tang 0001, Lizi Liao, Tat-Seng Chua
IEEE Trans. Knowl. Data Eng.5
2021 Reproducibility Companion Paper: Knowledge Enhanced Neural Fashion Trend Forecasting
abstract
This companion paper supports the replication of the fashion trend forecasting experiments with the KERN (Knowledge Enhanced Recurrent Network) method that we presented in the ICMR 2020. We provide an artifact that allows the replication of the experiments using a Python implementation. The artifact is easy to deploy with simple installation, training and evaluation. We reproduce the experiments conducted in the original paper and obtain similar performance as previously reported. The replication results of the experiments support the main claims in the original paper.
Yunshan Ma 0002, Yujuan Ding, Xun Yang 0001, Lizi Liao, Wai Keung Wong, Tat-Seng Chua, Jinyoung Moon, Hong-Han Shuai
ICMR4
2021 MMConv: An Environment for Multimodal Conversational Search across Multiple Domains
abstract
Although conversational search has become a hot topic in both dialogue research and IR community, the real breakthrough has been limited by the scale and quality of datasets available. To address this fundamental obstacle, we introduce the Multimodal Multi-domain Conversational dataset (MMConv), a fully annotated collection of human-to-human role-playing dialogues spanning over multiple domains and tasks. The contribution is two-fold. First, beyond the task-oriented multimodal dialogues among user and agent pairs, dialogues are fully annotated with dialogue belief states and dialogue acts. More importantly, we create a relatively comprehensive environment for conducting multimodal conversational search with real user settings, structured venue database, annotated image repository as well as crowd-sourced knowledge database. A detailed description of the data collection procedure along with a summary of data structure and analysis is provided. Second, a set of benchmark results for dialogue state tracking, conversational recommendation, response generation as well as a unified model for multiple tasks are reported. We adopt the state-of-the-art methods for these tasks respectively to demonstrate the usability of the data, discuss limitations of current methods and set baselines for future studies.
Lizi Liao, Le Hong Long, Zheng Zhang 0020, Minlie Huang, Tat-Seng Chua
SIGIR1
2021 Multi-domain Dialogue State Tracking with Recursive Inference
abstract
Multi-domain dialogue state tracking (DST) is a critical component for monitoring user goals during the course of an interaction. Existing approaches have relied on dialogue history indiscriminately or updated on the most recent turns incrementally. However, in spite of modeling it based on fixed ontology or open vocabulary, the former setting violates the interactive and progressing nature of dialogue, while the later easily gets affected by the error accumulation conundrum. Here, we propose a Recursive Inference mechanism (ReInf) to resolve DST in multi-domain scenarios that call for more robust and accurate tracking capability. Specifically, our agent reversely reviews the dialogue history until the agent has pinpointed sufficient turns confidently for slot value prediction. It also recursively factors in potential dependencies among domains and slots to further solve the co-reference and value sharing problems. The quantitative and qualitative experimental results on the MultiWOZ 2.1 corpus demonstrate that the proposed ReInf not only outperforms the state-of-the-art methods, but also achieves reasonable turn reference and interpretable slot co-reference.
Lizi Liao, Tongyao Zhu, Le Hong Long, Tat-Seng Chua
WWW1
2020 Knowledge Enhanced Neural Fashion Trend Forecasting
abstract
Fashion trend forecasting is a crucial task for both academia andindustry. Although some efforts have been devoted to tackling this challenging task, they only studied limited fashion elements with highly seasonal or simple patterns, which could hardly reveal thereal fashion trends. Towards insightful fashion trend forecasting,this work focuses on investigating fine-grained fashion element trends for specific user groups. We first contribute a large-scale fashion trend dataset (FIT) collected from Instagram with extracted time series fashion element records and user information. Furthermore, to effectively model the time series data of fashion elements with rather complex patterns, we propose a Knowledge Enhanced Recurrent Network model (KERN) which takes advantage of the capability of deep recurrent neural networks in modeling time series data. Moreover, it leverages internal and external knowledgein fashion domain that affects the time-series patterns of fashion element trends. Such incorporation of domain knowledge further enhances the deep learning model in capturing the patterns of specific fashion elements and predicting the future trends. Extensive experiments demonstrate that the proposed KERN model can effectively capture the complicated patterns of objective fashion elements, therefore making preferable fashion trend forecast.
Yunshan Ma 0002, Yujuan Ding, Xun Yang 0001, Lizi Liao, Wai Keung Wong, Tat-Seng Chua
ICMR4
2019 Neural Multimodal Belief Tracker with Adaptive Attention for Dialogue Systems
abstract
Multimodal dialogue systems are attracting increasing attention with a more natural and informative way for human-computer interaction. As one of its core components, the belief tracker estimates the user's goal at each step of the dialogue and provides a direct way to validate the ability of dialogue understanding. However, existing studies on belief trackers are largely limited to textual modality, which cannot be easily extended to capture the rich semantics in multimodal systems such as those with product images. For example, in fashion domain, the visual appearance of clothes play a crucial role in understanding the user's intention. In this case, the existing belief trackers may fail to generate accurate belief states for a multimodal dialogue system.
Zheng Zhang 0020, Lizi Liao, Minlie Huang, Xiaoyan Zhu 0001, Tat-Seng Chua
WWW2
2018 Attributed Social Network Embedding
abstract
Embedding network data into a low-dimensional vector space has shown promising performance for many real-world applications, such as node classification and entity retrieval. However, most existing methods focused only on leveraging network structure. For social networks, besides the network structure, there also exists rich information about social actors, such as user profiles of friendship networks and textual content of citation networks. These rich attribute information of social actors reveal the homophily effect, exerting huge impacts on the formation of social networks. In this paper, we explore the rich evidence source of attributes in social networks to improve network embedding. We propose a generic Attributed Social Network Embedding framework (ASNE), which learns representations for social actors (i.e., nodes) by preserving both the structural proximity and attribute proximity. While the structural proximity captures the global network structure, the attribute proximity accounts for the homophily effect. To justify our proposal, we conduct extensive experiments on four real-world social networks. Compared to the state-of-the-art network embedding approaches, ASNE can learn more informative representations, achieving substantial gains on the tasks of link prediction and node classification. Specifically, ASNE significantly outperforms node2vec with an 8.2 percent relative improvement on the link prediction task, and a 12.7 percent gain on the node classification task.
Lizi Liao, Xiangnan He 0001, Hanwang Zhang, Tat-Seng Chua
IEEE Trans. Knowl. Data Eng.1
2017 Neural Collaborative Filtering
abstract
In recent years, deep neural networks have yielded immense success on speech recognition, computer vision and natural language processing. However, the exploration of deep neural networks on recommender systems has received relatively less scrutiny. In this work, we strive to develop techniques based on neural networks to tackle the key problem in recommendation --- collaborative filtering --- on the basis of implicit feedback.
Xiangnan He 0001, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Ben Hu, Tat-Seng Chua
WWW2
2016 SLR: A scalable latent role model for attribute completion and tie prediction in social networks
abstract
Social networks are an important class of networks that span a wide variety of media, ranging from social websites such as Facebook and Google Plus, citation networks of academic papers and patents, caller networks in telecommunications, and hyperlinked document collections such as Wikipedia - to name a few. Many of these social networks now exceed millions of users or actors, each of which may be associated with rich attribute data such as user profiles in social websites and caller networks, or subject classifications in document collections and citation networks. Such attribute data is often incomplete for a number of reasons - for example, users may be unwilling to spend the effort to complete their profiles, while in the case of document collections, there may be insufficient human labor to accurately classify all documents. At the same time, the tie or link information in these networks may also be incomplete - in social websites, users may simply be unaware of potential acquaintances, while in citation networks, authors may be unaware of appropriate literature that should be referenced. Completing and predicting these missing attributes and ties is important to a spectrum of applications, such as recommendation, personalized search, and targeted advertising, yet large social networks can pose a scalability challenge to existing algorithms designed for this task. Towards this end, we propose an integrative probabilistic model, SLR, that captures both attribute and tie information simultaneously, and can be used for attribute completion and tie prediction, in order to enable the above mentioned applications. A key innovation in our model is the use of triangle motifs to represent ties in the network, in order to scale to networks with millions of nodes and beyond. Experiments on real world datasets show that SLR significantly improves the accuracy of attribute prediction and tie prediction compared to well-known methods, and our distributed, multi-machine implementation easily scales up to millions of users. In addition to fast and accurate attribute and tie prediction, we also demonstrate how SLR can identify the attributes most responsible for homophily within the network, thus revealing which attributes drive network tie formation.
Lizi Liao, Qirong Ho, Jing Jiang 0001, Ee-Peng Lim
ICDE1