This paper introduces a Context-Aware and User Intent-Aware follow-up Question Generation (CA-UIA-QG) method in multi-turn conversational settings. Our CA-UIA-QG model is designed to simultaneously consider the evolving context of a conversation and identify user intent. By integrating these aspects, it generates relevant follow-up questions, which can better mimic user behavior and align well with users’ conversational goals. When assessed using public Shopping datasets on Fashion domain, our approach demonstrates significant enhancements over CA-QG baseline models. Specifically, it achieves an improvement of up to 3% in BLEU, 7% in METEOR, and 8% in ROUGE-Lsum. Additionally, our findings show the efficacy of fine-tuning in enhancing the model’s capacity to better mimic user behavior, CoT prompting with fine-tuned model yields superior performance compared to the ensemble method. Furthermore, we investigate the impact of model size, model type, and intent granularity, highlighting their impact to overall model performance. The importance of our work lies in its effectiveness to improve follow-up question generation from the user’s perspective and application in developing user-centric conversational AI systems.

The KDD workshop on Evaluation and Trustworthiness of Generative AI Models aims to address the critical need for reliable generative AI technologies by exploring comprehensive evaluation strategies. This workshop will delve into various aspects of assessing generative AI models, including Large Language Models (LLMs) and diffusion models, focusing on trustworthiness, safety, bias, fairness, and ethical considerations. With an emphasis on interdisciplinary collaboration, the workshop will feature invited talks, peer-reviewed paper presentations, and panel discussions to advance the state of the art in generative AI evaluation.

Detecting audio-video (A/V) synchronization error is important to measure end user experience. Today, researches in this domain are mainly focused on contents such as movies or sports. The state of art algorithms usually first detect a specific type of events and then correlate the A/V data within during these events, e.g., find the human chatting events and then correlate the vocals with the lip shapes. Detecting A/V sync errors during Ads, on the other hand, has not received a lot of attentions. Compared with contents, an Ads section do not contain a particular type of events that can be used to detect A/V sync error. For example, many vocals in Ads are either from background narrators or have a very short period of time, so that the popular lip-sync based algorithms won’t work accurately. In this paper, we present a novel algorithm that uses the scene change time features: we first segment out individual Ad from a playback. Then for each pair of temporal adjacent Ads, we compute the scene change time for the video data and the audio data separately, and then build their time difference histogram. Next, we aggregate the histograms from all Ads pairs within one Ads section. Finally, we combine the aggregated histogram to compute the A/V off-sync time values. We show that compared with the traditional lip-sync based algorithms, the new algorithm not only significantly improves the prediction rate, but also increases the prediction accuracy.

Online streaming services have been growing at a fast pace. To provide the best user experience, it is needed to detect low quality images from videos so that we can repair or improve them before showing to customers. For example, for movie and TV-show streaming services, it is important to check if an original (master) video produced from a studio has low quality images that contain artifacts such as up-scaling or interlacing; for live streaming services, it is important to detect if a streamed video have hits due to encoding such as H.264 or MPEG-2. The impairment detection is usually measured by the nonreference (NR) metrics because it is often difficult and sometimes impossible to get the original (master) videos. On the other hand, today researches in the image quality area, such as super-resolution, are mainly focused on the full reference (FR) metrics like PSNR or VMAF. In this paper, we present an algorithm that is able to reliably compute five types of spatial NR metrics that are commonly used in the online streaming industries. The algorithm consists of two components: a pre-processing step that spatially de-correlates pixel intensity values and a novel deep neural network (DNN) that is able to quantify the NR metrics at the image region level. We show that our algorithm achieves better performance than state-of-art algorithms in this area.