While conversational agents’ (CAs) semantic and syntactic capabilities have advanced, their pragmatic skills, using language appropriately in context, have emerged as a critical focus in practical applications. Hence, scholars integrate conversational skills derived from human-human interaction into CA designs. However, existing research mainly adopts an empirical approach and focuses on specific CA deployment, making it challenging to identify overarching patterns or develop a comprehensive methodology for transferring human pragmatic skills to CA design. Thus, we conducted a systematic review of 85 studies from primary databases (e.g., ACM, IEEE, etc.), focusing on designing CAs with human-derived conversational skills. We identified skill categories (verbal, paralinguistic, nonverbal), transfer strategies (from dialog data, theories, and via co-design), implementations, and evaluation metrics. We consolidated these insights into a four-stage design process: human skill exploration, definition, transfer, and iterative evaluation. Future research can leverage this to design CAs that achieve conversational goals through contextually appropriate language use.

While large language models have advanced di-agnostic reasoning in clinical domains, but to fully support the patient care, accurate and personalized treatment (illness) management plans are also needed. Unlike conventional planning tasks with defined goals and constraints, illness management planning navigates through uncertainty, incomplete data, and nonlinear, cross-disease effects. As a result, existing methods (e.g., Chain-of- Thought and Reflexion), which rely on assumptions of defined goals and linear reasoning, fall short in the complexities and ambiguities inherent in illness management planning. Another major challenge is the lack of a high-quality dataset that pairs real-world patient narratives with actionable illness management plans, which are crucial for evaluating the capabilities of LLMs in illness management planning. To address these challenges, we propose a novel planning method that reconceives treatment planning as the modulation of a patient's current illness state toward a healthy state without any defined goal. The approach introduces Attractive Tendencies, a latent, directional vectors that define desirable shifts toward healthier states, then uses field mapping to identify modifiable life domains that define personalized illness management goals, and applies field sculpting to generate safe, individualized, and actionable interventions. To enable standardized evaluation, we release an evaluation dataset comprising 1,015 patient cases, each paired with a real-world, complex narrative and a personalized treatment plan. Our method outperforms baseline approaches, achieving improvements of up to +12 BLEU-4 and +11 METEOR, while maintaining strong clinical relevance and computational efficiency.

Unlike reasoning, which often entails a deep sequence of deductive steps, complex real-world planning is characterized by the need to synthesize a broad spectrum of parallel and potentially conflicting information and constraints. For example, in travel planning scenarios, it requires the integration of diverse real-world information and user preferences. While LLMs show promise, existing methods with long-horizon thinking struggle with handling multifaceted constraints, leading to suboptimal solutions. Motivated by the challenges of real-world travel planning, this paper introduces the Multiple Aspects of Planning (MAoP), empowering LLMs with "wide-horizon thinking" to solve planning problems with multifaceted constraints. Instead of direct planning, MAoP leverages the strategist to conduct pre-planning from various aspects and provide the planning blueprint for planners, enabling strong inference-time scalability by scaling aspects to consider various constraints. In addition, existing benchmarks for multi-constraint planning are flawed because they assess constraints in isolation, ignoring causal dependencies within the constraints, e.g, travel planning, where past activities dictate future itinerary. To address this, we propose Travel-Sim, an agent-based benchmark assessing plans via real-world simulation, thereby inherently resolving these causal dependencies. This paper advances LLM capabilities in complex planning and offers novel insights for evaluating sophisticated scenarios through simulation.

Regular screening is critical for individuals at risk of neurocognitive disorders (NCDs) to receive early intervention. Conversational agents (CAs) have been adopted to administer dialog-based NCD screening tests for their scalability compared to human-administered tests. However, unique communication skills are required for CAs during NCD screening, e.g., clinicians often apply scaffolding to ensure subjects’ understanding of and engagement in screening tests. Based on scaffolding theories and analysis of clinicians’ practices from human-administered test recordings, we designed a scaffolding framework for the CA. In an exploratory wizard-of-Oz study, the CA empowered by ChatGPT administered tasks in the Grocery Shopping Dialog Task with 15 participants (10 diagnosed with NCDs). Clinical experts verified the quality of the CA’s scaffolding and we explored its effects on task understanding of the participants. Moreover, we proposed implications for the future design of CAs that enable scaffolding for scalable NCD screening.

Advancements in multimodal learning, particularly in video understanding and generation, require high-quality video-text datasets for improved model performance. Vript addresses this issue with a meticulously annotated corpus of 12K high-resolution videos, offering detailed, dense, and script-like captions for over 420K clips. Each clip has a caption of ~145 words, which is over 10x longer than most video-text datasets. Unlike captions only documenting static content in previous datasets, we enhance video captioning to video scripting by documenting not just the content, but also the camera operations, which include the shot types (medium shot, close-up, etc) and camera movements (panning, tilting, etc). By utilizing the Vript, we explore three training paradigms of aligning more text with the video modality rather than clip-caption pairs. This results in Vriptor, a top-performing video captioning model among open-source models, comparable to GPT-4V in performance. Vriptor is also a powerful model capable of end-to-end generation of dense and detailed captions for long videos. Moreover, we introduce Vript-Hard, a benchmark consisting of three video understanding tasks that are more challenging than existing benchmarks: Vript-HAL is the first benchmark evaluating action and object hallucinations in video LLMs, Vript-RR combines reasoning with retrieval resolving question ambiguity in long-video QAs, and Vript-ERO is a new task to evaluate the temporal understanding of events in long videos rather than actions in short videos in previous works. All code, models, and datasets are available in https://github.com/mutonix/Vript.

Masked Language Modeling (MLM) has been widely used as the denoising objective in pretraining language models (PrLMs).Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training.However, the model may receive a complicated impact from pre-training status, which changes accordingly as training time goes on.In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings.We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks.Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pretraining 1 .