Facial analysis is a fundamental problem in vision–language research, with important applications in affective computing. However, existing methods primarily focus on global facial attributes or single-dimension analysis, lacking fine-grained, interpretable multi-attribute modeling of arbitrary local facial regions. We introduce FaceFocalDesc, a new problem that aims to generate and recognize multi-attribute natural language descriptions for arbitrarily selected facial regions. The target attributes include facial action units, emotional states, and age. We argue that explicit region-level modeling enables more controllable and interpretable facial understanding. To support this task, we construct a new dataset with region-level annotations and corresponding language descriptions. We further propose Focal-RegionFace, a vision–language model fine-tuned from Qwen2.5-VL, which progressively refines its focus on localized facial features through multi-stage training. Experiments show that Focal-RegionFace achieves state-of-the-art performance on the proposed benchmark under both standard and newly introduced metrics, demonstrating its effectiveness in fine-grained region-focused facial analysis.

Generative recommendation provides a novel paradigm in which each item is represented by a discrete semantic ID (SID) learned from rich content. Most methods treat SIDs as predefined and train recommenders under static indexing. In practice, SIDs are optimized only for content reconstruction rather than recommendation accuracy. This leads to an objective mismatch : the system optimizes an indexing loss to learn the SID, and a recommendation loss for interaction prediction, but because the tokenizer is trained independently, the recommendation loss cannot update it. A natural approach is to make semantic indexing differentiable so recommendation gradients can directly influence SID learning, but this often causes codebook collapse with only a few codes used. We attribute this to early deterministic assignments that limit codebook exploration, leading to imbalance and unstable optimization. In this paper, we therefore propose DIGER (Differentiable Semantic ID for GEnerative Recommendation). DIGER is a first step towards an effective differentiable semantic ID for generative recommendation. The Gumbel noise explicitly encourages early-stage exploration over codes, mitigating collapse and improving code utilization. To better balance exploration and convergence, we introduce two uncertainty decay strategies that reduce the Gumbel noise, enabling a gradual shift from early-stage exploration to the exploitation of learned SIDs. Extensive experiments across multiple public datasets demonstrate consistent improvements from differentiable semantic ID. These results confirm the effectiveness of aligning indexing and recommendation objectives through differentiable SIDs. This identifies differentiable SID as a promising area of study. Our code is released under https://github.com/junchen-fu/DIGER.

The advent of generative artificial intelligence (AI), driven by advancements in large language models (LLMs), has unlocked transformative possibilities for information retrieval (IR), giving rise to a new wave of interactive and conversational paradigms. This workshop, titled New Interaction Paradigms for Information Retrieval in the Era of Generative AI, aims to serve as a collaborative platform for researchers and practitioners to explore the challenges and opportunities of integrating generative AI into IR systems. By focusing on tasks such as multi-turn conversational search, adaptive retrieval interfaces, and context-aware response generation, this workshop will address key areas including system design, user engagement, and evaluation methodologies. The workshop will also delve into broader concerns such as trust, transparency, and fairness, emphasizing the ethical implications of deploying generative AI in IR systems. Through panel discussions, poster sessions, and interactive roundtables, this workshop will foster critical dialogue and innovation, paving the way for a new era of user-centric, generative AI-powered IR systems.

Negative sampling plays a crucial role for cross-domain recommendation as it provides contrastive signals to learn user preference. Existing methods usually select items with high predicted scores or popularity as hard negative samples to improve model training. However, such methods suffer from choosing false negative samples since items with high predicted scores or popularity could also indicate potential positive user preference. Although several studies devoted to discovering true negative samples, few of them leverage user cross-domain behaviors to alleviate the false negative issue. How to effectively mine and utilize hard negative samples to improve cross-domain recommendation remains an open question.

Image-Text Matching (ITM) is a fundamental problem in computer vision. The key issue lies in jointly learning the visual and textual representation to estimate their similarity accurately. Most existing methods focus on feature enhancement within modality or feature interaction across modalities, which, however, neglects the contextual information of the object representation based on the inter-object relationships that match the corresponding sentences with rich contextual semantics. In this article, we propose a Hybrid-modal Interaction with multiple Relational Enhancements (termed Hire ) for ITM, which correlates the intra- and inter-modal semantics between objects and words with implicit and explicit relationship modeling. In particular, the explicit intra-modal spatial-semantic graph-based reasoning network is designed to improve the contextual representation of visual objects with salient spatial and semantic relational connectivities, guided by the explicit relationships of the objects’ spatial positions and their scene graph. We use implicit relationship modeling for potential relationship interactions before explicit modeling to improve the fault tolerance of explicit relationship detection. Then the visual and textual semantic representations are refined jointly via inter-modal interactive attention and cross-modal alignment. To correlate the context of objects with the textual context, we further refine the visual semantic representation via cross-level object-sentence and word-image-based interactive attention. Extensive experiments validate that the proposed hybrid-modal interaction with implicit and explicit modeling is more beneficial for ITM. And the proposed Hire obtains new state-of-the-art results on MS-COCO and Flickr30K benchmarks.

Multimodal foundation models (MFMs) have revolutionized sequential recommender systems through advanced representation learning. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt these models, studies often prioritize parameter efficiency, neglecting GPU memory and training speed. To address this, we introduced the IISAN framework, significantly enhancing efficiency. However, IISAN was limited to symmetrical MFMs and identical text and image encoders, preventing the use of state-of-the-art Large Language Models. To overcome this, we developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs. IISAN-Versa employs a Decoupled PEFT structure and utilizes both intra- and inter-modal adaptation. It effectively handles asymmetry through a simple yet effective combination of group layer-dropping and dimension transformation alignment. Our research demonstrates that IISAN-Versa effectively adapts large text encoders, and we further identify a scaling effect where larger text encoders generally perform better. IISAN-Versa also demonstrates strong versatility in our defined multimodal scenarios, which include raw titles and captions generated from images and videos. Additionally, IISAN-Versa achieved state-of-the-art performance on the MicroLens public benchmark.

Accuracy has been the primary benchmark for assessing recommenders learned from sequential interactions. To improve user experience by diverse and novel recommendation, our paper focuses on Multi-objective Sequential Recommendation (MOSR) to balance these conflicting objectives. Although a few studies leveraged reinforcement learning (RL) to solve MOSR, these methods can lead to sub-optimal results. First, traditional offline RL approach typically optimizes various objectives independently via multiple RL heads, accumulating prediction errors and leading to unstable performance. Furthermore, the offline policy cannot dynamically adjust objective weights during the inference stage, limiting adaptability to varying contexts. To this end, we introduce Multi-objective Decision Transformer for Reward-driven Recommendation (MODT4R), a novel framework that addresses MOSR as sequence modeling problem. First, we propose a user trajectory to capture user state transitions along with their multi-objective interests, represented by sequential expected cumulative rewards (returns). Moreover, the supervised learning paradigm makes the training process more stable while naturally integrating multi-objective optimization into sequence modeling by using multiple returns as conditional inputs. During inference, a score function is used to adjust the weights of diversity and novelty. Experimental evaluations on real-world datasets demonstrate that MODT4R significantly enhances diversity and novelty while maintaining accuracy compared to existing state-of-the-art methods.

Personalized Session-based Recommendation (PSR) extends the traditional sequential recommendation models-which typically recommends the next item based on a recent active session-to leverage historical sessions of a user for short-term recommendations in current session. However, existing PSR methods face two limitations: (1) treating offline sessions uniformly as static data and relying on user embeddings to represent personalized information overlook the dynamic evolution of interests over time, which can change significantly as sessions progress in practical application. (2) focusing on accuracy, i.e., recommending items relevant to recent interactions, ignores the balance of multi-faceted requirements for user satisfaction, i.e., diversity, novelty, and serendipity.

Multimodal foundation models are transformative in sequential recommender systems, leveraging powerful representation learning capabilities. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt foundation models for recommendation tasks, most research prioritizes parameter efficiency, often overlooking critical factors like GPU memory efficiency and training speed. Addressing this gap, our paper introduces IISAN (Intra- and Inter-modal Side Adapted Network for Multimodal Representation), a simple plug-and-play architecture using a Decoupled PEFT structure and exploiting both intra- and inter-modal adaptation. IISAN matches the performance of full fine-tuning (FFT) and state-of-the-art PEFT. More importantly, it significantly reduces GPU memory usage - from 47GB to just 3GB for multimodal sequential recommendation tasks. Additionally, it accelerates training time per epoch from 443s to 22s compared to FFT. This is also a notable improvement over the Adapter and LoRA, which require 37-39 GB GPU memory and 350-380 seconds per epoch for training. Furthermore, we propose a new composite efficiency metric, TPME (Training-time, Parameter, and GPU Memory Efficiency) to alleviate the prevalent misconception that "parameter efficiency represents overall efficiency". TPME provides more comprehensive insights into practical efficiency comparisons between different methods. Besides, we give an accessible efficiency analysis of all PEFT and FFT approaches, which demonstrate the superiority of IISAN. We release our codes and other materials at https://github.com/GAIR-Lab/IISAN.

Text-to-image retrieval aims to find the relevant images based on a text query, which is important in various use-cases, such as digital libraries, e-commerce, and multimedia databases. Although Multimodal Large Language Models (MLLMs) demonstrate state-of-the-art performance, they exhibit limitations in handling large-scale, diverse, and ambiguous real-world needs of retrieval, due to the computation cost and the injective embeddings they produce. This paper presents a two-stage Coarse-to-Fine Index-shared Retrieval (CFIR) framework, designed for fast and effective large-scale long-text to image retrieval. The first stage, Entity-based Ranking (ER), adapts to long-text query ambiguity by employing a multiple-queries-to-multiple-targets paradigm, facilitating candidate filtering for the next stage. The second stage, Summary-based Re-ranking (SR), refines these rankings using summarized queries. We also propose a specialized Decoupling-BEiT-3 encoder, optimized for handling ambiguous user needs and both stages, which also enhances computational efficiency through vector-based similarity inference. Evaluation on the AToMiC dataset reveals that CFIR surpasses existing MLLMs by up to 11.06% in Recall@1000, while reducing training and retrieval times by 68.75% and 99.79%, respectively. We will release our code to facilitate future research at https://github.com/longkukuhi/CFIR.

The Facial Action Coding System (FACS) encodes the action units (AUs) in facial images, which has attracted extensive research attention due to its wide use in facial expression analysis. Many methods that perform well on automatic facial action unit (AU) detection primarily focus on modeling various AU relations between corresponding local muscle areas or mining global attention–aware facial features; however, they neglect the dynamic interactions among local-global features. We argue that encoding AU features just from one perspective may not capture the rich contextual information between regional and global face features, as well as the detailed variability across AUs, because of the diversity in expression and individual characteristics. In this article, we propose a novel Multi-level Graph Relational Reasoning Network (termed MGRR-Net ) for facial AU detection. Each layer of MGRR-Net performs a multi-level (i.e., region-level, pixel-wise, and channel-wise level) feature learning. On the one hand, the region-level feature learning from the local face patch features via graph neural network can encode the correlation across different AUs. On the other hand, pixel-wise and channel-wise feature learning via graph attention networks (GAT) enhance the discrimination ability of AU features by adaptively recalibrating feature responses of pixels and channels from global face features. The hierarchical fusion strategy combines features from the three levels with gated fusion cells to improve AU discriminative ability. Extensive experiments on DISFA and BP4D AU datasets show that the proposed approach achieves superior performance than the state-of-the-art methods.