Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL

The goal of continual learning (CL) is to acquire new knowledge while retaining previously learned information. CNN-based and prompt-based CL methods have achieved remarkable progress in recent years. However, most prior work has primarily focused on reducing forgetting from the perspective of the model itself. In this paper, we investigate CL from the perspective of decision boundaries, analyzing the impact of instance-level feature overlap. To address this issue, we propose a generic Decoupled Gaussian Softmax Classifier that enhances class discriminability during CL process. Specifically, we decouple the features extracted by the backbone into multiple Gaussian distributions, which are directly fused into the feature space through weighted integration. A regularization term is introduced to penalize the overlap of similar features, while an adaptive decision boundary is assigned to each class to encourage inter-class separation and intra-class compactness. Experiments on 4 widely used continual learning datasets and 12 CL scenarios show that our method has good plug-and-play capability. It improves the average accuracy by 1%–2.63% over the baseline models, while effectively reducing both the forgetting rate and the Expected Calibration Error. Our code is available at: https://anonymous.4open.science/r/DGSC-main-310D .

Pre-trained language models (PLMs) are trained on data that inherently contains gender biases, leading to undesirable impacts. Traditional debiasing methods often rely on external corpora, which may lack quality, diversity, or demographic balance, affecting the effectiveness of debiasing. With the rise of large language models and their extensive knowledge, we propose enhancing fairness (Fair-Gender) in PLMs by absorbing coherent, attribute-balanced, and semantically rich sentences. However, these sentences cannot be directly used for debiasing due to alignment issues and the risk of negative transfer. We address this by applying causal analysis to estimate causal effects, filtering out unaligned sentences, and identifying aligned ones for incorporation into PLMs, thereby ensuring positive transfer. Experiments show that our approach significantly reduces gender biases in PLMs while preserving their language expressiveness.

The goal of Class Incremental Learning (CIL) is to continuously learn new classes while preventing forgetting of old ones. Most previous works focused on reducing catastrophic forgetting from model’s perspective. However, the model is not the only factor contributing to forgetting. In this paper, we take the perspective of class instances and find that fine-grained class increments can lead to feature overlap between classes, further reducing instance margins. We call this interesting phenomenon as Fine-grained class confusion effect in CIL. Since preserving instance margins is crucial for resisting forgetting, it is beneficial to maintain the margin amount as much as possible. To achieve this, we propose a general Gaussian decoupling classifier to enhance the discriminability of similar classes during incremental learning. Specifically, we decouple the features of different classes extracted by the backbone network into multiple independent Gaussian distributions. By directly integrating them into the features with weighted fusion, we introduce a regularization penalty that encourages minimizing the overlap of similar features, thus increasing the feature distance between classes. Extensive experiments show that our method effectively improves class separation and better preserves instance margins, ultimately alleviating forgetting. The improved model achieves better performance on CUB-200 and CARS-196.

Photometric stereo aims to recover detailed surface normal maps from images captured under varying illuminations. However, existing methods often rely on extensive real images under controlled lighting conditions to achieve accurate surface normal estimation and struggle to perform effectively when the number of input images is limited.Hence, we propose Diffusion Model based Relighting for Uncalibrated Photometric Stereo (Delight-UPS) to address the above limitations. Specifically, we first employ diffusion model to simplify complex illumination with wide-angle light. Then, narrow-angle light is applied to generate images with object illumination variations. Finally, we recover the normal map from the illumination-variant images.This enables photometric stereo using only a single image, even under complex lighting conditions.Experiments show that Delight-UPS outperforms the SOTA model SDM by 38.0% using limited input images.Moreover, Delight-UPS improves previous models’ performance by 18.4% on average and allows them to recover normal maps from synthetic images.* Our source code is anonymously hosted at ${\text{Delight - UPS}}$.

Commonsense knowledge graph acquisition (CKGA) is essential for many knowledge-intensive applications, such as natural language understanding, question answering, and conversational system. Traditional methods that directly use word-level triplet for knowledge generation often lack sufficient context, leading to ambiguous relations and a limited ability to handle complex or abstract concepts. Moreover, they also rely on forward relations and struggle to fully capture the reverse connections between entities, which can lead to the "reversal curse". To address these, we firstly transform pre-defined relations into sentence templates, and introduce a new pipeline that enhances CKGA task via bi-directional relation-enhanced contrastive pretraining from recent large-scale foundation models. Our closed-loop Bi-REACT includes data preprocessing, contrastive pre-training, task-driven instruction tuning, filtering model, and evaluation system. Experiments show Bi-REACT can easily harvest extensive high-quality knowledge (390K), achieving up to 90.02% (ATOMIC) and 84.29% (ConceptNet) accuracy, approaching human-level performance for these resources, and effectively reducing the "reversal curse" issues. Our code is available at https://anonymous.4open.science/r/CKGA-345E.

Pre-trained language models (PLMs) have greatly transformed various downstream tasks, yet frequently display social biases from training data, raising fairness concerns. Recent efforts to debias PLMs come with limitations: they either fine-tune the entire parameters in PLMs, which is time-consuming and disregards the expressiveness of PLMs, or ignore the reintroducing biases from downstream tasks when applying debiased models to them. Hence, we propose a two-stage pipeline to mitigate biases from both internal and downstream contexts while preserving expressiveness in language models. Specifically, for the debiasing procedure, we resort to continuous prefix-tuning, not fully fine-tuning the PLM, in which we design a debiasing term for optimization and an alignment term to keep words’ relative distances and ensure the model's expressiveness. For downstream tasks, we perform causal intervention across different demographic groups for invariant predictions. Results on three GLUE tasks show our method alleviates biases from internal and downstream contexts, while keeping PLM expressiveness intact.

Conventional commonsense knowledge graph completion (CKGC) methods provide inadequate sequence when fine-tuning or generating stages and incorporate full fine-tuning, which fail to align with the autoregressive model's pre-training patterns and have insufficient parameter efficiency. Moreover, decoding through beam or greedy search produces low diversity and high similarity in generated tail entities. Hence, we resort to prefix-tuning and propose a lightweight, effective pipeline to enhance the quality and diversity of extracted commonsense knowledge. Precisely, we measure head entity similarity to yield and then concatenate top-k tuples before each target tuple for prefix-tuning the source LM, thereby improving the efficiency and speed for pretrained models; then, we design a penalty-tailored diverse beam search (p-DBS) for decoding tail entities, producing a greater quantity and diversity of generated commonsense tuples; besides, a filter strategy is utilized to filter out invalid commonsense knowledge. Through extensive automatic evaluations, including ChatGPT scoring, our method can extract diverse, novel, and accurate commonsense knowledge (CK).

Demographic biases and social stereotypes are common in pretrained language models (PLMs), while the fine-tuning in downstream applications can also produce new biases or amplify the impact of the original biases. Existing works separate the debiasing from the fine-tuning procedure, which results in a gap between intrinsic bias and application bias. In this work, we propose a debiasing framework CauDebias to eliminate both biases, which directly combines debiasing with fine-tuning and can be applied for any PLMs in downstream tasks. We distinguish the bias-relevant (non-causal factors) and label-relevant (causal factors) parts in sentences from a causal invariant perspective. Specifically, we perform intervention on non-causal factors in different demographic groups, and then devise an invariant risk minimization loss to trade-off performance between bias mitigation and task accuracy. Experimental results on three downstream tasks show that our CauDebias can remarkably reduce biases in PLMs while minimizing the impact on downstream tasks.

Demographic biases and social stereotypes are common in pretrained language models (PLMs), and a burgeoning body of literature focuses on removing the unwanted stereotypical associations from PLMs.However, when fine-tuning these bias-mitigated PLMs in downstream natural language processing (NLP) applications, such as sentiment classification, the unwanted stereotypical associations resurface or even get amplified.Since pretrain&fine-tune is a major paradigm in NLP applications, separating the debiasing procedure of PLMs from fine-tuning would eventually harm the actual downstream utility.In this paper, we propose a unified debiasing framework Causal-Debias to remove unwanted stereotypical associations in PLMs during fine-tuning.Specifically, Causal-Debias mitigates bias from a causal invariant perspective by leveraging the specific downstream task to identify bias-relevant and labelrelevant factors.We propose that bias-relevant factors are non-causal as they should have little impact on downstream tasks, while labelrelevant factors are causal.We perform interventions on non-causal factors in different demographic groups and design an invariant risk minimization loss to mitigate bias while maintaining task performance.Experimental results on three downstream tasks show that our proposed method can remarkably reduce unwanted stereotypical associations after PLMs are finetuned, while simultaneously minimizing the impact on PLMs and downstream applications. *Corresponding author unwanted stereotypical associations in PLMs.For example, some works (Zmigrod et al., 2019) pretrain a language model using original and counterfactual corpus in order to cancel-out biased associations, some works (Liang et al., 2020) focus on debiasing post-hoc sentence representations, and others (Guo et al., 2022;Cheng et al., 2021) design bias-equalizing objectives to fine-tune PLM's parameters.

Detecting rumors on social media has grown in importance as the amount of digital material available online grows quickly. Recent approaches to rumor detection heavily rely on supervised learning, which requires a significant amount of labeled data for training and provides limited interpretability of prediction results. Furthermore, these approaches show a lack of robustness and are vulnerable to overfitting. In this paper, we propose a novel framework Self-Supervised Rumor Detection with Augmented Variational Graphs (SSRD-AVG) from a self-supervised learning (SSL) view, which employs a pre-trained generative model to facilitate data augmentation with enhanced interpretability. Specifically, the generative model first harnesses neighboring information to extract salient features for rumor propagation structures and user engagement. Then, the obtained augmented features are subsequently utilized for self-supervised learning. Finally, we fine-tuned the Graph Neural Network (GNN) with labeled data for rumor detection. Comprehensive experiments demonstrate that our model attains state-of-the-art performance and remains robust in real-world scenarios.

Unwanted social biases are usually encoded in pretrained language models (PLMs). Recent efforts are devoted to mitigating intrinsic bias encoded in PLMs. However, the separate fine-tuning on applications is detrimental to intrinsic debiasing. A bias resurgence issue arises when fine-tuning the debiased PLMs on downstream tasks. To eliminate undesired stereotyped associations in PLMs during fine-tuning, we present a mixup-based framework Mix-Debias from a new unified perspective, which directly combines debiasing PLMs with fine-tuning applications. The key to Mix-Debias is applying mixup-based linear interpolation on counterfactually augmented downstream datasets, with expanded pairs from external corpora. Besides, we devised an alignment regularizer to ensure original augmented pairs and gender-balanced counterparts are spatially closer. Experimental results show that Mix-Debias can reduce biases in PLMs while maintaining a promising performance in applications.

Information cascade is typically formalized as a process of (simplified) discrete sequence of events, and recent approaches have tackled its prediction via variants of recurrent neural networks. However, the information diffusion process is essentially an evolving directed acyclic graph (DAG) in the continuous-time domain. In this paper, we propose a transformer enhanced Hawkes process (Hawkesformer), which links the hierarchical attention mechanism with Hawkes process to model the arrival stream of discrete events continuously. A two-level attention architecture is used to parameterize the intensity function of Hawkesformer, which captures the long-term dependencies between nodes in graph and better embeds the cascade evolution rate for modeling short-term outbreaks. Experimental results demonstrate the significant improvements of Hawkesformer over the state-of-the-art.

Effectively predicting the size of information cascades is crucial for understanding the evolution of many social applications, such as influence maximization and fake news detection. Conventional methods face the challenge of data imbalance which, in turn, yields unsatisfactory prediction performance. To prevent the loss functions or metrics from being affected by extreme values and assure numerical stability, previous works reformulate the problem definitions or adopt other types of evaluation metrics. However, solving the regression prediction of information cascades from a long-tailed distribution perspective is under explored. In this paper, we propose a general decoupling prediction solution -- first extracting the representation, then fine-tuning the regressor, which combines the original prediction value and weighted bias generated by a sub-network (SUB) that we designed. Our experiments conducted on long-tailed benchmarks demonstrate that our method significantly improves the prediction accuracy over state-of-the-art methods and mitigates the long-tailed cascade prediction problem.