Out-of-distribution (OOD) detection is crucial for deploying models in open-world environments. This process aims to mitigate the issue of overconfident predictions, which is a common problem for models designed for closed-domain tasks when they encounter OOD data. Recent progress in OOD detection has shown that integrating auxiliary datasets during model training can greatly enhance OOD detection performance. However, existing methods tend to heavily depend on these auxiliary datasets to establish the decision boundary for in-distribution (ID) data, while not adequately addressing OOD object detection in safety–critical applications. In this article, we propose object-level OOD detection and introduce a hierarchical hard negative sampling (HNS) strategy that does not necessitate auxiliary data. Specifically, we offer a new metric that strategically considers difficult negatives near the decision boundary between inter-class and intra-class instances. Inspired by adversarial thinking, we sample outliers for each class, ensuring that the negative samples capture both diversity and informative traits. We conducted comprehensive experiments on three public datasets. The results demonstrate that HNS performs superiorly in object-level OOD detection, even without auxiliary datasets. The source code can be accessed at: https://github.com/Aurevior1/HNS .

Image classification serves as the cornerstone of computer vision, traditionally achieved through discriminative models based on deep neural networks. Recent advancements have introduced classification methods derived from generative models, which offer the advantage of zero-shot classification. However, these methods suffer from two main drawbacks: high computational overhead and inferior performance compared to discriminative models. Inspired by the coordinated cognitive processes of rapid-slow pathway interactions in the human brain during visual signal recognition, we propose the Diffusion-Based Discriminative Model Enhancement Framework (DBMEF). This framework seamlessly integrates discriminative and generative models in a training-free manner, leveraging discriminative models for initial predictions and endowing deep neural networks with rethinking capabilities via diffusion models. Consequently, DBMEF can effectively enhance the classification accuracy and generalization capability of discriminative models in a plug-and-play manner. We have conducted extensive experiments across 17 prevalent deep model architectures with different training methods, including both CNN-based models such as ResNet and Transformer-based models like ViT, to demonstrate the effectiveness of the proposed DBMEF.Specifically, the framework yields a 1.51% performance improvement for ResNet-50 on the ImageNet dataset and 3.02% on the ImageNet-A dataset. In conclusion, our research introduces a novel paradigm for image classification, demonstrating stable improvements across different datasets and neural networks.

Diffusion models have achieved impressive success in generating photorealistic images, but challenges remain in ensuring precise semantic alignment with input prompts. Optimizing the initial noisy latent offers a more efficient alternative to modifying model architectures or prompt engineering for improving semantic alignment. A latest approach, InitNo, refines the initial noisy latent by leveraging attention maps; however, these maps capture only limited information, and the effectiveness of InitNo is highly dependent on the initial starting point, as it tends to converge on a local optimum near this point. To this end, this paper proposes leveraging the language comprehension capabilities of large vision-language models (LVLMs) to guide the optimization of the initial noisy latent, and introduces the Noise Diffusion process, which updates the noisy latent to generate semantically faithful images while preserving distribution consistency. Furthermore, we provide a theoretical analysis of the condition under which the update improves semantic faithfulness. Experimental results demonstrate the effectiveness and adaptability of our framework, consistently enhancing semantic alignment across various diffusion models.

Large Vision-Language Models (LVLMs) are gradually becoming the foundation for many artificial intelligence applications. However, understanding their internal working mechanisms has continued to puzzle researchers, which in turn limits the further enhancement of their capabilities. In this paper, we seek to investigate how multimodal knowledge evolves and eventually induces natural languages in LVLMs. We design a series of novel strategies for analyzing internal knowledge within LVLMs, and delve into the evolution of multimodal knowledge from three levels, including single token probabilities, token probability distributions, and feature encodings. In this process, we identify two key nodes in knowledge evolution: the critical layers and the mutation layers, dividing the evolution process into three stages: rapid evolution, stabilization, and mutation. Our research is the first to reveal the trajectory of knowledge evolution in LVLMs, providing a fresh perspective for understanding their underlying mechanisms. Our codes are avaiable at https://github.com/XIAO4579/Vlm-Interpretability.

Abstract Amidst the rapid advancement of artificial intelligence, research on large vision-language models (LVLMs) has emerged as a pivotal area. However, understanding their internal mechanisms remains challenging due to the limitations of existing interpretability methods, especially regarding faithfulness and plausibility. To address this, we first construct a human response interpretability dataset that evaluates the plausibility of model explanations by comparing the attention regions between the model and humans when answering the same questions. We then propose a patchwise cooperative game-based interpretability method for LVLMs, which employs Shapley values to quantify the impact of individual image patches on generation likelihood and enhances computational efficiency through a single input approximation approach. Experimental results demonstrate our method’s faithfulness, plausibility, and robustness. Our method provides researchers with deeper insights into model behavior, allowing for an examination of the specific image regions each layer relies on during response generation, ultimately enhancing model reliability. Our code is available at https://github.com/ZY123-GOOD/Patchwise_Cooperative.

In the future, technical systems, e.g., cyber-physical systems will increasingly become adaptive to changing contexts. However, programming context-adaptive systems is challenging. The context-specific behavior as well as when contexts change their activeness must be specified. Common general-purpose object-oriented languages require encoding the context-specific behavior in if-cascades. Standard polymorphism is insufficient to express the changing behavior of objects in different contexts because the complexity rises with the combinatorial explosion of a large, potentially multi-dimensional context space. Moreover, specifying when contexts are active has to be implemented as part of the application logic. In conclusion, there are two problems: the definition of context-specific behavior and the management of context changes. In this paper, we present a framework for the Julia programming language to develop context-adaptive systems. The framework also enables context-adaptive equation-based modeling. For context management, Petri nets are utilized. Julia was chosen for the implementation due to its simulation ecosystem, rich metaprogramming, and multiple dispatch, which enables precise specification of behavioral variants. We evaluate our approach by using two examples. The first scenario is a smart home control application. The second example shows how our framework can be used together with equation-based modeling for simulation.

Due to differences of solar irradiance, ambient temperatures, or inconsistent degradation of photovoltaic (PV) modules, the unbalanced output power between cascaded H-bridge (CHB) legs will lead to the unbalanced or even distorted grid currents between three phases. This article proposes a novel CHB-based PV grid-tied system integrating centralized energy storage (CHB-PV/ES), which can realize power balanced operation by utilizing the centralized energy storage (ES) interconnecting the CHB three-phase sub-modules (SMs) by isolated DC/DC converter. In addition, the centralized ES group can facilitate the regulation between grid-tied power references and PV power generation, such as stabilizing the power injected into the grid by smoothing the PV power, as well as responding to changes in grid-tied power references. The configuration and security domain of PV and centralized ES SMs are critical factors for CHB-PV/ES, which are thoroughly analyzed in this article to ensure safe system operation. The simulation and experimental results validate the effectiveness of the proposed CHB-PV/ES scheme.

Crowdsourcing plays a vital role in today’s AI industry. However, existing crowdsourcing research mainly focuses on those simple tasks that are often formulated as label classification, while complex open-ended tasks such as question answering and translation have not received much attention. Such tasks usually have open solution spaces and non-unique true answers, which pose great challenges for designing effective crowdsourcing algorithms. In this work, we are concerned specifically with complex text annotation crowdsourcing tasks, where each answer of a task is in the form of free text. We propose an error consistency-based approach to inferring a satisfying result from a set of open-ended answers. First, each answer is represented with two vectors that capture the local word collocation and the global sentence semantics respectively. Second, the true answer is approximated by the sum of the answer vectors weighted by the reciprocals of their respective errors. Third, an algorithm called AEC (Aggregation based on Error Consistency) is designed to infer the aggregated result by maximizing the consistency of the errors of an answer in two vector spaces. Experimental results on two datasets demonstrate the effectiveness of our approach.

Knowledge acquisition (e.g. through labeling) is one of the most successful applications in crowdsourcing. In practice, collecting as specific as possible knowledge via crowdsourcing is very useful since specific knowledge can be generalized easily if we have a knowledge base, but it is difficult to infer specific knowledge from general knowledge. Meanwhile, tasks for acquiring more specific knowledge can be more difficult for workers, thus need more answers to infer high-quality results. Given a limited budget, assigning workers to difficult tasks will be more effective for the goal of specific knowledge acquisition. However, existing crowdsourcing task scheduling cannot incorporate the specificity of workers' answers. In this paper, we present a new framework for task scheduling with the limited budget, targeting an effective solution to more specific knowledge acquisition. We propose novel criteria for evaluating the quality of specificity-dependent answers and result inference algorithms to aggregate more specific answers with budget constraints. We have implemented our framework with real crowdsourcing data and platform, and have achieved significant performance improvement compared with existing approaches.

Finding suitable developers for a given task is critical and challenging for successful crowdsourcing software development. In practice, the development skills will be improved as developers accomplish more development tasks. Prior studies on crowdsourcing developer recommendation do not consider the changing of skills, which can underestimate developers' skills to fulfill a task. In this work, we first conducted an empirical study of the performance of 74 developers on Topcoder. With a difficulty-weighted algorithm, we re-compute the scores of each developer by eliminating the effect of task difficulty from the performance. We find out that the skill improvement of Topcoder developers can be fitted well with the negative exponential learning curve model. Second, we design a skill prediction method based on the learning curve. Then we propose a skill improvement aware framework for recommending developers for software development with crowdsourcing.