Existing Vision-Language Pretraining (VLP) methods have achieved remarkable improvements across a variety of vision-language tasks, confirming their effectiveness in capturing coarse-grained semantic correlations. However, their capability for fine-grained understanding, which is critical for many nuanced vision-language applications, remains limited. Prevailing VLP models often overlook the intricate distinctions in expressing different modal features and typically depend on the similarity of holistic features for cross-modal interactions. Moreover, these models directly align and integrate features from different modalities, focusing more on coarse-grained general representations, thus failing to capture the nuanced differences necessary for tasks demanding a more detailed perception. In response to these limitations, we introduce Negative Augmented Samples(NAS), a refined vision-language pretraining model that innovatively incorporates NAS to specifically address the challenge of fine-grained understanding. NAS utilizes a Visual Dictionary(VD) as a semantic bridge between visual and linguistic domains. Additionally, it employs a Negative Visual Augmentation(NVA) method based on the VD to generate challenging negative image samples. These samples deviate from positive samples exclusively at the token level, thereby necessitating that the model discerns the subtle disparities between positive and negative samples with greater precision. Comprehensive experiments validate the efficacy of NAS components and underscore its potential to enhance fine-grained vision-language comprehension.

The impressive performance of Large Language Model (LLM) has prompted researchers to develop Multi-modal LLM (MLLM), which has shown great potential for various multi-modal tasks. However, current MLLM often struggles to effectively address fine-grained multi-modal challenges. We argue that this limitation is closely linked to the models’ visual grounding capabilities. The restricted spatial awareness and perceptual acuity of visual encoders frequently lead to interference from irrelevant background information in images, causing the models to overlook subtle but crucial details. As a result, achieving fine-grained regional visual comprehension becomes difficult. In this paper, we break down multi-modal understanding into two stages, from Coarse to Fine (CoF). In the first stage, we prompt the MLLM to locate the approximate area of the answer. In the second stage, we further enhance the model’s focus on relevant areas within the image through visual prompt engineering, adjusting attention weights of pertinent regions. This, in turn, improves both visual grounding and overall performance in downstream tasks. Our experiments show that this approach significantly boosts the performance of baseline models, demonstrating notable generalization and effectiveness. Our CoF approach is available online at https://github.com/Gavin001201/CoF.

Due to the flexibility of heterogeneous information networks (HINs) in modeling heterogeneous data, researchers begin to use it to integrate the objects and relationships in recommender systems. However, how to abstract and exploit effective information and apply the information to recommender systems is a challenge. To fully mine nodes' structural features and better integrate these features simultaneously, we present a nonlinear feature fusion-based rating prediction algorithm. This algorithm first uses a meta-path-based HIN embedding model to extract the nodes' structural features. Then, the structural features are converted by a nonlinear fusion method. Finally, the fused features are input into the multilayer perceptron to achieve rating prediction. Experiments on real-life data sets, such as Movielens-100k, Yelp, Douban Book, and Douban Movie, are designed to prove the performance of the proposed model. Experimental results on the four data sets reveal that our algorithm is superior to the baselines.

Abstract Modelling and simulation of tree growth that is faithful to the living environment and numerically consistent to botanic knowledge are important topics for realistic modelling in computer graphics. The realism factors concerned include the effects of complex environment on tree growth and the reliability of the simulation in botanical research, such as horticulture and agriculture. This paper proposes a new approach, namely, integrated growth modelling, to model virtual trees and simulate their growth by enforcing constraints of environmental resources and tree morphological properties. Morphological properties are integrated into a growth equation with different parameters specified in the simulation, including its sensitivity to light, allocation and usage of received resources and effects on its environment. The growth equation guarantees that the simulation procedure numerically matches the natural growth phenomenon of trees. With this technique, the growth procedures of diverse and realistic trees can also be modelled in different environments, such as resource competition among multiple trees.

We present a novel system, named PhotoReference, to help users, especially amateur photographers, to interactively learn to improve photograph skills by leveraging the available web image collections. The idea behind is based on the observation that the quality of a photo is mainly determined with the important camera parameters set by users when the photo is taken. In the proposed system, given a user-provided photo and its associated metadata, the system first retrieves sets of high-quality community-contributed photos with similar visual content and shoot settings to the input photo. Then users are allowed to interactively explore the returned photos as per different camera parameters, such that they can easily share and learn the photograph experiences of other people; and more importantly, users are able to get intuitive knowledge on the relationship between camera parameters and the resulted shooting effects and thus can use it to guide their future photograph activities.