In-context learning (ICL) has emerged as a powerful paradigm for Large Visual Language Models (LVLMs), enabling them to leverage a few examples directly from input contexts. However, the effectiveness of this approach is heavily reliant on the selection of demonstrations, a process that is NP-hard. Traditional strategies, including random, similarity-based sampling and infoscore-based sampling, often lead to inefficiencies or suboptimal performance, struggling to balance both efficiency and effectiveness in demonstration selection. In this paper, we propose a novel demonstration selection framework named Coreset-based Dual Retrieval (CoDR). We show that samples within a diverse subset achieve a higher expected mutual information. To implement this, we introduce a cluster-pruning method to construct a diverse coreset that aligns more effectively with the query while maintaining diversity. Additionally, we develop a dual retrieval mechanism that enhances the selection process by achieving global demonstration selection while preserving efficiency. Experimental results demonstrate that our method significantly improves the ICL performance compared to the existing strategies, providing a robust solution for effective and efficient demonstration selection.

We propose a novel Transformer-based image-to-text generation model termed asACFthat adaptively clusters vision patches into object regions and language words into phrases to implicitly learn object-phrase alignments for better visual-text coherence. To achieve this, we design a novel self-attention layer that applies self-attention over the elements in a local cluster window instead of the whole sequence. The window size is softly decided by a clustering matrix that is calculated by the current input data and thus this process is adaptive. By stacking these revised self-attention layers to construct ACF, the small clusters in the lower layers can be grouped into a bigger cluster, e.g., vision/language. ACF clusters small objects/phrases into bigger ones. In this gradual clustering process, a parsing tree is generated which embeds the hierarchical knowledge of the input sequence. As a result, by using ACF to build the vision encoder and language decoder, the hierarchical object-phrase alignments are embedded and then transferred from vision to language domains in two popular image-to-text tasks: Image captioning and Visual Question Answering. The experiment results demonstrate the effectiveness of ACF, which outperforms most SOTA captioning and VQA models and achieves comparable scores compared with some large-scale pre-trained models.

Accurately and rapidly detecting traffic object has been attracted intensive attention due to its potential applications in the fields of autonomous driving, traffic flow monitoring, augmented reality (AR) and so on. However, there are many difficulties in the process of traffic object detection indeed, such as occlusion and aggregation between objects, insufficient feature extraction of objects, in particular the presence of a large number of small objects, which bring great challenges to these traffic objects detection. In this paper, an improved traffic object detection model based on You-Only-Look-Once version 5 small (YOLOv5s) is proposed to address the issues. By utilizing spatial pyramids to extract multi-scale spatial features and applying Squeeze-and-Excitation (SE) channel attention to capture more global and local semantic features, especially by designing a sub-network in the neck to fuse high-resolution information in shallow layers with more accurately semantic information in deep layers, the detection sensitivity of object features is enhanced. More importantly, by explanting decoupled-head into the network, outstanding performance of the model with high detection accuracy and rapid detection speed is realized. The experimental results on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) and Laboratory for Intelligent and Safe Automobiles (LISA) traffic signs datasets both show that the modified model significantly improves the detection accuracy. Meanwhile, the high real-time performance is still maintained. Undoubtedly, the modified model proposed in this paper can effectively address many difficulties in traffic object detection under various complex scenes, which would be greatly helpful for its potential applications in the future.

Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, Yu Zhang. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Nuclei instance segmentation on histopathology images is of great clinical value for disease analysis. Generally, fully-supervised algorithms for this task require pixel-wise manual annotations, which is especially time-consuming and laborious for the high nuclei density. To alleviate the annotation burden, we seek to solve the problem through image-level weakly supervised learning, which is underexplored for nuclei instance segmentation. Compared with most existing methods using other weak annotations (scribble, point, etc.) for nuclei instance segmentation, our method is more labor-saving. The obstacle to using image-level annotations in nuclei instance segmentation is the lack of adequate location information, leading to severe nuclei omission or overlaps. In this paper, we propose a novel image-level weakly supervised method, called cyclic learning, to solve this problem. Cyclic learning comprises a front-end classification task and a back-end semi-supervised instance segmentation task to benefit from multi-task learning (MTL). We utilize a deep learning classifier with interpretability as the front-end to convert image-level labels to sets of high-confidence pseudo masks and establish a semi-supervised architecture as the back-end to conduct nuclei instance segmentation under the supervision of these pseudo masks. Most importantly, cyclic learning is designed to circularly share knowledge between the front-end classifier and the back-end semi-supervised part, which allows the whole system to fully extract the underlying information from image-level labels and converge to a better optimum. Experiments on three datasets demonstrate the good generality of our method, which outperforms other image-level weakly supervised methods for nuclei instance segmentation, and achieves comparable performance to fully-supervised methods.

A high level of HER2 expression in breast cancer correlates with a higher tumor growth rate, high metastatic potential, and a poor long-term patient survival rate. Pertuzumab, a human monoclonal antibody, can reduce the effect of HER2 overexpression by preventing HER2 dimerization. In this study, a combination protocol of molecular dynamics modeling and MM/GBSA binding free energy calculations was applied to design peptides that interact with HER2 based on the HER2/pertuzumab crystal structure. Based on a β hairpin in pertuzumab from Glu46 to Lys65-which plays a key role in interacting with HER2-mutations were carried out in silico to improve the binding free energy of the hairpin that interacts with the Phe256-Lys314 of the HER2 protein. Combined the use of one-bead-one-compound library screening, among all the mutations, a peptide (58F63Y) with the lowest binding free energy was confirmed experimentally to have the highest affinity, and it may be used as a new probe in diagnosing and treating HER2-positive breast cancer.