Incorporating human feedback to optimize text-to-image models has demonstrated significant effectiveness. However, the process of collecting high-quality human preference labels is both resource-intensive and time-consuming. To address this challenge, we propose a novel approach that leverages a large language model (LLM) to generate sophisticated prompts, guiding the diffusion model towards enhanced image generation. This process inherently produces ranking pairs that approximate human preferences. We further introduce a novel integration of AI feedback with a Supervised Fine-Tuning (SFT) policy, aligning the model with preference labels derived from AI. Our experiments demonstrate that our approach achieves a notable approximation of human preferences, achieving a performance level of 68.13% compared to human-level benchmarks and delivering competitive results. Furthermore, we showcase the synergistic effects of combining AI feedback with human feedback, resulting in further improvements in image quality. This research offers fresh insights into AI feedback learning within text-to-image generation and lays the groundwork for more efficient and cost-effective training methodologies.

The role of high-quality pseudo-labels is pivotal in semi-supervised instance segmentation (SSIS). However, existing SSIS frameworks predominantly produce pseudo-labels at a single resolution, which can introduce noise that adversely affects the quality of learning at both the pixel level and in terms of class discrimination. This paper introduces the Dual-Resolution Pseudo-Label Integration for Semi-Supervised Instance Segmentation (DuPI), a novel framework designed to enhance learning by integrating pseudo-labels derived from dual-resolution inputs. The DuPI framework incorporates a Dual-Resolution Pseudo-Label Correction (DPC) module, which refines pseudo-labels through a process of cross-resolution rectification and fusion. Furthermore, the framework introduces an Area-Adaptive Learning (AAL) strategy aimed at enhancing the quality of pseudo-labels sourced from extra-resolution inputs. The AAL strategy addresses the training challenges associated with small objects at lower resolutions by re-weighting pseudo-labels corresponding to tiny mask areas using Intersection over Union (IoU) metrics from the assignments. Experiments on the COCO and BDD100K datasets demonstrate that DuPI achieves state-of-the-art SSIS performance under various semi-supervised settings.

Camouflaged Object Detection (COD) poses a significant challenge in computer vision, playing a critical role in applications. Existing COD methods often exhibit challenges in accurately predicting nuanced boundaries with high-confidence predictions. In this work, we introduce CamoDiffusion, a new learning method that employs a conditional diffusion model to generate masks that progressively refine the boundaries of camouflaged objects. In particular, we first design an adaptive transformer conditional network, specifically designed for integration into a Denoising Network, which facilitates iterative refinement of the saliency masks. Second, based on the classical diffusion model training, we investigate a variance noise schedule and a structure corruption strategy, which aim to enhance the accuracy of our denoising model by effectively handling uncertain input. Third, we introduce a Consensus Time Ensemble technique, which integrates intermediate predictions using a sampling mechanism, thus reducing overconfidence and incorrect predictions. Finally, we conduct extensive experiments on three benchmark datasets that show that: 1) the efficacy and universality of our method is demonstrated in both camouflaged and salient object detection tasks. 2) compared to existing state-of-the-art methods, CamoDiffusion demonstrates superior performance 3) CamoDiffusion offers flexible enhancements, such as an accelerated version based on the VQ-VAE model and a skip approach.

Camouflaged Object Detection (COD) is a challenging task in computer vision due to the high similarity between camouflaged objects and their surroundings. Existing COD methods struggle with nuanced object boundaries and overconfident incorrect predictions. In response, we propose a new paradigm that treats COD as a conditional mask-generation task leveraging diffusion models. Our method, dubbed CamoDiffusion, employs the denoising process to progressively refine predictions while incorporating image conditions. Due to the stochastic sampling process of diffusion, our model is capable of sampling multiple possible predictions, avoiding the problem of overconfident point estimation. Moreover, we develop specialized network architecture, training, and sampling strategies, to enhance the model’s expressive power, refinement capabilities and suppress overconfident mis-segmentations, thus aptly tailoring the diffusion model to the demands of COD. Extensive experiments on three COD datasets attest to the superior performance of our model compared to existing state-of-the-art methods, particularly on the most challenging COD10K dataset, where our approach achieves 0.019 in terms of MAE. Codes and models are available at https://github.com/Rapisurazurite/CamoDiffusion.

The Segment Anything Model (SAM) marks a notable milestone in segmentation models, highlighted by its robust zero-shot capabilities and ability to handle diverse prompts. SAM follows a pipeline that separates interactive segmentation into image preprocessing through a large encoder and interactive inference via a lightweight decoder, ensuring efficient real-time performance. However, SAM faces stability issues in challenging samples upon this pipeline. These issues arise from two main factors. Firstly, the image preprocessing disables SAM to dynamically use image-level zoom-in strategies to refocus on the target object during interaction. Secondly, the lightweight decoder struggles to sufficiently integrate interactive information with image embeddings. To address these two limitations, we propose FocSAM with a pipeline redesigned on two pivotal aspects. First, we propose Dynamic Window Multi-head Self-Attention (Dwin-MSA) to dynamically refocus SAM's image embeddings on the target object. Dwin-MSA localizes attention computations around the target object, enhancing object-related embeddings with minimal computational overhead. Second, we propose Pixel-wise Dynamic ReLU (P-DyReLU) to enable sufficient integration of interactive information from a few initial clicks that have significant impacts on the overall segmentation results. Experimentally, FocSAM augments SAM's interactive segmentation performance to match the existing state-of-the-art method in segmentation quality, requiring only about 5.6% of this method's inference time on CPUs. Code is available at https://github.com/YouHuang67/focsam.

Video-text retrieval has widespread applications in economic and security domains, making it crucial to evaluate its robustness through adversarial attack. However, the existing research in this field is inadequate. In this paper, we first introduce adversarial attack to this task. By leveraging the concept of metric learning, we propose novel attack methods Cross-modal Dual Level Contrastive Attack (CDCA) and Cross-modal Rank Pairing Attack (CRPA). In the white-box scenario, CDCA utilizes the distribution of head and tail examples in the retrieval list to form positive and negative example sets, employing both coarse and fine-grained features. In the black-box scenario, CRPA employs the rank difference in retrieval list as example pairs and utilizes the Rank Difference Loss (RDL) as the attack objective function. Experiments validate the superiority of our methods. Furthermore, we contribute a benchmark, which lays a foundation for understanding the vulnerability of multi-modal models.

Masked autoencoders have become popular training paradigms for self-supervised visual representation learning. These models randomly mask a portion of the input and reconstruct the masked portion according to assigned target representations. In this paper, we show that a careful choice of the target representation is unnecessary for learning good visual representation since different targets tend to derive similarly behaved models. Driven by this observation, we propose a multi-stage masked distillation pipeline and use a randomly initialized model as the teacher, enabling us to effectively train high-capacity models without any effort to carefully design the target representation. On various downstream tasks, the proposed method to perform masked knowledge distillation with bootstrapped teachers (dbot) outperforms previous self-supervised methods by nontrivial margins. We hope our findings, as well as the proposed method, could motivate people to rethink the roles of target representations in pre-training masked autoencoders.

Recent works have indicated that deep neural networks (DNNs) are vulnerable to adversarial attacks, wherein an attacker perturbs an input example with human-imperceptible noise that can easily fool the DNNs, resulting in incorrect predictions. This severely limits the application of deep learning in security-critical scenarios, such as face authentication. Adversarial training (AT) is one of the most practical approaches to strengthening the robustness of DNNs. However, existing AT-based methods treat each training sample independently, thereby ignoring the underlying topological structure in the training data. To this end, in this paper, we take full advantage of the topology information and introduce a Topology Aligning Adversarial Training (TAAT) algorithm. TAAT aims to encourage the trained model to maintain consistency in the topological structure within the feature space of both natural and adversarial examples. To ensure the stability and efficiency of topology alignment, we further introduce a novel Knowledge-Guided (KG) training scheme. This scheme explicitly aligns local logit outputs with global topological structures, leveraging a robust auxiliary model to enhance the target model’s performance. To verify the effectiveness of the proposed method, we conduct extensive experiments on popular benchmark datasets (e.g., CIFAR and ImageNet) and evaluate the robustness against state-of-the-art adversarial attacks (e.g., PGD-attack and AutoAttack). The experimental results demonstrate that the proposed method has superior robustness over the previous state-of-the-art methods. Our code and pre-trained models are available at https://github.com/SkyKuang/TAAT.

The task of Human-Object Interaction (HOI) detection is to detect humans and their interactions with surrounding objects, where transformer-based methods show dominant advances currently. However, these methods ignore the relationship among humans, objects, and interactions: 1) human features are more contributive than object ones to interaction prediction; 2) interactive information disturbs the detection of objects but helps human detection. In this article, we propose aHuman and Object Disentangling Network(HODN) to model the HOI relationships explicitly, where humans and objects are first detected by two disentangling decoders independently and then processed by an interaction decoder. Considering that human features are more contributive to interaction, we propose aHuman-Guide Linkingmethod to make sure the interaction decoder focuses on the human-centric regions with human features as the positional embeddings. To handle the opposite influences of interactions on humans and objects, we propose aStop-Gradient Mechanismto stop interaction gradients from optimizing the object detection but to allow them to optimize the human detection. Our proposed method achieves competitive performance on both the V-COCO and the HICO-Det datasets. It can be combined with existing methods easily for state-of-the-art results.

Video-text retrieval has greatly benefited from the massive web video in recent years, while the performance is still limited to the weak supervision from the uncurated data. In this work, we propose to leverage the well-represented information of each original modality and exploit complementary information in two views of the same video, i.e., video clips and captions, by using one view to obtain positive samples with the neighboring samples of the other. Respecting the hierarchical organization of real-world data, we further design a hierarchical cross-modal pre-training method (HCP) to learn good representations in the common embedding space. We evaluate the pre-trained model on three downstream tasks, i.e. text-to-video retrieval, action step localization and video question answering and our method outperforms previous works under the same setting.

Human-Object Interaction (HOI) detection aims to understand the interactions between humans and objects, which plays a curtail role in high-level semantic understanding tasks. However, most works pursue designing better architectures to learn overall features more efficiently, while ignoring the long-tail nature of interaction-object pair categories. In this paper, we propose to alleviate the impact of such an unbalanced distribution via Virtual Image Leaning (VIL). Firstly, a novel label-to-image approach, Multiple Steps Image Creation (MUSIC), is proposed to create a high-quality dataset that has a consistent distribution with real images. In this stage, virtual images are generated based on prompts with specific characterizations and selected by multi-filtering processes. Secondly, we use both virtual and real images to train the model with the teacher-student framework. Considering the initial labels of some virtual images are inaccurate and inadequate, we devise an Adaptive Matching-and-Filtering (AMF) module to construct pseudo-labels. Our method is independent of the internal structure of HOI detectors, so it can be combined with off-the-shelf methods by training merely 10 additional epochs. With the assistance of our method, multiple methods obtain significant improvements, and new state-of-the-art results are achieved on two benchmarks.

Transfer adversarial attack is a non-trivial black-box adversarial attack that aims to craft adversarial perturbations on the surrogate model and then apply such perturbations to the victim model. However, the transferability of perturbations from existing methods is still limited, since the adversarial perturbations are easily overfitting with a single surrogate model and specific data pattern. In this paper, we propose a Learning to Learn Transferable Attack (LLTA) method, which makes the adversarial perturbations more generalized via learning from both data and model augmentation. For data augmentation, we adopt simple random resizing and padding. For model augmentation, we randomly alter the back propagation instead of the forward propagation to eliminate the effect on the model prediction. By treating the attack of both specific data and a modified model as a task, we expect the adversarial perturbations to adopt enough tasks for generalization. To this end, the meta-learning algorithm is further introduced during the iteration of perturbation generation. Empirical results on the widely-used dataset demonstrate the effectiveness of our attack method with a 12.85% higher success rate of transfer attack compared with the state-of-the-art methods. We also evaluate our method on the real-world online system, i.e., Google Cloud Vision API, to further show the practical potentials of our method.

Consider a robust multiple-input single-output downlink beamforming optimization problem in a frequency division duplexing system. The base station (BS) sends training signals to the users, and every user estimates the channel coefficients, quantizes the gain and the direction of the estimated channel and sends them back to the BS. Suppose that the channel state information at the transmitter is imperfectly known mainly due to the channel direction quantization errors, channel estimation errors and outdated channel effects. The actual channel is modeled as in an uncertainty set composed of two inequality homogeneous and one equality inhomogeneous quadratic constraints, in order to account for the aforementioned errors and effects. Then the transmit power minimization problem is formulated subject to robust signal-to-noise-plus-interference ratio constraints. Each robust constraint is transformed equivalently into a quadratic matrix inequality (QMI) constraint with respect to the beamforming vectors. The transformation is accomplished by an equivalent phase rotation process and the strong duality result for a quadratically constrained quadratic program. The minimization problem is accordingly turned into a QMI problem, and the problem is solved by a restricted linear matrix inequality relaxation with additional valid convex constraints. Simulation results are presented to demonstrate the performance of the proposed method, and show the efficiency of the restricted relaxation.

Vehicle Re-Identification targets at searching for vehicle instances of the same identity with a given query. It has gained an increasing attention recently with a wide application prospects in video surveillance and intelligent transportation. The main challenge lies in how to distinguish the subtle differences between different vehicles, while capturing the slight similarity between the instance of the same vehicle in different viewpoints or illuminations. To this end, most existing methods focus on learning discriminative global representations, which leave the unique local details such as stickers, inspection labels and driver wearing being unexploited. In this paper, we present a novel coarse-to-fine scheme that aggregates global and local visual representations to boost the retrieval accuracy. Specifically, a multi-task learning framework combining an Attribute Learning branch and a Deep Ranking branch (termed ALDR) is first adopted to learn robust global features, which produces an initial ranking list. Then the local similarities between image patches in the initial ranking list and in the query is computed via a Multi-Channel and Multi-Scale Siamese network (termed MCMS-Siam). Finally, the retrieval result is returned after re-ranking the initial list according to such a combination of global and local similarities. Experimental results on the widely-used VehicleID dataset and VECH-WILD dataset demonstrate the merits of the proposed method over the state-of-the-art methods.

Most existing fine-grained image retrieval schemes are built based upon deep feature learning paradigms, which typically leverage the feature maps of the last convolutional layer as features. However, such representation focuses only on the global information of the object, leaving the local details unexploited, which is however crucial to identifying subtle differences for fine-grained retrieval. In this paper, we have discovered that the mid-level feature map roles as local salient regions, which well complements the existing global feature representations. To this end, a multi-layer framework is proposed to integrate both local and global representations with generalized mean (GeM) pooling and attention mechanism, trained with the proposed N-pair Center loss to learn more discriminative features. By doing so, state-of-the-art performance can be achieved without using the hard or negative example minings. In the experiments, our approach outperforms favourably compared to the current state-of-the-art methods on the CUB-200-2011, CARS196 and In-shop Clothes Retrieval datasets.

Fisher Vector (FV) has been widely used to aggregate the local descriptors of an image into a global representation in large-scale image retrieval. However, FV has limited learning capability and its parameters are mostly fixed after constructing the codebook, which is inflexible and cannot be trained jointly with deep networks. Moreover, the high dimension of FV makes it difficult to be applied in scenarios compact descriptors are needed. In this paper, we propose a novel compact image description scheme based on Fisher network with binary embedding to solve the large-scale image retrieval problem, which consists of two components: a Fisher encoder component and a binary embedding component. Concretely, the Fisher encoder is a trainable neural network functions as the traditional FV, which aggregates the local descriptors into a global representation. And the binary encoder embeds the high-dimensional FV to a binary vector, which outputs the compact global binary descriptor. To learn such a descriptor, we further introduce a novel and effective loss function, in which maximum margin criterion is exploited to minimize the distances of positive pairs, as well as maximizing the distances of negative pairs. Extensive experiments performed on MPEG-7 CDVS benchmarks and ILSVR2010 demonstrate that the proposed framework can achieve very superior performance over the state-of-the-art methods.

We tackle the problem of mobile visual search. Moving pictures experts group (MPEG) has completed a standard named compact descriptor for visual search (CDVS) to provide a standardized syntax in the context of image retrieval application. CDVS applies principal components analysis to reduce the dimension of local feature descriptor as the input of global descriptor pipeline, and utilizes traditional fisher vector as the local feature descriptor aggregation algorithm. However, the descriptor components of SIFT and Fisher Vector (FV) have highly non-Gaussian statistics, and applying a single PCA transform can in-fact hurt compression performance at high rates. We develop a net-based architecture combining neural networks with FV layer to obtain fisher vector. There are two advantages in our architecture comparing with CDVS global descriptor pipeline. One is that we employ “autoencoder” networks to reduce the dimensionality of data, the other is that we exploit a trainable system to learn parameters after the FV codebook obtained. The experiments demonstrate an obvious advantage of our proposed architecture in terms of CDVS retrieval task.

Eye state estimation has been widely applied in many real-world systems, e.g., driver monitoring and smart television. However, it is still an open issue towards accurate and realtime estimation, especially under the scenario of uncontrolled scenes, for instance severely occlusion, viewpoint change, and illumination variances. In this paper, we introduce a robust eye state estimation algorithm, the core of which is a novel discriminative and fast feature encoding scheme to represent the eye state. To further improve the robustness, we also introduce a brightness adjustment algorithm to overcome the variances of environmental illumination. The experimental results demonstrate that our algorithm is extremely effective under various illumination conditions. The proposed algorithm has also been integrated into the next-generation AOC intelligent TV systems.

Objective: To evaluate the quality of three type of literature, clinical randomized controlled trials(RCTs), case-control trials and case series trials, about acupuncture on tension-type headache. Methods: Carried out electronic retrieval on several databases, including CNKI, VIP, Wanfang, PubMed and Embase database, to widely collect target documents that published between 2001 and 2013. Then to evaluate the quality of these documents, according to the clinical epidemiology and evidence-based medicine principles and methods. Results: A total of 26 documents met the criteria. 13 of them were RCTs, 3 were case-control trials and 10 case series trials. The mean percent scores of these RCTs, case-control trials and case series trials were 35.3, 23.5 and 33.75 respectively. Conclusion: For the reports of acupuncture for tension-type headache, the overall quality RCTs, case-control trials and case series trials were low. It should be paid great attention to introduce international advanced writing specification, in order to provide more detailed and reliable scientific data for clinical practices.

Objective: Through data mining technology to summarize the application characteristics of acupoints in the treatment of tension-type headache with acupuncture in modern times. Methods: CNKI, VIP, Wanfang and PubMed databases as our retrieval source were used to collect literature about acupuncture on tention-type headache, SPSS 18.0 software to establish data and IBM SPSS modeler 14.1 Data Mining Software to analysis. Then, Calculate the Support degree(S), confidence coefficient (C) and Lift(L) of acupoints itemsets to select the common acupoints as well as to summarize the law of choosing acupoints. Results: Participants included 27 qualified paper. From the point of the statistical results of association rules first, we found that the law of acupoints selection gave priority to points located in head Yang meridians, the most cited acupoints was GB20 (S 77.78%), and then the EX - HN5 (S 66.67%) and DU20 (S 51.85%). In addition, The Compatibility of three acupoints were also the most commonly. Support is 44.44%, the Confidence is 100%, the Lift is 1.5. the Gallbladder Meridian of foot-Shaoyang, the Stomach Meridian of Foot-yangming and the Governor meridian were the most common used meridian. Conclusion: According to the results of data mining, the law of choosing acupoints on tension-type headache mainly embodied in selecting acupoints located in head combined with distinguishing meridian. The results of apriori algorithm were agreed with the meridian theory of TCM and worth spreading and using in clinic.