Recent advances in vision-language pre-training have significantly enhanced the model capabilities on grounded object detection. However, these studies often pre-train with coarse-grained text prompts, such as plain category names and brief grounded phrases. This limitation curtails the model's capacity for fine-grained linguistic comprehension and leads to a significant decline in performance when faced with detailed descriptions or contextual information. To tackle these problems, we develop DoGA: Detect objects with Grouped Attributes, which employs commonly apparent attributes to bridge different granular semantics and uses specific attributes to identify the object discrepancy. Our DoGA incorporates three principle components: 1) Generation of attribute-based prompts, consisting of linguistic definitions enriched with common-sense visible attributes and hard negative notations deriving from the image-specific attribute features; 2) Paralleled entity fusion and optimization, designed to manage long attribute-based descriptions and negative concepts efficiently; and 3) Prompt-wise grouped training to accommodate model to perform many-to-many assignments, facilitating simultaneous training and inferring with multiple attribute-based synonyms. Extensive experiments demonstrate that training with synonymous attribute-based prompts allows DoGA to generalize multi-granular prompts and surpass previous state-of-the-art approaches, yielding 50.2 on the COCO and 38.0 on the LVIS benchmarks under the zero-short setting. We will make our code publicly available upon acceptance.

Yepeng Weng, Dianwen Mei, Huishi Qiu, Xujie Chen, Li Liu, Jiang Tian, Zhongchao Shi. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_{\text{X}}$ and ARB-LLM$ _{\text{RC}} $ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$ _{\text{RC}} $ is the first to surpass FP16 models of the same size. Code: https://github.com/ZHITENGLI/ARB-LLM.

Speculative decoding is a promising approach for accelerating large language models. The primary idea is to use a lightweight draft model to speculate the output of the target model for multiple subsequent timesteps, and then verify them in parallel to determine whether the drafted tokens should be accepted or rejected. To enhance acceptance rates, existing frameworks typically construct token trees containing multiple candidates in each timestep. However, their reliance on token-level verification mechanisms introduces two critical limitations: First, the probability distribution of a sequence differs from that of individual tokens, leading to suboptimal acceptance length. Second, current verification schemes begin from the root node and proceed layer by layer in a top-down manner. Once a parent node is rejected, all its child nodes should be discarded, resulting in inefficient utilization of speculative candidates. This paper introduces Traversal Verification, a novel speculative decoding algorithm that fundamentally rethinks the verification paradigm through leaf-to-root traversal. Our approach considers the acceptance of the entire token sequence from the current node to the root, and preserves potentially valid subsequences that would be prematurely discarded by existing methods. We theoretically prove that the probability distribution obtained through Traversal Verification is identical to that of the target model, guaranteeing lossless inference while achieving substantial acceleration gains. Experimental results on various models and multiple tasks demonstrate that our method consistently improves acceptance length and throughput over token-level verification.

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

Food recognition has a wide range of applications, such as health-aware recommendation and self-service restaurants. Most previous methods of food recognition firstly locate informative regions in some weakly-supervised manners and then aggregate their features. However, location errors of informative regions limit the effectiveness of these methods to some extent. Instead of locating multiple regions, we propose a Progressive Self-Distillation (PSD) method, which progressively enhances the ability of network to mine more details for food recognition. The training of PSD simultaneously contains multiple self-distillations, in which a teacher network and a student network share the same embedding network. Since the student network receives a modified image from its teacher network by masking some informative regions, the teacher network outputs stronger semantic representations than the student network. Guided by such teacher network with stronger semantics, the student network is encouraged to mine more useful regions from the modified image by enhancing its own ability. The ability of the teacher network is also enhanced with the shared embedding network. By using progressive training, the teacher network incrementally improves its ability to mine more discriminative regions. In inference phase, only the teacher network is used without the help of the student network. Extensive experiments on three datasets demonstrate the effectiveness of our proposed method and state-of-the-art performance.

Recently, the dominant DETR-based approaches apply central-concept spatial prior to accelerating Transformer detector convergency. These methods gradually refine the reference points to the center of target objects and imbue object queries with the updated central reference information for spatially conditional attention. However, centralizing reference points may severely deteriorate queries' saliency and confuse detectors due to the indiscriminative spatial prior. To bridge the gap between the reference points of salient queries and Transformer detectors, we propose SAlient Point-based DETR (SAP-DETR) by treating object detection as a transformation from salient points to instance objects. Concretely, we explicitly initialize a query-specific reference point for each object query, gradually aggregate them into an instance object, and then predict the distance from each side of the bounding box to these points. By rapidly attending to query-specific reference regions and the conditional box edges, SAP-DETR can effectively bridge the gap between the salient point and the query-based Transformer detector with a significant convergency speed. Experimentally, SAP-DETR achieves 1.4× convergency speed with competitive performance and stably promotes the SoTA approaches by ∼1.0 AP. Based on ResNet-DC-101, SAP-DETR achieves 46.9 AP. The code will be released at https://github.com/liuyang-ict/SAP-DETR.

Most previous approaches for analyzing food images have relied on extensively annotated datasets, resulting in significant human labeling expenses due to the varied and intricate nature of such images. Inspired by the effectiveness of contrastive self-supervised methods in utilizing unlabelled data, weiqing explore leveraging these techniques on unlabelled food images. In contrastive self-supervised methods, two views are randomly generated from an image by data augmentations. However, regarding food images, the two views tend to contain similar informative contents, causing large mutual information, which impedes the efficacy of contrastive self-supervised learning. To address this problem, we propose Feature Suppressed Contrast (FeaSC) to reduce mutual information between views. As the similar contents of the two views are salient or highly responsive in the feature map, the proposed FeaSC uses a response-aware scheme to localize salient features in an unsupervised manner. By suppressing some salient features in one view while leaving another contrast view unchanged, the mutual information between the two views is reduced, thereby enhancing the effectiveness of contrast learning for self-supervised food pre-training. As a plug-and-play module, the proposed method consistently improves BYOL and SimSiam by 1.70% ~ 6.69% classification accuracy on four publicly available food recognition datasets. Superior results have also been achieved on downstream segmentation tasks, demonstrating the effectiveness of the proposed method.

Recently, deep learning (DL)-based skin lesion segmentation in dermoscopic images has advanced the efficient diagnosis of skin diseases. Commonly, most of the DL-based methods require a large amount of training data and can only perform accurate predictions on pre-defined classes. However, there exist some rare skin diseases with very limited labeled samples, which poses great challenges to typical DL-based methods. Few-shot learning (FSL) technique, which aims to train models with abundant seen classes and then generalizes to related unseen classes, is promising in addressing a similar problem. Unfortunately, simply borrowing the typical FSL is infeasible since collecting such abundant seen-class data (common skin diseases), is also difficult. In this paper, we propose a cross-domain few-shot segmentation (CD-FSS) framework, which enables the model to leverage the learning ability obtained from the natural domain, to facilitate rare-disease skin lesion segmentation with limited data of common diseases. Specifically, the framework consists of two processes, i.e., specific learning and generic learning, which are alternately optimized in a meta-training manner. A specific learner and a generic learner are tailored to build relationships between both processes. Experimental results demonstrate that our framework significantly improves the generalization ability from natural domain to unseen medical domain.

This paper proposes a novel Boundary-aware Consistent Hidden Representation Learning Network (BA-CHRLN), which contains two branches for semi-supervised 3D medical image segmentation. Inspired by the contrastive learning, the two branches share the same encoder and each has its individual decoder, namely supervised decoder and unsupervised one. A stop-gradient operation is also utilized to prevent collapsing of solutions. Taking the unlabeled images as references, BA-CHRLN imposes the consistency by applying a perturbation on the high-level hidden feature representations, which significantly improves the encoder’s representation and the network’s robustness. A boundary-aware map is further introduced to capture the organ’s boundary without any prior knowledge and additional parameters. Experiments on the Left Atrium (LA) benchmark dataset demonstrate the effectiveness of the BA-CHRLN.

Automated organ segmentation in CTs is an essential prerequisite for many clinical applications, such as computer-aided diagnosis and intervention. As medical data annotation requires massive human labor from experienced radiologists, how to effectively improve the segmentation performance with limited annotated training data remains a challenging problem. Few-shot learning imitates the learning process of humans, which turns out to be a promising way to overcome the aforementioned challenge. In this paper, we propose a novel anatomical knowledge embedded few-shot network (AKFNet), where an anatomical knowledge embedded support unit (AKSU) is carefully designed to embed the anatomical priors from support images into our model. Moreover, a similarity guidance alignment unit (SGAU) is proposed to impose a mutual alignment between the support and query sets. As a result, AKFNet fully exploits anatomical knowledge and presents good learning capability. Without bells and whistles, AKFNet outperforms the state-of-the-art methods with 0.84-1.76% Dice increase. Transfer learning experiments further verify its learning capability.

Automatic liver tumor segmentation from 3D Computed Tomography (CT) is a necessary prerequisite in the interventions of hepatic abnormalities and surgery planning. However, accurate liver tumor segmentation remains challenging due to the large variability of tumor sizes and inhomogeneous texture. Recent advances based on Fully Convolutional Network (FCN) in liver tumor segmentation draw on success of learning discriminative multi-level features. In this paper, we propose a Deep Attentive Refinement Network (DARN) for improved liver tumor segmentation from CT volumes by fully exploiting both low and high level features embedded in different layers of FCN. Different from existing works, we exploit attention mechanism to leverage the relation of different levels of features encoded in different layers of FCN. Specifically, we introduce a Semantic Attention Refinement (SemRef) module to selectively emphasize global semantic information in low level features with the guidance of high level ones, and a Spatial Attention Refinement (SpaRef) module to adaptively enhance spatial details in high level features with the guidance of low level ones. We evaluate our network on the public MICCAI 2017 Liver Tumor Segmentation Challenge dataset (LiTS dataset) and it achieves state-of-the-art performance. The proposed refinement modules are an effective strategy to exploit multi-level features and has great potential to generalize to other medical image segmentation tasks.

This paper presents an efficient method to 3D model compression based on repetition detection. The proposed Pattern-Based 3D Mesh Codec (PB3DMC) can achieve good rate-distortion performance on 3D models comprising multiple components. The repetition among constituent components is first exploited to generate a compact representation. An optimal bit allocation scheme is then proposed in order to compress the resultant "pattern-instances" representation. Experimental results show that PB3DMC yields a significant gain compared to the algorithms in MPEG's Scalable Complexity 3D Mesh Coding (SC3DMC) toolset, particularly for those models containing repetitive components. Furthermore, a benchmark for PB3DMC is built using 444 models. And PB3DMC is going to be published as an amendment of, MPEG-4 standard.

In this paper, we propose a method to measure the relationship between data samples, which is dependent on the possibility whether they are within a homogeneous region or not. By considering the regional variation, this possibility is formulated in terms of the maximum local variation along the shortest path connecting the samples. The metric is concretized in both 2D images and 3D meshes, and then integrated into smoothing filters. Benefited from our method, the improved filters tend to effectively preserve the structural component of data. Moreover, our method is implemented in various applications such as image denoising, image decomposition and mesh smoothing, which demonstrates better performance in comparison to the previous work.

Summary form only given. Octree decomposition has been proven to be one of the most successful approaches for progressive geometry compression of 3D models. An octree is built up by recursively subdividing the bounding box of 3D models into a number of sub-cells. Each node of the octree has an 8-bit-binary code called occupancy code, indicating the non- emptiness of its children, given a pre-defined traversal order. Then the point positions can be represented by a sequence of occupancy codes. Typically, the non-empty child-cells are close to evenly distributed among the 8 candidate positions. It has been proposed to change the traversal order based on the probabilities of the child-cells being non-empty; then the statistical distribution of the occupancy codes becomes more concentrated, which is beneficial for compression. We observe an intrinsic property of 3D models that their tangent-planes tend to be continuous at high fidelity layers. Thus, we take the tangent-plane continuity as a criterion for the non-emptiness estimation. The continuity is measured by the surface area of the convex hull which is formed by current child-cell's centroid and centroids of the parent neighbors. Our algorithm results in more concentrated distribution of occupancy code than the existing reordering approach. Therefore, our codec outperforms the existing work significantly.

Octree decomposition has been proven to be one of the most successful approaches for progressive geometry compression of 3D models. To exploit the spatial correlation among the points, people strive to improve the estimation of the non-emptiness of the child-cells during space subdivision. We observe an intrinsic property of 3D models that their tangent planes tend to be continuous at high fidelity layers. Thus, we take the tangent-plane-continuity as a criterion for the non-emptiness estimation, and design a 3D model compression scheme based on this idea. The experimental results show that the proposed codec improves the quality by up to 6dB at equivalent bit-rates.

Grasping a deformable object instantaneously requires maintaining equilibrium of its pre- and post-deformed shapes using the same set of forces. This paper studies the type of grasps generated by squeezing a planar object with two fingers. It is shown that the success of such a grasp is independent of the applied forces in the case of small deformation. Numerical algorithms are introduced to compute sets of squeeze grasps with small and large deformations modeled using the finite element method (FEM) based on the linear and nonlinear elasticity theories, respectively.

This paper studies the reconstruction of unknown curved surfaces through finger tracking. A patch can be generated from tactile data points along three concurrent surface curves under the Darboux frame estimated at the curve intersection point. Surface fitting while minimizing the total (absolute) Gaussian curvature effectively prevents unnecessary folds otherwise expected to result from the use of such "1-D" data. The implementation involves a two-axis joystick sensor, a three-fingered 4-DOF BarrettHand, and a 4-DOF Adept SCARA robot. Experiments have demonstrated good accuracy of reconstruction.

The robot hand applying force on a deformable object will result in a changing wrench space due to the varying shape and normal of the contact area. Design and analysis of a manipulation strategy thus depend on reliable modeling of the object's deformations as actions are performed. In this paper, shell-like objects are modeled. The classical shell theory [P. L. Gould, Analysis of Plates and Shells. Englewood Cliffs, NJ: Prentice-Hall, 1999; V. V. Novozhilov, The Theory of Thin Shells . Gronigen, The Netherlands: Noordhoff, 1959; A. S. Saada, Elasticity: Theory and Applications. Melbourne, FL: Krieger, 1993; S. P. Timoshenko and S. Woinowsky-Krieger, Theory of Plates and Shells, 2nd ed. New York: McGraw-Hill, 1959] assumes a parametrization along the two lines of curvature on the middle surface of a shell. Such a parametrization, while always existing locally, is very difficult, if not impossible, to derive for most surfaces. Generalization of the theory to an arbitrary parametric shell is therefore not immediate. This paper first extends the linear and nonlinear shell theories to describe extensional, shearing, and bending strains in terms of geometric invariants, including the principal curvatures and vectors, and their related directional and covariant derivatives. To our knowledge, this is the first nonparametric formulation of thin-shell strains. A computational procedure for the strain energy is then offered for general parametric shells. In practice, a shell deformation is conveniently represented by a subdivision surface [F. Cirak, M. Ortiz, and P. Schröder, “Subdivision surfaces: A new paradigm for thin-shell finite-element analysis,” Int. J. Numer. Methods Eng., vol. 47, pp. 2039-2072, 2000]. We compare the results via potential-energy minimization over a couple of benchmark problems with their analytical solutions and numerical ones generated by two commercial software packages: ABAQUS and ANSYS. Our method achieves a convergence rate that is one order of magnitude higher. Experimental validation involves regular and free-form shell-like objects of various materials that were grasped by a robot hand, with the results compared against scanned 3-D data with accuracy of 0.127 mm. Grasped objects often undergo sizable shape changes, for which a much higher modeling accuracy can be achieved using the nonlinear elasticity theory than its linear counterpart.

Large-scale genome analysis and drug discovery require an automated prediction method for protein subcellular localization, and Support Vector Machines (SVMs) effectively solve this problem in a supervised manner. However, the protein subcellular localization datasets obtained from experiments always contain outliers, which can lead to poor generalization ability and classification accuracy. To address this issue, we first analyzed the influence of Principal Component Analysis (PCA) on classification performance, and then proposed a hybrid method for prediction of protein subcellular localization based on Weighted Supported Vector Machine (WSVM) and PCA. Different weights were assigned to different data points, so the training algorithm could learn the decision boundary according to the relative importance of the data points. After performing dimension reduction operations on the datasets, kernel-based possibilistic c-means (KPCM) was chosen to generate weights for this algorithm, as it generates relative high values for important data points but low values for outliers. Experimental results on a benchmark dataset show promising results, which confirms the effectiveness of the proposed method in terms of prediction accuracy.

This paper models (large) deformations of shelllike objects under the grasping of a robot hand. Classical nonlinear theory of thin shells [21, pp. 186-194] is generalized to shells with arbitrary parametric middle surfaces, using a method introduced in our earlier work [13]. An experimental study demonstrates higher modeling accuracy using the nonlinear elasticity theory than its linear counterpart. Given that many deformable objects undergo sizable shape changes when they are grasped, our result supports the application of nonlinear elasticity theory in the future design of grasp strategies for this type of objects.

Health delivery systems, involving patients, physicians, health service providers, medication suppliers, medicare, and policies, etc., are a typical complex distributed systems problem. This paper analyses the characteristics of health delivery systems, reviews common coordination mechanisms, and proposes the synergization of multi-agent systems to tackle the complexities of health delivery systems.

A shell is a body enclosed between two closely spaced and curved surfaces. Classical theory of shells [38], [33], [16] assumes a parametrization along the lines of principal curvature on the middle surface of a shell. Such a parametrization, while always existing locally, is not known for many surfaces, and deriving one can be very difficult if not impossible. This paper generalizes the classical strain-displacement equations and strain energy formula to a shell with an arbitrary parametric middle surface. We show that extensional and shearing strains can all be represented in terms of geometric invariants including principal curvatures, principal vectors, and the related directional and covariant derivatives. Computation of strains and strain energy is also described for a general parametrization. The displacement field on a shell is represented as a B-spline surface. By minimization of potential energy, we have simulated deformations of algebraic surfaces under applied loads, and performed experiments on an aluminum soda can and a stretched cloth using a three-fingered Barrett Hand. The measured deformations on each object match those in the simulation with good accuracy. The presented work is an initial step in our research on robot grasping of deformable objects.

This paper introduces a method that reconstructs a surface patch by sampling along three concurrent curves on the surface with a touch sensor. These data curves, each lying in a different plane, form a "skeleton" from which the patch is built in two phases. First, the Darboux frame at the curve intersection is estimated to reflect the local geometry. Second, polynomial fitting is carried out in the Darboux frame. The use of total (absolute) Gaussian curvature effectively prevents unnecessary folding of the surface normally expected to result from fitting over one-dimensional data. The reconstructed patch attains remarkable accuracy as demonstrated through experiments. This work carries a promise for in-hand manipulation. It also has potential application in building accurate models for complex curved objects which can cause occlusion to a camera or a range sensor