User interface understanding with vision-language models (VLMs) has received much attention due to its potential for enhancing software automation.However, existing datasets used to build UI-VLMs either only contain large-scale context-free element annotations or contextualized functional descriptions for elements at a small scale.In this work, we propose the AutoGUI pipeline for automatically annotating UI elements with detailed functionality descriptions at scale.Specifically, we leverage large language models (LLMs) to infer element functionality by comparing UI state changes before and after simulated interactions. To improve annotation quality, we propose LLM-aided rejection and verification, eliminating invalid annotations without human labor.We construct a high-quality AutoGUI-704k dataset using the proposed pipeline, featuring diverse and detailed functionality annotations that are hardly provided by previous datasets.Human evaluation shows that we achieve annotation correctness comparable to a trained human annotator. Extensive experiments show that our dataset remarkably enhances VLM’s UI grounding capabilities and exhibits significant scaling effects. We also show the interesting potential use of our dataset in UI agent tasks. Please view our project at https://autogui-project.github.io/.

Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in multimodal understanding, but they frequently suffer from hallucination -generating content inconsistent with visual inputs.In this work, we explore a novel perspective on hallucination mitigation by examining the intermediate activations of LVLMs during generation.Our investigation reveals that hallucinated content manifests as distinct, identifiable patterns in the model's hidden state space.Motivated by this finding, we propose Activation Steering Decoding (ASD), a training-free approach that mitigates hallucination through targeted intervention in the model's intermediate activations.ASD operates by first identifying directional patterns of hallucination in the activation space using a small calibration set, then employing a contrast decoding mechanism that computes the difference between positive and negative steering predictions.This approach effectively suppresses hallucination patterns while preserving the model's general capabilities.Extensive experiments demonstrate that our method significantly reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks.Notably, our approach requires no model re-training or architectural modifications, making it readily applicable to existing deployed models.

Building autonomous agents that perceive and operate graphical user interfaces (GUIs) like humans has long been a vision in the field of artificial intelligence. Central to these agents is the capability for GUI interaction, which involves GUI understanding and planning capabilities. Existing methods have tried developing GUI agents based on the multi-modal comprehension ability of vision-language models (VLMs). However, the limited scenario, insufficient size, and heterogeneous action spaces hinder the progress of building generalist GUI agents. To resolve these issues, this paper proposes \textbf{UIPro}, a novel generalist GUI agent trained with extensive multi-platform and multi-task GUI interaction data, coupled with a unified action space. We first curate a comprehensive dataset encompassing 20.6 million GUI understanding tasks to pre-train UIPro, granting it a strong GUI grounding capability, which is key to downstream GUI agent tasks. Subsequently, we establish a unified action space to harmonize heterogeneous GUI agent task datasets and produce a merged dataset to foster the action prediction ability of UIPro via continued fine-tuning. Experimental results demonstrate UIPro's superior performance across multiple GUI task benchmarks on various platforms, highlighting the effectiveness of our approach.

Computer end users have spent billions of hours completing daily tasks like tabular data processing and project timeline scheduling. Most of these tasks are repetitive and error-prone, yet most end users lack the skill to automate these burdensome works. With the advent of large language models (LLMs), directing software with natural language user requests become a reachable goal. In this work, we propose a SheetCopilot agent that takes natural language task and control spreadsheet to fulfill the requirements. We propose a set of atomic actions as an abstraction of spreadsheet software functionalities. We further design a state machine-based task planning framework for LLMs to robustly interact with spreadsheets. We curate a representative dataset containing 221 spreadsheet control tasks and establish a fully automated evaluation pipeline for rigorously benchmarking the ability of LLMs in software control tasks. Our SheetCopilot correctly completes 44.3\% of tasks for a single generation, outperforming the strong code generation baseline by a wide margin. Our project page: https://sheetcopilot.github.io/.

Object detection is a fundamental task in remote sensing image analysis and scene understanding. Previous remote sensing object detectors are typically based on convolutional neural networks (CNNs), whose performance is significantly limited by the intrinsic locality of convolution operations. The emergence of vision Transformers brings potential solutions to this problem, which have the capability to be a solid alternative to CNNs. However, three crucial obstacles hinder the application and performance of Transformers in the task of remote sensing object detection, i.e., 1) high computational complexity, especially for high-resolution remote sensing images, 2) training-and sample-inefficiency caused by lack of inductive bias, and 3) difficulty in learning arbitrary orientation knowledge of geospatial objects. To address these issues, in this paper, a novel efficient inductive vision Transformer framework is proposed for oriented object detection in remote sensing imagery. This framework follows the hierarchical feature pyramid structure and makes threefold contributions, as follows. 1) Spatial redundancy in remote sensing images is fully explored and an adaptive multi-grained routing mechanism is proposed to facilitate token sparsity in Transformers, which can dramatically reduce the computational cost without comprising the accuracy. 2) A compact dual-path encoding architecture, where both global long-range dependencies and local semantic relations are jointly and complementarily captured, is proposed to enhance inductive bias in Transformers. 3) An angle tokenization technique is proposed to promote the encoding, embedding, and learning of direction knowledge for oriented objects in remote sensing scenarios. In this work, the above three contributions are instantiated in an advanced Transformer-based object detector, namely EIA-PVT. Comprehensive experiments on two publicly available datasets have demonstrated its effectiveness and superiority for oriented object detection in remote sensing images.

Millimeter-wave (mmWave) radar has been widely used in autonomous driving due to its good performance under harsh weather conditions. In recent years, with the development of mmWave radar hardware performance, radar point clouds, as an important data format of mmWave radar, have been widely used in high-level perception tasks of mobile robots and autonomous driving. However, at present, compared to LiDAR point clouds, in common application scenes of mobile robots, mmWave radar point clouds have shortcomings such as sparsity and containing many “ghost” targets. Therefore, in this article, we analyze the reasons that cause these problems and propose a new method for point cloud generation as well as a new evaluation metric. After building a new dataset and carrying out experiments in real-world scenes, our method shows better performance on the quality of radar point clouds compared to other methods. In addition, by evaluating the performance of applying the high-quality radar point clouds to object detection tasks as well as localization and mapping tasks, the result shows that radar point clouds generated using our method can significantly improve the environment perception ability of mobile robots.

The millimeter-wave radar, as an important sensor, is widely used in autonomous driving. In recent years, to meet the requirement of high level autonomous driving applications, attentions have been paid to generate high-quality radar point clouds. However, in the complex roadway environment, the weaknesses of classical radar detectors are exposed, such as too much clutter points and sparse valid point clouds. Therefore, in this paper, we propose a new automotive radar detector based on deep learning using the spatial distribution feature of the real targets, in order to improve the performance of automotive radar detector in the real-world driving scene. Besides, aiming at the lack of radar data labels, we propose an autonomous labeling method by using synchronized Lidar data. Finally, we evaluate the detector on data collected in the real-world roadway scene and the result shows that the proposed radar detector out-performs the classical radar detectors in suppressing the clutter and generating denser point clouds.

As a new biometric feature, gait brings new possibilities for personal recognition. At present, gait recognition methods mainly extract appearance features, but seldom use temporal information, such as the method based on gait energy images (GEI) to fuse gait sequence into one image. In order to mine the motion patterns contained in gait sequences, this paper proposes a model that can gradually fuse temporal features while extracting spatial features to achieve the spatiotemporal feature extraction: 1) the model mines temporal information by passing partial channels of feature maps and fusing features of adjacent frames; 2) the model adapts the part-based method to split feature map into several parts, which can refine the spatial features. Extensive experiments on the challenging datasets CASIA-B demonstrate the superiority and effectiveness of our proposed model.

Gait information has gradually attracted people's attention duing to its uniqueness. Methods based on deep metric learning are successfully utlized in gait recognition tasks. However, most of the previous studies use losses which only consider a small number of samples in the mini-batch, such as Triplet loss and Quadruplet Loss, which is not conducive to the convergence of the model. Therefore, in this paper, a novel loss named Center-ranked is proposed to integrate all positive and negative samples information. We also propose a simple model for gait recognition tasks to verify the validity of the loss. Extensive experiments on two challenging datasets CASIA-B and OU-MVLP demonstrate the superiority and effectiveness of our proposed Center-ranked loss and model.

In this paper, a novel introspective learning strategy for remote sensing scene classification is proposed. Through this strategy, the neural network used for classification can introspectively generate negative samples. In most training deep neural networks, negative samples are rarely noticed. We are the first to actively introduce negative samples into the remote sensing scene classification tasks. The goal of this paper is to analyze the effect of introspective negative samples on remote sensing scene classification tasks. Experiments demonstrate that the introduction of negative samples in training can effectively improve the classification accuracy and robustness. In addition, we found that our method can effectively against invalid remote sensing images.