Visual imagery does not consist of solitary objects, but instead reflects the composition of a multitude of fluid concepts. While there have been great advances in visual representation learning, such advances have focused on building better representations for a small number of discrete objects bereft of an understanding of how these objects are interacting. One can observe this limitation in representations learned through captions or contrastive learning – where the learned model treats an image essentially as a bag of words. Several works have attempted to address this limitation through the development of bespoke architectures. In this work, we focus on simple and scalable approaches. In particular, we demonstrate that by improving weakly labeled data, i.e. captions, we can vastly improve the performance of standard contrastive learning approaches. Previous CLIP models achieved near chance rate on challenging tasks probing compositional learning. However, our simple approach boosts performance of CLIP substantially and achieves state of the art results on compositional benchmarks such as ARO and SugarCrepe. Furthermore, we showcase our results on a relatively new captioning benchmark derived from DOCCI. We demonstrate through a series of ablations that a standard CLIP model trained with enhanced data may demonstrate impressive performance on image retrieval tasks.

While image-text representation learning has become very popular in recent years, existing models tend to lack spatial awareness and have limited direct applicability for dense understanding tasks. For this reason, self-supervised image-only pretraining is still the go-to method for many dense vision applications (e.g. depth estimation, semantic segmentation), despite the lack of explicit supervisory signals. In this paper, we close this gap between image-text and self-supervised learning, by proposing a novel general-purpose image-text model, which can be effectively used off the shelf for dense and global vision tasks. Our method, which we refer to as Text-Image Pretraining with Spatial awareness (TIPS), leverages two simple and effective insights. First, on textual supervision: we reveal that replacing noisy web image captions by synthetically generated textual descriptions boosts dense understanding performance significantly, due to a much richer signal for learning spatially aware representations. We propose an adapted training method that combines noisy and synthetic captions, resulting in improvements across both dense and global understanding tasks. Second, on the learning technique: we propose to combine contrastive image-text learning with self-supervised masked image modeling, to encourage spatial coherence, unlocking substantial enhancements for downstream applications. Building on these two ideas, we scale our model using the transformer architecture, trained on a curated set of public images. Our experiments are conducted on $8$ tasks involving $16$ datasets in total, demonstrating strong off-the-shelf performance on both dense and global understanding, for several image-only and image-text tasks. Code and models are released at https://github.com/google-deepmind/tips .

The image matching field has been witnessing a contin-uous emergence of novel learnable feature matching techniques, with ever-improving performance on conventional benchmarks. However, our investigation shows that de-spite these gains, their potential for real-world applications is restricted by their limited generalization capabili-ties to novel image domains. In this paper, we introduce OmniGlue, the first learnable image matcher that is de-signed with generalization as a core principle. OmniGlue leverages broad knowledge from a vision foundation model to guide the feature matching process, boosting general-ization to domains not seen at training time. Addition-ally, we propose a novel keypoint position-guided attention mechanism which disentangles spatial and appear-ance information, leading to enhanced matching descrip-tors. We perform comprehensive experiments on a suite of 7 datasets with varied image domains, including scene-level, object-centric and aerial images. OmniGlue's novel components lead to relative gains on unseen domains of 20.9% with respect to a directly comparable reference model, while also outperforming the recent LightGlue method by 9.5% relatively. Code and model can be found at https://hwjiang151o.github.io/OmniGlue.

In existing window time-frequency analysis methods, its resolution is commonly limited by the uncertainty principle. The time resolution and the frequency resolution restrict each other and cannot reach the maximum at the same time. For this reason, we employ the secondary processing technology of the time-frequency spectrum to enhance the time resolution of the S-transform (ST). By performing multiband filtering with a filter bank, the ST is capable of receiving the signal under different frequencies and its time-frequency coefficient. The window size is different, that is, the low-frequency time window is long, resulting in low time resolution and high-frequency resolution. The high-frequency time window is short, which makes the time resolution high and the frequency resolution low; however, the window size for the time component of each frequency is fixed. Based on the above idea, we perform “de-window” processing for each frequency component signal. We assume that the time-frequency spectrum is sparse without “windowing,” so the window effect can be removed by sparse inversion. Based on the alternating direction method of multipliers (ADMM), we use the nonnegative penalty terms and sparse terms for the joint constraints to solve the optimization problem, and obtain the sparse time-frequency spectrum to enhance the time resolution. This time sparse ST (TSST) obtained by the proposed method is suitable for improving the ability of thin-layer identification and also for the analysis of transient signals.

This article proposes a pre-stack deconvolution algorithm for the seismic common midpoint (CMP) gathers. Due to the low signal-to-noise ratio (SNR), poor lateral continuity of seismic CMP gathers, and residual time differences, conventional deconvolution algorithms struggle to enhance the resolution while maintaining the SNR. As a result, the data after deconvolution are overwhelmed by noise. In addition, the deconvolution methods in the Radon transform domain are limited by the tailing of focal points in the Radon domain. Therefore, this research employs the Radon transform as a sparse-promoting transform for deconvolution. By applying thresholds in the Radon domain, this algorithm suppresses noise and reduces the instability of deconvolution. Depending on the noise distribution, either the$L_{2}$norm or the$L_{1}$norm is flexibly chosen as the fitting term to enhance the algorithm’s versatility. Leveraging the strong denoising capability of the Radon transform, this algorithm improves resolution on gathers with a low SNR while enhancing lateral continuity. Model and actual data tests indicate that the algorithm effectively enhances the resolution of gathers, thus facilitating pre-stack amplitude versus offset (AVO) analysis and pre-stack inversion.

Image retrieval systems conventionally use a two-stage paradigm, leveraging global features for initial retrieval and local features for reranking. However, the scalability of this method is often limited due to the significant storage and computation cost incurred by local feature matching in the reranking stage. In this paper, we present SuperGlobal, a novel approach that exclusively employs global features for both stages, improving efficiency without sacrificing accuracy. SuperGlobal introduces key enhancements to the retrieval system, specifically focusing on the global feature extraction and reranking processes. For extraction, we identify sub-optimal performance when the widely-used ArcFace loss and Generalized Mean (GeM) pooling methods are combined and propose several new modules to improve GeM pooling. In the reranking stage, we introduce a novel method to update the global features of the query and top-ranked images by only considering feature refinement with a small set of images, thus being very compute and memory efficient. Our experiments demonstrate substantial improvements compared to the state of the art in standard benchmarks. Notably, on the Revisited Oxford+1M Hard dataset, our single-stage results improve by 7.1%, while our two-stage gain reaches 3.7% with a strong 64,865× speedup. Our two-stage system surpasses the current single-stage state-of-the-art by 16.3%, offering a scalable, accurate alternative for high-performing image retrieval systems with minimal time overhead.Code: https://github.com/ShihaoShao-GH/SuperGlobal.

Fine-grained and instance-level recognition methods are commonly trained and evaluated on specific domains, in a model per domain scenario. Such an approach, however, is impractical in real large-scale applications. In this work, we address the problem of universal image embedding, where a single universal model is trained and used in multiple domains. First, we leverage existing domain-specific datasets to carefully construct a new large-scale public benchmark for the evaluation of universal image embeddings, with 241k query images, 1.4M index images and 2.8M training images across 8 different domains and 349k classes. We define suitable metrics, training and evaluation protocols to foster future research in this area. Second, we provide a comprehensive experimental evaluation on the new dataset, demonstrating that existing approaches and simplistic extensions lead to worse performance than an assembly of models trained for each domain separately. Finally, we conducted a public research competition on this topic, leveraging industrial datasets, which attracted the participation of more than 1k teams world-wide. This exercise generated many interesting research ideas and findings which we present in detail. Project webpage: https://cmp.felk.cvut.cz/univ_emb/

Cooperative Simultaneous Localization and Mapping (C-SLAM) is an active research topic in mobile robotics. However, its application in the field of autonomous driving is rare. While the advent of Vehicle-to-Everything (V2X) communication has empowered Connected Autonomous Vehicles (CAV) to exchange data with each other, recent research on CAV cooperation tasks has primarily focused on cooperative perception and global positioning improvement. Techniques for organizing multiple CAV to work together to achieve localization and mapping in unknown environments have not been actively explored. We propose a C-SLAM system for CAVs that employs sparse LiDAR feature representations to enable vehicles to exchange data using standard V2X messages. The system was tested in real environments using two connected vehicles. The results show that the proposed V2X-based C-SLAM system can operate in both centralized and decentralized manners and output accurate pose estimates and global maps, showing promising application possibilities.

Simultaneous localization and mapping (SLAM) is an essential technique for autonomous driving. Recently, combining image recognition technology to generate semantically meaningful maps has become a new trend in visual SLAM research. However, in the field of LiDAR SLAM, this potential has not been fully explored. We propose a novel object-level SLAM system using 3D LiDARs for autonomous vehicles. We detect and track poles, walls, and parked cars, which are common along urban roads. This paper presents how we process the measurement data of three different shapes of objects to build a graph-based optimization system and facilitate the geometric distribution of poles to detect loops. Experiments were carried out on datasets collected with a test vehicle in city traffic. The results show that our object-level SLAM system can build precise and semantically meaningful maps and produce more accurate pose estimations compared to the state-of-the-art systems on our datasets.

While image retrieval and instance recognition techniques are progressing rapidly, there is a need for challenging datasets to accurately measure their performance -- while posing novel challenges that are relevant for practical applications. We introduce the Google Landmarks Dataset v2 (GLDv2), a new benchmark for large-scale, fine-grained instance recognition and image retrieval in the domain of human-made and natural landmarks. GLDv2 is the largest such dataset to date by a large margin, including over 5M images and 200k distinct instance labels. Its test set consists of 118k images with ground truth annotations for both the retrieval and recognition tasks. The ground truth construction involved over 800 hours of human annotator work. Our new dataset has several challenging properties inspired by real-world applications that previous datasets did not consider: An extremely long-tailed class distribution, a large fraction of out-of-domain test photos and large intra-class variability. The dataset is sourced from Wikimedia Commons, the world's largest crowdsourced collection of landmark photos. We provide baseline results for both recognition and retrieval tasks based on state-of-the-art methods as well as competitive results from a public challenge. We further demonstrate the suitability of the dataset for transfer learning by showing that image embeddings trained on it achieve competitive retrieval performance on independent datasets. The dataset images, ground-truth and metric scoring code are available at https://github.com/cvdfoundation/google-landmark.

Previous database accelerator proposals such as the Q100 provide a fixed set of database operators, chosen to support a target query workload. Some queries may not be well-supported by a fixed accelerator, typically because they need more resources/operators of a particular kind than the accelerator provides. By Amdahl's law, these queries become relatively more expensive as they are not fully accelerated. We propose a second-level accelerator, DB-Mesh, to take up some of this workload. DB-Mesh is an asynchronous systolic array that is more generic than the Q100, and can be configured to run a variety of operators with configurable parameters such as record widths. We demonstrate DB-Mesh applied to nested loops joins, an operator that is not directly supported on the Q100. We show that a naïve implementation has the potential for deadlock, and show how to avoid deadlock with a careful design. We also demonstrate how the data flow policy used in the array influences system throughput.

We have built a display wall system with four liquid crystal displays to provide a very large visual area for university lectures. A program was developed to transform the display wall into an electronic blackboard. It has been used in a long-term classroom practice. Based on the research results, we demonstrate the benefits of the digital multi-display system for classroom learning.

To simplify the implementation of dataflow systems in hardware, we present a technique for designing latency- insensitive dataflow blocks. We provide buffering with backpressure, resulting in blocks that compose into deep, high-speed pipelines without introducing long combinational paths. Our input and output buffers are easy to assemble into simple unit- rate dataflow blocks, arbiters, and blocks for Kahn networks. We prove the correctness of our buffers, illustrate how they can be used to assemble arbitrary dataflow blocks, discuss pitfalls, and present experimental results that suggest our pipelines can operate at a high clock rate independent of length.

Massively parallel computing on throughput computers such as GPUs requires myriad memory accesses to register files, on-chip scratchpad, caches, and off-chip DRAM. Unlike CPUs, these processors have a large register file and on-chip scratchpad memory, which consume a significant portion of compute core power (35%-45%). In this paper, we introduce novel throughput architecture by integrating resistive memory (Spin Transfer Torque RAM) inside the compute core, which reduces leakage significantly, but introduces write power overhead and longer write latencies in GPU shared memory and register file accesses. We enhance the compute core by introducing register file organization with differential memory update mechanism to remove update redundancy during write operations. Furthermore, using merged register-write-mechanism and write-back buffer, we coalesce multithreaded GPU register write accesses to save write energy. In addition, we introduce hybrid shared memory design using SRAM and STT-MRAM that provides significant leakage/dynamic power savings without affecting performance. On average, across 23 GPGPU/graphics workloads, our schemes save 46% dynamic power due to register access (83% leakage power saving) with negligible performance degradation. On average, hybrid shared memory provides 10% reduction in dynamic power with maximum 1.6× performance improvement for the current workloads at no additional area overhead.