Automated pavement distress detection across large-scale roadway networks generates massive volumes of multimodal image data that must be efficiently indexed, searched, and analyzed within infrastructure monitoring and multimedia retrieval pipelines. Leveraging paired intensity–range imagery is attractive because intensity encodes appearance and texture, while range captures complementary three-dimensional geometry for robust 2D–3D distress understanding. However, modality utility is highly class-dependent, and naive early fusion can entangle modalities and suppress class-critical evidence, degrading detection and downstream retrieval performance. We study nine asphalt concrete pavement (ACP) distress types in the TxDOT 2D/3D pavement dataset, observe heterogeneous modality preferences across categories, and propose a decoupled multimodal YOLO variant with a full-backbone split design that keeps intensity and range feature streams separate in the backbone and fuses them only in the neck. Building on this principle, we further develop a class- and modality-aware online augmentation strategy: geometric transforms are synchronized across channels to preserve 2D/3D alignment, while pixel-level perturbations are applied per modality, with strengths selected based on each class’s modality dominance. On the ACP benchmark, the split backbone improves validation AP50 from 0.766 to 0.812, and the proposed augmentation further boosts it to 0.869. On the held-out test set, our system achieves 0.833 AP50 and 0.428 AP50–95, outperforming early fusion and uniform two-channel augmentation baselines under the same sensing configuration. Overall, our results indicate that late-fusion split learning with modality-aligned augmentation is a simple, data-centric design pattern for robust multimodal detection on large-scale intensity–range imagery.

With the rapid advancement of Large Language Models (LLMs), Approximate Nearest Neighbor Search (ANNS) in vector databases has become a cornerstone of modern AI infrastructure. ANNS excels at efficiently retrieving similar objects from billion-scale datasets, achieving remarkable success in delivering fast, high-recall searches. However, in-memory ANNS algorithms are prohibitively expensive when scaling to large datasets, creating a growing need for cost-effective memory-disk hybrid solutions. This paper introduces BAM-ANN, an efficient graph-based memory-disk hybrid solution for batch query processing. Specifically, we introduce three key innovations: (1) a multi-tiered indexing strategy with iterative balanced clustering to efficiently manage billion-scale data under limited memory resources, (2) a learning-driven pruning mechanism that eliminates redundant I/O and computations while preserving high accuracy, and (3) a batch-aware query organization scheme that further accelerates the search process. Experimental results demonstrate that BAM-ANN achieves 90% recall@100 in approximately one millisecond, outperforming state-of-the-art solutions like SPANN and DiskANN by 1.25x to 2.5x in speed.

State-of-the-art object detection methods applied to satellite and drone imagery largely fail to identify cross-domain small and dense objects. The high content variability in the overhead imagery is due to the different sensors, terrestrial regions, lighting conditions, and the image acquisition time of the day. Moreover, the number and size of objects in aerial imagery are very different than in the consumer data. We propose a small object detection pipeline that improves the feature extraction process by spatial pyramid pooling, cross-stage partial networks, and heatmap-based region proposal networks. Next, we propose the instance-aware image difficulty score that adapts the overall focal loss to improve object localization and identification. Finally, we add the two progressive domain adaptation blocks using contrastive learning in the pipeline. The blocks align the local and global features extracted from the customized CSP Darknet backbone, as the different levels of feature alignment alleviate the degradation of object identification in previously unseen datasets. We create a first-ever domain adaptation benchmark using contrastive learning for the object detection task in highly imbalanced satellite datasets with significant domain gaps and dominant small objects from existing satellite benchmarks—the proposed method results in up to a 7.4% and 4.6% increase in mAP over the best state-of-art method for the DOTA and NWPU-VHR10 datasets, respectively.

In this paper, we present a novel method for efficient and effective retrieval of similar deep descriptors. Our new hybrid method for indexing and searching for the approximate nearest neighbors in high-dimensional large deep-descriptor databases retrieves truly similar items in the database, even if the retrieval set is large. The proposed solution —- hybrid approximate nearest neighbor indexing and search (HANNIS) —- partitions the whole data space using the kmeans++ algorithm and then indexes each cluster using adapted hierarchical navigable graphs. This approach enables us to load items that are truly close to the incoming query at retrieval time. HANNIS outperforms all state-of-the-art methods in terms of recall at depths of up to 100 and offers consistent index loading and retrieval performance.

In this paper, we evaluate the performance of a novel method for efficient and effective retrieval of similar high-dimensional image features. The proposed solution —- hybrid approximate nearest neighbor indexing and search (HANNIS) —-retrieves truly similar items in the database, even if the retrieval set is large. This approach enables us to load items that are truly close to the incoming query at retrieval time. HANNIS outperforms all state-of-the-art methods in terms of recall, precision, and F1 score at depths of up to 100 and offers the fastest index loading and consistent retrieval performance.

We present NewsImages, a dataset of online news items, and the related NewsImages rematching task. The goal of NewsImages is to provide researchers with a means of studying the depiction gap, which we define to be the difference between what an image literally depicts and the way in which it is connected to the text that it accompanies. Online news is a domain in which the image-text connection is known to be indirect: The news article does not describe what is literally depicted in the image. We validate NewsImages with experiments that show the dataset's and the task's use for studying occurring connections between image and text, as well as addressing the depiction gap, which include sparse data, diversity of content, and importance of background knowledge.

In recent years, computational accessibility has enabled the use of Deep Neural Network (DNN) for computer vision applications on devices with limited computational resources. We focus on the real-time object detection algorithms deployed on UAV -friendly devices. The hardware deployed on UAV must be lightweight and thus limited in processing power, memory, and storage capacity. Lightweight modeling architecture does not suffice for high-recall reconnaissance applications. In this paper, we propose to reduce power consumption of YOLOv5 DNN architecture. We decided to use compressed convolutional technique, transfer learning, backbone shrinkage, and scale prediction to reduce the number of learnable parameters from the YOLOv5model. Our approach reduced the size of the model significantly and lowered the power consumption in turn. GPU memory and the Billion Floating-Point Operations Per Second (GFLOPS) for the YOLOv5model will keep the performance measure of the model as the baseline state-of-the-art. The best resulting model has a 63.86% mean average precision (mAP) and a GFLOPS of 97.7 on “DIOR”, an overhead imagery data set. The proposed approach has lowered GPU memory consumption of the model by 34% and lowered the energy consumption by 10 Watts compared to the baseline model.

On-board video sensors on large ships capture video data at high rates and provide real-time object tracking for maritime applications such as piracy and illegal fishing. High volumes of collected video and imagery data require advanced technology to analyze the video feeds, reduce data smog, and alert the crew on ships or coastal guard when unusual activities are detected. We present an integrated end-to-end system that analyzes multi-camera ship video feed; localizes maritime vessels in the video feed; identifies the maritime vessel over multiple cameras; maps the vessel track onto an overhead plane; and identifies anomalous vessel movement around the ship. In this paper, we focus on a specific activity detection approach in maritime vessel overhead tracks and on synthetic data generation to realistically model maritime boat movements around onboard ship cameras using real-world examples. We propose and compare three novel modes of trajectory analysis and activity classification, using Computing with Words (CWW), a Markov trajectory feature classifier (MTFC), and Naïve Bayes Radial Classifier (NBRC) to detect the activity of vessel approaching the ship, vessel chasing another vessel, and vessel circling around the ship.

For decades, the x86 architecture supported by Intel and AMD has been the dominate target for software development. Recently, ARM has solidified itself as a highly competitive and promising CPU architecture by exhibiting both high performance and low power consumption simultaneously. In the foreseeable future, a copious amount of software will be fully migrated to the ARM architecture or support both x86 and ARM simultaneously. Nevertheless, software ports from x86 to ARM are not trivial for a number of reasons. First, it is time consuming to write code that resolves all compatibility issues for a new architecture. Second, specific hardware (e.g. ARM chips) and supporting toolkits (e.g. libraries and compilers) may not be readily available for developers, which will delay the porting process. Third, it is hard to predict the performance of software before testing it on production chips. In this paper, we strive to tackle these challenges by proposing an instruction prediction method that can automatically generate AARCH64 code from existing x86-64 executables. Although the generated code might not be directly executable, it provides a cheap and efficient solution for developers to estimate certain runtime metrics before actually building, deploying and testing code on an ARM-based CPU. Our experimental results show that AARCH64 instructions derived using prediction can achieve a high Bilingual Evaluation Understudy (BLEU) Score. This indicates a quality match between generated executables and natively ported AARCH64 software.

Computing consensus states via global sign balancing is a key step in social network analysis. This paper presents graphB+, a fast algorithm for balancing signed graphs based on a new vertex and edge labeling technique, and a parallel implementation thereof for rapidly detecting and balancing all fundamental cycles. The main benefits of graphB+ are that the labels can be computed with linear time complexity, only require a linear amount of memory, and that the running time for balancing a cycle is linear in the length of the cycle times the vertex degrees but independent of the size of the graph. We parallelized graphB+ using OpenMP and CUDA. It takes 0.85 seconds on a Titan V GPU to balance the signs on the edges of an Amazon graph with 10 million vertices and 22 million edges, amounting to over 14 million fundamental cycles identified, traversed, and balanced per second.

Abstract Attitudinal network graphs are signed graphs where edges capture an expressed opinion; two vertices connected by an edge can be agreeable (positive) or antagonistic (negative). A signed graph is called balanced if each of its cycles includes an even number of negative edges. Balance is often characterized by the frustration index or by finding a single convergent balanced state of network consensus. In this paper, we propose to expand the measures of consensus from a single balanced state associated with the frustration index to the set of nearest balanced states. We introduce thefrustration cloudas a set of all nearest balanced states and use a graph-balancing algorithm to find all nearest balanced states in a deterministic way. Computational concerns are addressed by measuring consensus probabilistically, and we introduce new vertex and edge metrics to quantifystatus,agreement, andinfluence. We also introduce a new global measure of controversy for a given signed graph and show that vertex status is a zero-sum game in the signed network. We propose an efficient scalable algorithm for calculating frustration cloud-based measures in social network and survey data of up to 80,000 vertices and half-a-million edges. We also demonstrate the power of the proposed approach to provide discriminant features for community discovery when compared to spectral clustering and to automatically identify dominant vertices and anomalous decisions in the network.

We describe a system for content-based retrieval from large surveillance video archives, using behavior, action and appearance of objects. Objects are detected, tracked, and classified into broad categories. Their behavior and appearance are characterized by action detectors and descriptors, which are indexed in an archive. Queries can be posed as video exemplars, and the results can be refined through relevance feedback. The contributions of our system include the fusion of behavior and action detectors with appearance for matching; the improvement of query results through interactive query refinement (IQR), which learns a discriminative classifier online based on user feedback; and reasonable performance on low resolution, poor quality video. The system operates on video from ground cameras and aerial platforms, both RGB and IR. Performance is evaluated on publicly-available surveillance datasets, showing that subtle actions can be detected under difficult conditions, with reasonable improvement from IQR.

This paper presents probabilistic visual concept trees, a model for large visual semantic taxonomy structures and its use in visual concept detection. Organizing visual semantic knowledge systematically is one of the key challenges towards large-scale concept detection, and one that is complementary to optimizing visual classification for individual concepts. Semantic concepts have traditionally been treated as isolated nodes, a densely-connected web, or a tree. Our analysis shows that none of these models are sufficient in modeling the typical relationships on a real-world visual taxonomy, and these relationships belong to three broad categories -- semantic, appearance and statistics. We propose probabilistic visual concept trees for modeling a taxonomy forest with observation uncertainty. As a Bayesian network with parameter constraints, this model is flexible enough to account for the key assumptions in all three types of taxonomy relations, yet it is robust enough to accommodate expansion or deletion in a taxonomy. Our evaluation results on a large web image dataset show that the classification accuracy has considerably improved upon baselines without, or with only a subset of concept relationships

In this paper we examine a novel approach to the difficult problem of querying video databases using visual topics with few examples. Typically with visual topics, the examples are not sufficiently diverse to create a robust model of the user's need. As a result, direct modeling using the provided topic examples as training data is inadequate. Otherwise, systems resort to multiple content-based searches using each example in turn, which typically provides poor results. We propose a new technique of leveraging unlabeled data to expand the diversity of the topic examples as well as provide a robust set of negative examples that allow direct modeling. The approach intelligently models a pseudo-negative space using unbiased and biased methods for data sampling and data selection. We apply the proposed method in a fusion framework to improve discriminative support vector machine modeling, and improve the overall system performance. The result is an enhanced performance over any of the baseline models, as well as improved robustness with respect to training examples, visual features, and visual support of video topics in TRECVID. The proposed method outperforms a baseline retrieval approach by more than 18% on the TRECVID 2006 video collection and query topics.

We propose effective multimodal fusion strategies for video search. Multimodal search is a widely applicable information-retrieval problem, and fusion strategies are essential to the system in order to utilize all available retrieval experts and to boost the performance. Prior work has focused on hard- and soft- modeling of query classes and learning weights for each class, while the class partition is either manually defined or learned from data but still insensitive to the testing query. We propose a query-dependent fusion strategy that dynamically generates a class among the training queries that are closest to the testing query, based on light-weight query features defined on the outcome of semantic analysis on the query text. A set of optimal weights are then learned on the dynamic class, which aims to model both the co-occurring query features and unusual test queries. Used in conjunction with the rest of our multimodal retrieval system, dynamic query classes performs favorably with hard and soft query classes, and the system performance improves upon the best automatic search run of TRECVID05 and TRECVID06 by 34% and 8%, respectively.

Typical approaches to the multi-label classification problem require learning an independent classifier for every label from all the examples and features. This can become a computational bottleneck for sizeable datasets with a large label space. In this paper, we propose an efficient and effective multi-label learning algorithm called model-shared subspace boosting (MSSBoost) as an attempt to reduce the information redundancy in the learning process. This algorithm automatically finds, shares and combines a number of base models across multiple labels, where each model is learned from random feature subspace and boots trap data samples. The decision functions for each label are jointly estimated and thus a small number of shared subspace models can support the entire label space. Our experimental results on both synthetic data and real multimedia collections have demonstrated that the proposed algorithm can achieve better classification performance than the non-ensemble baselineclassifiers with a significant speedup in the learning and prediction processes. It can also use a smaller number of base models to achieve the same classification performance as its non-model-shared counterpart.

We study the problem of semantic concept-based query expansion and re-ranking for multimedia retrieval. In particular, we explore the utility of a fixed lexicon of visual semantic concepts for automatic multimedia retrieval and re-ranking purposes. In this paper, we propose several new approaches for query expansion, in which textual keywords, visual examples, or initial retrieval results are analyzed to identify the most relevant visual concepts for the given query. These concepts are then used to generate additional query results and/or to re-rank an existing set of results. We develop both lexical and statistical approaches for text query expansion, as well as content-based approaches for visual query expansion. In addition, we study several other recently proposed methods for concept-based query expansion. In total, we compare 7 different approaches for expanding queries with visual semantic concepts. They are evaluated using a large video corpus and 39 concept detectors from the TRECVID-2006 video retrieval benchmark. We observe consistent improvement over the baselines for all 7 approaches, leading to an overall performance gain of 77% relative to a text retrieval baseline, and a 31% improvement relative to a state-of-the-art multimodal retrieval baseline.

In this paper we present a novel approach for labeling clusters of multimedia content that leverages supervised classification techniques in conjunction with unsupervised clustering. Recent research has produced significant results for automatic tagging of video content such as broadcast news. For example, powerful techniques have been demonstrated in the context of the NIST TRECVID video retrieval benchmark [1]. However, the information needs of users typically span a range of semantic concepts. One of the challenges of these multimedia retrieval systems is to organize the video data in such a way that allows the user to most efficiently navigate the semantic space for the video data set. One important tool for video data organization is clustering. However, clustering results cannot be leveraged effectively when they are not labeled. We propose to build on clustering by aggregating the automatically tagged semantics. We propose and compare four techniques for labeling the clusters and evaluate the performance compared to human labeled ground-truth. We present examples of the cluster labeling results obtained on the BBC stock shots from the TRECVID-2005 video data set.

In this paper we unify two supposedly distinct tasks in multimedia retrieval. One task involves answering queries with a few examples. The other involves learning models for semantic concepts, also with a few examples. In our view these two tasks are identical with the only differentiation being the number of examples that are available for training. Once we adopt this unified view, we then apply identical techniques for solving both problems and evaluate the performance using the NIST TRECVID benchmark evaluation data [15]. We propose a combination hypothesis of two complementary classes of techniques, a nearest neighbor model using only positive examples and a discriminative support vector machine model using both positive and negative examples. In case of queries, where negative examples are rarely provided to seed the search, we create pseudo-negative samples. We then combine the ranked lists generated by evaluating the test database using both methods, to create a final ranked list of retrieved multimedia items. We evaluate this approach for rare concept and query topic modeling using the NIST TRECVID video corpus.In both tasks we find that applying the combination hypothesis across both modeling techniques and a variety of features results in enhanced performance over any of the baseline models, as well as in improved robustness with respect to training examples and visual features. In particular, we observe an improvement of 6% for rare concept detection and 17% for the search task.

We introduce the problem of repetitive nearest neighbor search in relevance feedback and propose an efficient search scheme for high dimensional feature spaces. Relevance feedback learning is a popular scheme used in content based image and video retrieval to support high-level concept queries. The paper addresses those scenarios in which a similarity or distance matrix is updated during each iteration of the relevance feedback search and a new set of nearest neighbors is computed. This repetitive nearest neighbor computation in high dimensional feature spaces is expensive, particularly when the number of items in the data set is large. In this context, we suggest a search algorithm that supports relevance feedback for the general quadratic distance metric. The scheme exploits correlations between two consecutive nearest neighbor sets thus significantly reducing the overall search complexity. Detailed experimental results are provided using 60 dimensional texture feature dataset.

Texture has been recognized as an important visual primitive in image analysis. A widely used texture descriptor, which is part, of the MPEG-7 standard, is that computed using multiscale Gabor filters. The high dimensionality and computational complexity of this descriptor adversely affect the efficiency of content-based retrieval systems. We propose a modified texture descriptor that has comparable performance, but with nearly half the dimensionality and less computational expense. This gain is based on a claim that the distribution of (absolute values of) filter outputs have a strong tendency to be Rayleigh. Experimental results show that the dimensionality can be reduced by almost 50%, with a tradeoff of less than 3% on the error rate. Furthermore, it is easy to compute the new feature using the old, one, without having to repeat the computationally expensive filtering step. We also propose a new normalization method that improves similarity retrieval and indexing efficiency.

This work introduces a conceptual representation for complex spatial arrangements of image features in large multimedia datasets. A novel data structure, termed the spatial event cube (SEC), is formed from the co-occurrence matrices of perceptually classified features with respect to specific spatial relationships. A visual thesaurus constructed using supervised and unsupervised learning techniques is used to label the image features. SECs can be used to not only visualize the dominant spatial arrangements of feature classes but also discover non-obvious configurations. SECs also provide the framework for high-level data mining techniques such as using the generalized association rule approach. Experimental results are provided for a large dataset of aerial images.