Science undergraduate students benefit from gaining computing skills due to the growing overlap between scientific discovery and computational methods. One approach is to encourage students majoring in science to pursue a computing minor; however, only a few seem to undertake this endeavor. Financially disadvantaged students face additional barriers that may further deter them from this pursuit. Unfortunately, there is limited research on the benefits, limitations, and barriers related to earning a computing minor. This study aims to explore the perspectives of undergraduate college students from the United States in a scholarship program designed to aid low-income science majors in completing an information science and technology (IS&T) minor. Using an exploratory qualitative research approach, semi-structured interviews were conducted with seven students in a scholarship program. Based on the findings, three main recommendations for program stakeholders are proposed. First, explicitly explain the unique advantages of gaining computing knowledge for science majors. Second, ensure students have a foundational understanding of computing and related study skills. Third, guide students in strategically planning their course sequence to optimize their time and workload.

Web search engines have long served as indispensable tools for information retrieval; user behavior and query formulation strategies have been well studied. The introduction of search engines powered by large language models (LLMs) suggested more conversational search and new types of query strategies. In this paper, we compare traditional and LLM-based search for the task of image geolocation, i.e., determining the location where an image was captured. Our work examines user interactions, with a particular focus on query formulation strategies. In our study, 60 participants were assigned either traditional or LLM-based search engines as assistants for geolocation. Participants using traditional search more accurately predicted the location of the image compared to those using the LLM-based search. Distinct strategies emerged between users depending on the type of assistant. Participants using the LLM-based search issued longer, more natural language queries, but had shorter search sessions. When reformulating their search queries, traditional search participants tended to add more terms to their initial queries, whereas participants using the LLM-based search consistently rephrased their initial queries.

AUC maximization is an effective approach to address the imbalanced data classification problem in federated learning. In the past few years, a couple of federated AUC maximization approaches have been developed based on the minimax optimization. However, directly solving a minimax optimization problem to maximize the AUC score cannot achieve satisfactory performance. To address this issue, we propose to maximize AUC via optimizing a federated multi-level compositional minimax problem. Specifically, we develop a novel federated multi-level compositional minimax algorithm with rigorous theoretical guarantees to solve this new learning paradigm in both algorithmic design and theoretical analysis. To the best of our knowledge, this is the first work studying the multi-level minimax optimization problem. Additionally, extensive empirical evaluations confirm the efficacy of our proposed approach.

Annotated images are required for numerous computer vision tasks; however, the annotation process can be time-consuming for crowdworkers and experts. Previous work has investigated novel interaction techniques and task reformulation to speed up this process; however, there remains a gap in optimizing more complex annotation tasks, such as object matching. In this paper, we explore the impact of varying the amount of context provided to annotators. We hypothesize that reducing the context around the object being matched will improve speed without sacrificing the accuracy of the annotation task. To test this hypothesis, we developed a semi-automated annotation pipeline that pre-processes images to adjust the amount of context shown around an object of interest. We conducted two studies (n = 130, n = 10) to assess the effects of context quantitatively and qualitatively. We found that while the accuracy remained the same, the time spent on the task was significantly reduced when there was less context surrounding the object. However, our qualitative findings revealed multiple scenarios in which context served as a means of guiding the object matching task, and many others in which the distinctiveness of the object guided the matching task and additional context was not needed.

Multi-view classification problems are common in medical image analysis, forensics, and other domains where problem queries involve multi-image input. Existing multi-view classification methods are often tailored to a specific task. In this paper, we repurpose off-the-shelf Hybrid CNN-Transformer networks for multi-view classification with either structured or unstructured views. Our approach incorporates a novel fusion scheme, mutual distillation, and minimal additional parameters. We demonstrate the effectiveness and generalization capability of our approach, MV-HFMD, on multiple multi-view classification tasks and show that it outperforms other multi-view approaches, even task-specific methods. Code is available at https://github.com/vidarlab/multi-view-hybrid.

Federated learning has attracted increasing attention due to the promise of balancing privacy and large-scale learning; numerous approaches have been proposed. However, most existing approaches focus on problems with balanced data, and prediction performance is far from satisfactory for many real-world applications where the number of samples in different classes is highly imbalanced. To address this challenging problem, we developed a novel federated learning method for imbalanced data by directly optimizing the area under curve (AUC) score. In particular, we formulate the AUC maximization problem as a federated compositional minimax optimization problem, develop a local stochastic compositional gradient descent ascent with momentum algorithm, and provide bounds on the computational and communication complexities of our algorithm. To the best of our knowledge, this is the first work to achieve such favorable theoretical results. Finally, extensive experimental results confirm the efficacy of our method.

Transformer architectures have shown promise for a wide range of computer vision tasks, including image embedding. As was the case with convolutional neural networks and other models, explainability of the predictions is a key concern, but visualization approaches tend to be architecture-specific. In this paper, we introduce a new method for producing interpretable visualizations that, given a pair of images encoded with a Transformer, show which regions contributed to their similarity. Additionally, for the task of image retrieval, we compare the performance of Transformer and ResNet models of similar capacity and show that while they have similar performance in aggregate, the retrieved results and the visual explanations for those results are quite different. Code is available at https://github.com/vidarlab/xformer-paired-viz.

Amassing large-scale datasets used to train machine learning algorithms often includes crowd-sourcing or web scraping. The data resulting from these approaches can carry undesired societal biases that are reflected in the predictions of the learning system. Recently, researchers have proposed mitigation strategies targeting either the learning algorithms or the data used for training. In this paper, we evaluate a simple data augmentation strategy for the task of automated image captioning, namely substituting gendered terms for gender-neutral equivalents. We evaluated this approach on multiple recent image captioning models using both objective and subjective analysis. We found that human raters did not find a difference in the quality of the image captions and, in some cases, the model was able to generate more accurate captions with additional detail when trained with gender-neutral data.

A common approach to censoring digital image content is masking the region(s) of interest with a solid color or pattern. In the case where the masked image will be used as input for classification or matching, the mask itself may impact the results. Recent work in image inpainting provides an alternative to masking by replacing the foreground with predicted background. In this paper, we perform an extensive evaluation of inpainting approaches to understand how well inpainted images can serve as proxies for the original in classification and retrieval. Results indicate that the metrics typically used to evaluate inpainting performance (e.g., reconstruction accuracy) do not necessarily correspond to improved classification or retrieval, especially in the case of person-shaped masked regions.

In this paper, we propose a method for estimating 3D human pose from a single RGB image. Compared to methods that either provide point estimates for coordinate regression or unimodal predictions of joint locations, our approach predicts joint locations using multimodal distributions. In addition, we apply a data-driven approach to learn the conditional dependencies of the relative positions of joints. Our end-to-end approach takes as input images with either 2D or 3D labels and performs on par or better than the state-of-the-art on the Human3.6M and MPII datasets.

Recognizing a hotel from an image of a hotel room is important for human trafficking investigations. Images directly link victims to places and can help verify where victims have been trafficked, and where their traffickers might move them or others in the future. Recognizing the hotel from images is challenging because of low image quality, uncommon camera perspectives, large occlusions (often the victim), and the similarity of objects (e.g., furniture, art, bedding) across different hotel rooms. To support efforts towards this hotel recognition task, we have curated a dataset of over 1 million annotated hotel room images from 50,000 hotels. These images include professionally captured photographs from travel websites and crowd-sourced images from a mobile application, which are more similar to the types of images analyzed in real-world investigations. We present a baseline approach based on a standard network architecture and a collection of data-augmentation approaches tuned to this problem domain.

For convolutional neural network models that optimize an image embedding, we propose a method to highlight the regions of images that contribute most to pairwise similarity. This work is a corollary to the visualization tools developed for classification networks, but applicable to the problem domains better suited to similarity learning. The visualization shows how similarity networks that are fine-tuned learn to focus on different features. We also generalize our approach to embedding networks that use different pooling strategies and provide a simple mechanism to support image similarity searches on objects or sub-regions in the query image.

While natural beauty is often considered a subjective property of images, in this paper, we take an objective approach and provide methods for quantifying and predicting the scenicness of an image. Using a dataset containing hundreds of thousands of outdoor images captured throughout Great Britain with crowdsourced ratings of natural beauty, we propose an approach to predict scenicness which explicitly accounts for the variance of human ratings. We demonstrate that quantitative measures of scenicness can benefit semantic image understanding, content-aware image processing, and a novel application of cross-view mapping, where the sparsity of ground-level images can be addressed by incorporating unlabeled overhead images in the training and prediction steps. For each application, our methods for scenicness prediction result in quantitative and qualitative improvements over baseline approaches.

We address the problem of automated discretization for continuous labels in the context of head pose estimation from overhead cameras. Due to the lack of visual detail, precise head pose estimates are not always possible. A common approach is to discretize the space of head pose angles, turning a real-valued prediction task into a coarser (ordered) classification variant. Often, however, the ranges are arbitrarily defined (e.g., dividing up parameter space evenly). Our work incorporates label discretization into the feature learning process and improves the accuracy of coarse head pan angle prediction from overhead cameras on a benchmark dataset.

We present a data-driven method for semi-supervised multioutput regression on image manifolds, which simultaneously considers the manifold structure of the input data and complex output labels. Compared to related methods, our method achieves superior prediction accuracy on a variety of data sets, with as few as 5% of the input examples labeled. Also, with a few labeled examples and no domain-specific tuning, our method performs on par with specialized algorithms for tasks such as face landmark detection.

In this paper, we present a computationally efficient and non-parametric method for robust regression on manifolds. We apply our algorithm to the problem of correcting mislabeled examples from image collections with ordered (e.g., real-valued, ordinal) labels. Compared to related methods for robust regression, our method achieves superior denoising accuracy on a variety of data sets, with label corruption levels as high as 80%. For a diverse set of widely-used, large-scale, publicly-available data sets, our approach results in image labels that more accurately describe the associated images.

We propose to use deep convolutional neural networks to address the problem of cross-view image geolocalization, in which the geolocation of a ground-level query image is estimated by matching to georeferenced aerial images. We use state-of-the-art feature representations for ground-level images and introduce a cross-view training approach for learning a joint semantic feature representation for aerial images. We also propose a network architecture that fuses features extracted from aerial images at multiple spatial scales. To support training these networks, we introduce a massive database that contains pairs of aerial and ground-level images from across the United States. Our methods significantly out-perform the state of the art on two benchmark datasets. We also show, qualitatively, that the proposed feature representations are discriminative at both local and continental spatial scales.

Gamesourcing has emerged as an approach for rapidly acquiring labeled data for learning-based, computer vision recognition algorithms. In this article, we present an approach for using RGB-D sensors to acquire annotated training data for human pose estimation from 2D images. Unlike other gamesourcing approaches, our method does not require a specific game, but runs alongside any gesture-based game using RGB-D sensors. The automatically generated datasets resulting from this approach contain joint estimates within a few pixel units of manually labeled data, and a gamesourced dataset created using a relatively small number of players, games, and locations performs as well as large-scale, manually annotated datasets when used as training data with recent learning-based human pose estimation methods for 2D images.

The expected appearance of a human face depends strongly on age, ethnicity and gender. While these relationships are well-studied, our work explores the little-studied dependence of facial appearance on geographic location. To support this effort, we constructed GeoFaces, a large dataset of geotagged face images. We examine the geo-dependence of Eigenfaces and use two supervised methods for extracting geo-informative features. The first, canonical correlation analysis, is used to find location-dependent component images as well as the spatial direction of most significant face appearance change. The second, linear discriminant analysis, is used to find countries with relatively homogeneous, yet distinctive, facial appearance.

Cloud shadows dramatically affect the appearance of outdoor scenes. We describe two approaches that use video of cloud shadows to estimate a cloudmap, a spatio-temporal function that represents the clouds passing over the scene. Our first method makes strong assumptions about the camera geometry and estimates the cloud motion direction. Our second method uses techniques from manifold learning and does not require known geometry. Neither method requires directly viewing the clouds, but instead uses the pattern of intensity changes caused by the cloud shadows. We show renderings of cloudmaps extracted using both methods from videos of real outdoor scenes as well as quantitative results on synthetic datasets. An accurate estimate of the cloudmap has potential applications in surveillance and graphics, as well as scientific studies that depend on solar radiation.

For human action recognition methods, there is often a trade-off between classification accuracy and computational efficiency. Methods that include 3D information from multiple cameras are often computationally expensive and not suitable for real-time application. 2D, frame-based methods are generally more efficient, but suffer from lower recognition accuracies. In this paper, we present a hybrid keypose-based method that operates in a multi-camera environment, but uses only a single camera at a time. We learn, for each keypose, the relative utility of a particular viewpoint compared with switching to a different available camera in the network for future classification. On a benchmark multi-camera action recognition dataset, our method outperforms approaches that incorporate all available cameras.

Visual analytics is inherently a collaboration between human and computer. However, in current visual analytics systems, the computer has limited means of knowing about its users and their analysis processes. While existing research has shown that a user's interactions with a system reflect a large amount of the user's reasoning process, there has been limited advancement in developing automated, real-time techniques that mine interactions to learn about the user. In this paper, we demonstrate that we can accurately predict a user's task performance and infer some user personality traits by using machine learning techniques to analyze interaction data. Specifically, we conduct an experiment in which participants perform a visual search task, and apply well-known machine learning algorithms to three encodings of the users' interaction data. We achieve, depending on algorithm and encoding, between 62% and 83% accuracy at predicting whether each user will be fast or slow at completing the task. Beyond predicting performance, we demonstrate that using the same techniques, we can infer aspects of the user's personality factors, including locus of control, extraversion, and neuroticism. Further analyses show that strong results can be attained with limited observation time: in one case 95% of the final accuracy is gained after a quarter of the average task completion time. Overall, our findings show that interactions can provide information to the computer about its human collaborator, and establish a foundation for realizing mixed-initiative visual analytics systems.

Modeling the dynamics of heart and lung tissue is challenging because the tissue deforms between data acquisitions. To reconstruct complete volumes, sample data captured at different times and locations must be combined. This paper presents a novel end-to-end, data driven framework for the complete reconstruction of deforming tissue volumes. This framework is a joint optimization over an undeformed tissue volume, a deformation map that describes tissue motion forgiven pose parameters (i.e. breathing and heartbeat), and an estimate of those parameters for each data acquisition. Tissue motion is modeled by deforming a reference volume with a cubic B-spline free form deformation, and we use Isomap to derive initial estimates of the pose of sample data. An iterative method is used to simultaneously solve for the reference volume and deformation map while updating the pose estimates. This same process is demonstrated on 4D CT lung data and heart/lung MR data.

Researchers are increasingly interested in providing video-based, view-invariant action recognition for human motion. Addressing this problem will lead to more accurate modeling and analysis of the type of unconstrained video commonly collected in the areas of athletics and medicine. Previous viewpoint-invariant methods use multiple cameras in both the training and testing phases of action recognition or require storing many examples of a single action from multiple viewpoints. In this paper, we present a framework for learning a compact representation of primitive actions (e.g., walk, punch, kick, sit) that can be used for video obtained from a single camera for simultaneous action recognition and viewpoint estimation. Using our method, which models the low-dimensional structure of these actions relative to viewpoint, we show recognition rates on a publicly available data set previously only achieved using multiple simultaneous views.

Automated cell tracking using in vivo imagery is difficult, in general, due to the noise inherent in the imaging process, occlusions, varied cell appearance over time, motion of other tissue (distractors), and cells traveling in and out of the image plane. For certain types of cells these problems are exacerbated due to erratic motion patterns. In this paper, we introduce the Radial Flow Transform, which provides motion estimates for objects of interest in a scene without explicitly tracking each object. The transform is robust to misdetected objects, temporally-disjoint motion events, and can represent multiple directions of flow at a single location. We provide operations to convert to and from a vector field representation. This allows for intuitive reasoning about the motion patterns in a scene. We demonstrate results on synthetic data and in vivo microscopy video of a mouse liver.

We develop theory and algorithms to incorporate image manifold constraints in a level set segmentation algorithm. This provides a framework to simultaneously segment every image of data sets that vary due to two degrees of freedom - such as cardiopulmonary MR images which deform due to patient breathing and heartbeats. We derive two formulations: a 4D level set which loosely couples the level set function between neighbors in the 2D image manifold and a multilayer level set function which uses different levels of the level set function to represent shapes that shrink or grow. We characterize the set of shape manifolds that the multilayer level set function can represent, and derive the evolution equations for both frameworks. We offer results of segmenting the left ventricle in cardiopulmonary MRI; by automatically discovering the 2D manifold structure of the image set then simultaneously segmenting every frame. Both extensions improve on frame-by-frame approaches, and a comparison of the results offers insight into their strengths and weaknesses.

An important class of image data sets depict an object undergoing deformation. When there are only a few underlying causes of the deformation, these images have a natural low-dimensional structure which can be parameterized using manifold learning. This paper presents a method to solve for the deformation field as a function of the manifold coordinates-implicitly optimizing the deformation between all pairs of images simultaneously. Additionally, we provide a mechanism to create images for arbitrary coordinates of the manifold, addressing an important limitation of manifold learning algorithms for the case of images related through deformations. We give quantitative results in an artificial image morphing example and illustrate the method by finding the deformations relating all images of a cardiopulmonary MR image sequence.

Manifold learning has become a vital tool in data driven methods for interpretation of video, motion capture, and handwritten character data when they lie on a low dimensional, nonlinear manifold. This work extends manifold learning to classify and parameterize unlabeled data which lie on multiple, intersecting manifolds. This approach significantly increases the domain to which manifold learning methods can be applied, allowing parameterization of example manifolds such as figure eights and intersecting paths which are quite common in natural data sets. This approach introduces several technical contributions which may be of broader interest, including node-weighted multidimensional scaling and a fast algorithm for weighted low-rank approximation for rank-one weight matrices. We show examples for intersecting manifolds of mixed topology and dimension and demonstrations on human motion capture data.