Conventional VR experiences rely heavily on visual input, creating barriers for individuals who are blind or have low vision (BLV). We present a mobile VR system that enables BLV users to explore virtual environments independently using a smartphone as a simulated white cane, augmented with spatial audio and haptic feedback. Built on ubiquitous consumer devices, the system supports embodied exploration and encourages physical movement, with multimodal feedback to help users transfer real-world cane techniques into virtual spaces and build mental maps. A user study shows that BLV participants can independently navigate complex virtual environments using familiar cane strategies. We conclude with design recommendations for accessible mobile VR systems informed by our findings.

Ensuring safe and independent navigation poses considerable difficulties for individuals who are blind or have low vision (BLV), as it requires detailed knowledge of their immediate environment. Our research highlights the critical need for accessible data on sidewalk materials and objects, which is currently lacking in existing map services. To bridge this gap, we present the Sidewalk Material Data Acquisition Framework (SMDAF), designed for large-scale data collection. This framework includes (1) a lightweight data collection system embedded in a white cane, which captures audio data through the interaction of the cane tip with the sidewalk surface, and a mobile app that facilitates data storage and management, resulting in a novel multimodal dataset comprising both image and audio data; and (2) a unique Cross-Modal Knowledge Distillation (CMKD) technique for an enhanced audio material classifier. Our CMKD approach employs an image-based model as the teacher to improve the audio model, incorporating an Enhanced Bidirectional learning method with an intuitive filtering technique: Bidirectional Correct Sample Filtering (BCSF). BCSF filters correct samples to prevent the distillation of incorrect knowledge, addressing the issue of inaccurate cross-modal learning. This novel approach has resulted in a 1.84% improvement in Macro Accuracy, achieving an overall accuracy of 87.62%, surpassing all state-of-the-art KD and CMKD methods. This study underscores the efficacy of SMDAF and provides a practical CMKD technique for future cross-modal learning tasks. Code and dataset are available (https://github.com/FgSurewin/SMDAF-CMKD).

In this work, a storefront accessibility image dataset is collected from Google street view and is labeled with three main objects for storefront accessibility: doors (for store entrances), doorknobs (for accessing the entrances) and stairs (for leading to the entrances). Then MultiCLU, a new multi-stage context learning and utilization approach, is proposed with the following four stages: Context in Labeling (CIL), Context in Training (CIT), Context in Detection (CID) and Context in Evaluation (CIE). The CIL stage automatically extends the label for each knob to include more local contextual information. In the CIT stage, a deep learning method is used to project the visual information extracted by a Faster R-CNN based object detector to semantic space generated by a Graph Convolutional Network. The CID stage uses the spatial relation reasoning between categories to refine the confidence score. Finally in the CIE stage, a new loose evaluation metric for storefront accessibility, especially for knob category, is proposed to efficiently help BLV users to find estimated knob locations. Our experiment results show that the proposed MultiCLU framework can achieve significantly better performance than the baseline detector using Faster R-CNN, with +13.4% on mAP and +15.8% on recall, respectively. Our new evaluation metric also introduces a new way to evaluate storefront accessibility objects, which could benefit BLV group in real life.

Gatherings of thousands to millions of people frequently occur for an enormous variety of educational, social, sporting, and political events, and automated counting of these high-density crowds is useful for safety, management, and measuring significance of an event. In this work, we show that the regularly accepted labeling scheme of crowd density maps for training deep neural networks may not be the most effective one. We propose an alternative inverse k-nearest neighbor (i[Formula: see text]NN) map mechanism that, even when used directly in existing state-of-the-art network structures, shows superior performance. We also provide new network architecture mechanisms that we demonstrate in our own MUD-i[Formula: see text]NN network architecture, which uses multi-scale drop-in replacement upsampling via transposed convolutions to take full advantage of the provided i[Formula: see text]NN labeling. This upsampling combined with the i[Formula: see text]NN maps further improves crowd counting accuracy. We further analyze several variations of the i[Formula: see text]NN labeling mechanism, which apply transformations on the [Formula: see text]NN measure before generating the map, in order to consider the impact of camera perspective views, image resolutions, and the changing rates of the mapping functions. To alleviate the effects of crowd density changes in each image, we also introduce an attenuation mechanism in the i[Formula: see text]NN mapping. Experimentally, we show that inverse square root [Formula: see text]NN map variation (iR[Formula: see text]NN) provides the best performance. Discussions are provided on computational complexity, label resolutions, the gains in mapping and upsampling, and details of critical cases such as various crowd counts, uneven crowd densities, and crowd occlusions.

In this work, we use a generative adversarial network (GAN) to train crowd counting networks using minimal data. We describe how GAN objectives can be modified to allow for the use of unlabeled data to benefit inference training in semi-supervised learning. More generally, we explain how these same methods can be used in more generic multiple regression target semi-supervised learning, with crowd counting being a demonstrative example. Given a convolutional neural network (CNN) with capabilities equivalent to the discriminator in the GAN, we provide experimental results which show that our GAN is able to outperform the CNN even when the CNN has access to significantly more labeled data. This presents the potential of training such networks to high accuracy with little data. Our primary goal is not to outpreform the state-of-the-art using an improved method on the entire dataset, but instead we work to show that through semi-supervised learning we can reduce the data required to train an inference network to a given accuracy. To this end, systematic experiments are performed with various numbers of images and cameras to show under which situations the semi-supervised GANs can improve results.

We propose a content-based 3-D mosaic (CB3M) representation for long video sequences of 3-D and dynamic urban scenes captured by a camera on a mobile platform. In the first phase, a set of parallel-perspective (pushbroom) mosaics with varying viewing directions is generated to capture both the 3-D and dynamic aspects of the scene under the camera coverage. In the second phase, a segmentation-based stereo matching algorithm is applied to extract parametric representations of the color, structure and motion of the dynamic and/or 3-D objects in urban scenes, where a lot of planar surfaces exist. Multiple pairs of stereo mosaics are used for facilitating reliable stereo matching, occlusion handling, accurate 3-D reconstruction, and robust moving target detection. CB3M is a highly compressed visual representation for a dynamic 3-D scene, and has object contents of both 3-D and motion information. Experimental results are given for various real video sequences of large-scale 3-D scenes.

Finding traversable paths using computer vision is one of the most important components of an intelligent mobile robot system. For a wall climbing robot that operates in an urban environment, it is essential to automatically detect surface types and orientations for switching between moving and climbing, and for applying different adhesive forces both to save energy and ensure its own safety. This paper presents a novel segmentation-based stereovision approach in order to rapidly obtain accurate 3D estimations of urban scenes with largely textureless areas and sharp depth changes. The new approach takes advantage of the fact that many man-made objects in an urban setting consist of planar surfaces. Our approach has three main components: extraction of natural (planar) matching primitives, stereo matching via three-step algorithm (global match, local match and plane fitting), and plane merging and parameter refinement. Experimental results are provided for real indoor scenes.

We have developed several basic components for our virtualized classroom project: automated data collection, intelligent media integration, and flexible user interfaces. As an example of media integration, we show that when the PPT slides are projected onto the digital whiteboard where the instructor annotates, modifies, or expands the PPT presentation, geometrically registering high-resolution images from the two sources yields high quality digital presentations (e-presentations). We use a low-cost digital camera as a bridge to align the two sources and present a hybrid registration approach to align the PPT and handwriting images. Experimental results are presented to validate our approach