We introduce WearVQA, the first benchmark specifically designed to evaluate the visual questionanswering (VQA) capabilities of multi-modal AI assistant on wearable devices like smart glasses. Unlikeprior benchmarks that focus on high-quality, third-person imagery, WearVQA reflects the unique chal-lenges of ego-centric interaction—where visual inputs may be occluded, poorly lit, unzoomed, or blurry,and questions are grounded in realistic wearable use cases. The benchmark comprises 2,500 carefullycurated image-question-answer triplets, spanning 7 diverse image domains including both text-centricand general scenes, 10 cognitive task types ranging from basic recognition to various forms of reasoning,and 6 common wearables-specific image quality issues. All questions are designed to be answerable usingonly the visual input and common senses. WearVQA is paired with a rigorous LLM-as-a-judge evaluationframework with 96% labeling accuracy. Open-source and proprietary multi-modal LLMs achieved a QAaccuracy as low as 24–52% on WearVQA, with substantial drops on lower-quality images and reasoning-heavy tasks. These observations position WearVQA as a comprehensive and challenging benchmark forguiding technicial advancement towards robust, real-world multi-modal wearables AI systems.

Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness.In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance.Specifically, we show that many test instances are either overor under-represented during retrieval, significantly hurting the retrieval performance.To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval.Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new stateof-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering.

Capturing depth and reflectivity images at low light levels from active illumination of a scene has wide-ranging applications. Conventionally, even with single-photon detectors, hundreds of photon detections are needed at each pixel to mitigate Poisson noise. We introduce a robust method for estimating depth and reflectivity using on the order of 1 detected photon per pixel averaged over the scene. Our computational imager combines physically accurate single-photon counting statistics with exploitation of the spatial correlations present in real-world reflectivity and 3D structure. Experiments conducted in the presence of strong background light demonstrate that our computational imager is able to accurately recover scene depth and reflectivity, while traditional maximum likelihood-based imaging methods lead to estimates that are highly noisy. Our framework increases photon efficiency 100-fold over traditional processing and thus will be useful for rapid, low-power, and noise-tolerant active optical imaging.

Multiplexed imaging is a powerful mechanism for achieving high signal-to-noise ratio (SNR) in the presence of signal-independent additive noise. However, for imaging in presence of only signal-dependent shot noise, multiplexing has been shown to significantly degrade SNR. Hence, multiplexing to increase SNR in presence of Poisson noise is normally thought to be infeasible. In this paper, we present an exception to this view by demonstrating multiplexing advantage when the scene parameters are non-negative valued and are observed through a low-rate Poisson channel.

We present a new method for simultaneously denoising and unwrapping phase in multi-frequency homodyne time-of-flight ranging for the formation of accurate depth maps despite low SNR of raw measurements. This is achieved with a new generalized approximate message passing (GAMP) algorithm for minimum mean-squared error estimation of the phase. A detailed, physically-accurate acquisition model is central in achieving high accuracy, and the use of the GAMP methodology allows low computational complexity despite dense dependencies and the nonlinearity and non-Gaussianity of the acquisition model. Numerical simulations demonstrate that our integrated approach performs better than separate unwrapping followed by denoising. This performance translates to lowering the optical power consumption of time-of-flight cameras for a fixed acquisition quality.

Range estimation at low light-levels is accomplished using pulsed illumination of the target and time-of-flight measurement of backscattered light using single-photon detectors. Photon arrival statistics for this problem are time-inhomogeneous Poisson point processes where the rate function is determined by the illumination waveform. Given the flexibility to choose from different illumination waveforms, an important design question is - how does the range estimation performance depend on the pulse shape? The maximum-likelihood (ML) range estimation problem is nonlinear and thus it is difficult to analytically compare the estimation performance from different illumination waveforms. In this paper, we present an information-theoretic framework for evaluating ML range estimation performance. We derive relationships between the entropy of the photon arrival observations and the Cramér-Rao lower bound (CRLB) on the range estimate by extending De Brujin's identity and isoperimetric properties for non-Gaussian distributions.

We propose a framework for simultaneous phase unwrapping and multipath interference cancellation (SPUMIC) in homodyne time-of-flight (ToF) cameras. Our multi-frequency acquisition framework is based on parametric modeling of the multipath interference phenomena. We use robust spectral estimation methods with low computational complexity to detect and estimate multipath parameters. Using simulations and analysis we demonstrate that our proposed solution is implementable in real-time on existing ToF cameras without requiring any hardware modifications.

We present Mime, a compact, low-power 3D sensor for unencumbered free-form, single-handed gestural interaction with head-mounted displays (HMDs). Mime introduces a real-time signal processing framework that combines a novel three-pixel time-of-flight (TOF) module with a standard RGB camera. The TOF module achieves accurate 3D hand localization and tracking, and it thus enables motion-controlled gestures. The joint processing of 3D information with RGB image data enables finer, shape-based gestural interaction.

Active range acquisition systems such as light detection and ranging (LIDAR) and time-of-flight (TOF) cameras achieve high depth resolution but suffer from poor spatial resolution. In this paper we introduce a new range acquisition architecture that does not rely on scene raster scanning as in LIDAR or on a two-dimensional array of sensors as used in TOF cameras. Instead, we achieve spatial resolution through patterned sensing of the scene using a digital micromirror device (DMD) array. Our depth map reconstruction uses parametric signal modeling to recover the set of distinct depth ranges present in the scene. Then, using a convex program that exploits the sparsity of the Laplacian of the depth map, we recover the spatial content at the estimated depth ranges. In our experiments we acquired 64×64-pixel depth maps of fronto-parallel scenes at ranges up to 2.1 M using a pulsed laser, a DMD array and a single photon-counting detector. We also demonstrated imaging in the presence of unknown partially-transmissive occluders. The prototype and results provide promising directions for non-scanning, low-complexity range acquisition devices for various computer vision applications.

Light detection and ranging (LIDAR) systems use time of flight (TOF) in combination with raster scanning of the scene to form depth maps, and TOF cameras instead make TOF measurements in parallel by using an array of sensors. Here we present a framework for depth map acquisition using neither raster scanning by the illumination source nor an array of sensors. Our architecture uses a spatial light modulator (SLM) to spatially pattern a temporally-modulated light source. Then, measurements from a single omnidirectional sensor provide adequate information for depth map estimation at a resolution equal that of the SLM. Proof-of-concept experiments have verified the validity of our modeling and algorithms.

Conventional imaging uses steady-state illumination and light sensing with focusing optics; variations of the light field with time are not exploited. We develop a signal processing framework for estimating the reflectance of a Lambertian planar surface in a known position using omnidirectional, time-varying illumination and unfocused, time-resolved sensing in place of traditional optical elements such as lenses and mirrors. Our model associates time sampling of the intensity of light incident at each sensor with a linear functional of . The discrete-time samples are processed to obtain -regularized estimates of . Improving on previous work, using nonimpulsive, bandlimited light sources instead of impulsive illumination significantly improves signal-to-noise ratio (SNR) and reconstruction quality. Our simulations suggest that practical diffuse imaging applications may be realized with commercially-available temporal light intensity modulators and sensors used in standard optical communication systems.

We show that multi-path analysis using images from a timeof-flight (ToF) camera provides a tantalizing opportunity to infer about 3D geometry of not only visible but hidden parts of a scene. We provide a novel framework for reconstructing scene geometry from a single viewpoint using a camera that captures a 3D time-image I(x, y, t) for each pixel. We propose a framework that uses the time-image and transient reasoning to expose scene properties that may be beyond the reach of traditional computer vision. We corroborate our theory with free space hardware experiments using a femtosecond laser and an ultrafast photo detector array. The ability to compute the geometry of hidden elements, unobservable by both the camera and illumination source, will create a range of new computer vision opportunities.

Many important early vision techniques, such as active contours (ACs), can be formulated as energy minimization. However, finding global optimum (minimum energy) configurations is often computationally intractable. Approximate solutions obtained using iterative numerical methods may be ill-conditioned, and exhibit poor convergence and inaccuracy due to noise and discretization errors. We formulate AC as a statistical estimation problem and solve it using (Gaussian) message passing on factor graphs of linear models. The resulting algorithm exhibits faster convergence and the solutions possess higher numerical stability, robustness and accuracy.

Effective self-organization schemes lead to the creation of autonomous and reliable robot teams that can outperform a single, sophisticated robot on several tasks. We present here a novel, vision-based microscopic framework for active and distributed object-recognition and pose-estimation using a team of robots of simple construction. The team performs the task of locating a given object(s) in an unknown territory, recognizing it with sufficient confidence and estimating its pose. The larger goal is to experiment with probabilistic frameworks and graph-theoretic methods in the design of robot teams to achieve autonomous self-organization independent of the task at hand. We have chosen 3D object recognition as a first problem area to evaluate the effectiveness of our system design. The system comprises a probabilistic framework for the successful detection of the object in a coordinated manner and adaptive measures in case of machinery failures or presence of obstacles. A pose estimation method for the detected object and graph theoretic solutions for optimal field coverage by the robots are also presented. Each robot is provided with a part-based, spatial model of the object. The object to be recognized is taken to be much bigger than the robots and need not fit completely into the field of view of the robot cameras. We assume no knowledge of the internal parameters of the robot cameras and perform no camera calibration procedures. Initial simulation results corroborate our system design and field coverage methods