Task-aware navigation continues to be a challenging area of research, especially in scenarios involving open vocabulary. Previous studies primarily focus on finding suitable locations for task completion, often overlooking the importance of the robot’s pose. However, the robot’s orientation is crucial for successfully completing tasks because of how objects are arranged (e.g., to open a refrigerator door). Humans intuitively navigate to objects with the right orientation using semantics and common sense. For instance, when opening a refrigerator, we naturally stand in front of it rather than to the side. Recent advances suggest that Vision-Language Models (VLMs) can provide robots with similar common sense. Therefore, we develop a VLM-driven method called Navigation-to-Gaze (Navi2Gaze) for efficient navigation and object gazing based on task descriptions. This method uses the VLM to score and select the best pose from numerous candidates automatically. In evaluations on multiple photorealistic simulation benchmarks, Navi2Gaze significantly outperforms existing approaches by precisely determining the optimal orientation relative to target objects, resulting in a 68.8% reduction in Distance to Goal (DTG). Real-world video demonstrations can be found on the supplementary website1.

A noise-corrupted image often requires interpolation. Given a linear denoiser and a linear interpolator, when should the operations be independently executed in separate steps, and when should they be combined and jointly optimized? We study joint denoising / interpolation of images from a mixed graph filtering perspective: we model denoising using an undirected graph, and interpolation using a directed graph. We first prove that, under mild conditions, a linear denoiser is a solution graph filter to a maximum a posteriori (MAP) problem using an undirected graph smoothness prior, while a linear interpolator is a solution to a MAP problem using a directed graph smoothness prior. Next, we study two variants of the joint interpolation / denoising problem: a graph-based denoiser followed by an interpolator has an optimal separable solution, while an interpolator followed by a denoiser has an optimal non-separable solution. Experiments show that our joint denoising / interpolation method outperformed separate approaches noticeably.

The increasing prevalence of mobile devices has led to significant advancements in mobile camera systems and improved image quality. Nonetheless, mobile photography still grapples with flare corruptions such as reflective flare. The absence of a comprehensive real image dataset tailored for mobile phones hinders the development of effective flare mitigation techniques. To address this issue, we present a novel real image dataset specifically designed for mobile camera systems, focusing on flare removal. Capitalizing on the distinct properties of real images, this dataset serves as a solid foundation for developing advanced flare removal algorithms. The dataset comprises over 1,100 pairs of high-quality, full-resolution images for reflective flare, which generate 2,200 paired patches, ensuring broad adaptability across various imaging conditions. Experimental results demonstrate that networks trained with synthesized data struggle to cope with the complex lighting settings present in this real image dataset. Our dataset is expected to enable an array of new research in flare removal and contribute to substantial improvements in mobile image quality, benefiting mobile photographers and end-users alike.

The rise of mobile devices has spurred advancements in camera technology and image quality. However, mobile photography still faces issues like scattering and reflective flares. While previous research has acknowledged the negative impact of the mobile devices' internal image signal processing pipeline (ISP) on image quality, the specific ISP operations that hinder flare removal have not been fully identified. In addition, current solutions only partially address ISP-related deterioration due to a lack of comprehensive raw image datasets for flare study. To bridge these research gaps, we introduce a new raw image dataset tailored for mobile camera systems, focusing on eliminating flare. This dataset encompasses over 2,000 high-quality, full-resolution raw image pairs for scattering flare, and 1,200 for reflective flare, captured across various real-world scenarios, mobile devices, and camera settings. It is designed to enhance the generalizability of flare removal algorithms across a wide spectrum of conditions. Through detailed experiments, we have identified that ISP operations, such as denoising, compression, and sharpening, may either improve or obstruct flare removal, offering critical insights into optimizing ISP configurations for better flare mitigation. Our dataset is poised to advance the understanding of flare-related challenges, enabling more precise incorporation of flare removal steps into the ISP. Ultimately, this work paves the way for significant improvements in mobile image quality, benefiting both enthusiasts and professional mobile photographers alike.

During the Covid-19 pandemic, like the vast majority of countries in the world, Canada was under government-mandated lockdown, creating unprecedented challenges for the higher education system. This has exacerbated the problem of gender and ethnic inequalities in the STEM field due to the sudden disappearance of in-person communication and communities that had supported minority groups. To provide emergency support and reduce the known gender / ethnic gap, at York University in Toronto we designed a 3D imaging project for Ontario’s high school (HS) students, as part of an annual summer outreach program in the Lassonde School of Engineering. The project aims to create an equitable opportunity for HS students, providing a comprehensive introduction to image processing through experiential learning. We document our design methodology and experiences in the project, as well as feedback and evaluations from participants at all levels. We believe such documentation is valuable to promote gender- and ethnic-balanced education in image processing and the broader STEM field in the future, in an increasingly unpredictable environment due to climate change.

Recapturing attack can be employed as a simple but effective anti-forensic tool for digital document images. Inspired by the document inspection process that compares a questioned document against some known samples, we proposed a document recapture detection scheme by employing a Siamese network to compare and extract distinct features in a recaptured document image. The proposed algorithm takes advantage of both metric learning and image forensic techniques, and forms triplets by considering some important factors in document authentication, e.g., document types, resolutions, and content in each image patch. After training with our triplet selection strategy, the resulting feature embedding clusters the genuine samples near the reference while pushing the recaptured samples apart. In the experiment, we consider practical settings under domain differences, such as the variations in printing/imaging devices, substrates, recapturing channels, and document types. To evaluate the robustness of different approaches, we benchmark some popular off-the-shelf machine learning-based approaches, a state-of-the-art document image detection scheme, and the proposed schemes with different network backbones under various experimental protocols. Experimental results show that the proposed scheme consistently outperforms the state-of-the-art approaches under different experimental settings. Specifically, under the most challenging scenario in our experiment, i.e., evaluation across different types of documents (produced by different manufacturers, devices, and substrates), we have achieved 6.92% APCER (Attack Presentation Classification Error Rate) and 8.51% BPCER (Bona Fide Presentation Classification Error Rate) by the proposed network with ResNeXt101 backbone at 5.00% BPCER decision threshold.

To compose a 360° image from a rig with multiple fisheye cameras, a conventional processing pipeline first performs demosaicking on each fisheye camera’s Bayer-patterned grid, then translates demosaicked pixels from the camera grid to a rectified image grid—thus performing two image interpolation steps in sequence. Hence interpolation errors can accumulate, and acquisition noise in the captured pixels can pollute neighbors in two consecutive processing stages. In this paper, we propose a joint processing framework that performs demosaicking and grid-to-grid mapping simultaneously—thus limiting noise pollution to one interpolation. Specifically, we first obtain a reverse mapping function from a regular on-grid location in the rectified image to an irregular off-grid location in the camera’s Bayer-patterned image. For each pair of adjacent pixels in the rectified grid, we estimate its gradient using the pair’s neighboring pixel gradients in three colors in the Bayer-patterned grid. We construct a similarity graph based on the estimated gradients, and interpolate pixels in the rectified grid directly via graph Laplacian regularization (GLR). Experiments show that our joint method outperforms several competing local methods that execute demosaicking and rectification in sequence, by up to 0.52 dB in PSNR and 0.086 in SSIM on the publicly available dataset, and by up to 5.53dB in PSNR and 0.411 in SSIM on the in-house constructed dataset.