Wearable vibrotactile devices have many potential applications, including sensory substitution for accessibility and notifications. Currently, vibrotactile experimentation is done using large lab setups. However, most practical applications require standalone on-body devices and integration into small form factors. Such integration is time-consuming and requires expertise.

The authors present a framework for interactive design of new image stylisations using a wide range of predefined filter blocks. Both novel and off‐the‐shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters. In parallel to this manual design, they propose a novel procedural approach that automatically assembles sequences of filters, leading to unique and novel styles. An important aim of the authors’ framework is to allow for interactive exploration and design, as well as to enable videos and camera streams to be stylised on the fly. In order to achieve this real‐time performance, they use the Best Linear Adaptive Enhancement (BLADE) framework – an interpretable shallow machine learning method that simulates complex filter blocks in real time. Their representative results include over a dozen styles designed using their interactive tool, a set of styles created procedurally, and new filters trained with their BLADE approach.

In this work, we broadly connect kernel-based filtering (e.g. approaches such as the bilateral filter and nonlocal means, but also many more) with general variational formulations of Bayesian regularized least squares, and the related concept of proximal operators. Variational/Bayesian/proximal formulations often result in optimization problems that do not have closed-form solutions, and therefore typically require global iterative solutions. Our main contribution here is to establish how one can approximate the solution of the resulting global optimization problems using locally adaptive filters with specific kernels. Our results are valid for small regularization strength (i.e. weak noise) but the approach is powerful enough to be useful for a wide range of applications because we expose how to derive a "kernelized" solution to these problems that approximates the global solution in one shot, using only local operations. As another side benefit in the reverse direction, given a local data-adaptive filter constructed with a particular choice of kernel, we enable the interpretation of such filters in the variational/Bayesian/proximal framework.

The Rapid and Accurate Image Super Resolution (RAISR) method of Romano, Isidoro, and Milanfar is a computationally efficient image upscaling method using a trained set of filters. We describe a generalization of RAISR, which we name Best Linear Adaptive Enhancement (BLADE). This approach is a trainable edge-adaptive filtering framework that is general, simple, computationally efficient, and useful for a wide range of problems in computational photography. We show applications to operations which may appear in a camera pipeline including denoising, demosaicking, and stylization.

Denoising is a fundamental imaging problem. Versatile but fast filtering has been demanded for mobile camera systems. We present an approach to multiscale filtering which allows real-time applications on low-powered devices. The key idea is to learn a set of kernels that upscales, filters, and blends patches of different scales guided by local structure analysis. This approach is trainable so that learned filters are capable of treating diverse noise patterns and artifacts. Experimental results show that the presented approach produces comparable results to state-of-the-art algorithms while processing time is orders of magnitude faster.

We have implemented a near-ultrasonic communication protocol in the 18.5-20 kHz band, which is inaudible to most humans, using commodity smartphone speakers and microphones to transmit and receive signals. The protocol described in this paper is a component of Google's Nearby platform, where near-ultrasound signals are used to establish copresence between nearby devices by transmitting a short token. High-frequency sound does not pass through walls (most energy is reflected), so identified devices are constrained to approximately the same room, “within earshot” of one another. Our protocol has a raw data rate of 94.5 b/s, and we find in real indoor environments that transmission between mobile devices is reliable at 2 m distance and often works at 10 m. We use direct-sequence spread spectrum modulation, which makes it highly robust to multipath, motion, and narrowband noise. We use a 127-chip pseudorandom code, repeating once per data symbol, and modulate its amplitude with orthogonal sine waveforms encoding 4-bit symbol values. We add the orthogonal sines to a constant “pedestal,” which is inefficient in an information-theoretic sense, but makes synchronization easier. We describe a robust and computationally efficient transmitter and receiver implementations and show experiments on real and simulated data.

Robust and far-field speech recognition is critical to enable true hands-free communication. In far-field conditions, signals are attenuated due to distance. To improve robustness to loudness variation, we introduce a novel frontend called per-channel energy normalization (PCEN). The key ingredient of PCEN is the use of an automatic gain control based dynamic compression to replace the widely used static (such as log or root) compression. We evaluate PCEN on the keyword spotting task. On our large rerecorded noisy and far-field eval sets, we show that PCEN significantly improves recognition performance. Furthermore, we model PCEN as neural network layers and optimize high-dimensional PCEN parameters jointly with the keyword spotting acoustic model. The trained PCEN frontend demonstrates significant further improvements without increasing model complexity or inference-time cost.

With the journal Image Processing On Line (IPOL), we propose to promote software to the status of regular research material and subject it to the same treatment as research papers: it must be reviewed, it must be reusable and verifiable by the research community, it must follow style and quality guidelines. In IPOL, algorithms are published with their implementation, codes are peer-reviewed, and a web-based test interface is attached to each of these articles. This results in more software released by the researchers, a better software quality achieved with the review process, and a large collection of test data gathered for each article. IPOL has been active since 2010, and has already published thirty articles.

Image interpolation is the problem of increasing the resolution of a given image. An important aspect of interpolation is accurate estimation of edge orientations. This work introduces contour stencils, a new method for estimating the image contours based on total variation along curves. Contour stencils are able to distinguish lines of different orientations, curves, corners, and other geometric features with a computationally efficient formula. Contour stencils are applied in designing an edge directed color interpolation method. The method incorporates an efficient approximation of deconvolution. Although most interpolation methods that involve deconvolution require either solving a large linear system or running many iterations, this method has linear complexity in the number of pixels and can be computed in one or a small number of passes through the image. Comparisons show that the proposed interpolation is competitive with existing methods.