Learning multiple domains/tasks with a single model is important for improving data efficiency and lowering inference cost for numerous vision tasks, especially on resource-constrained mobile devices. However, hand-crafting a multi-domain/task model can be both tedious and challenging. This paper proposes a novel approach to automatically learn a multi-path network for multi-domain visual classification on mobile devices. The proposed multi-path network is learned from neural architecture search by applying one reinforcement learning controller for each domain to select the best path in the super-network created from a MobileNetV3-like search space. An adaptive balanced domain prioritization algorithm is proposed to balance optimizing the joint model on multiple domains simultaneously. The determined multi-path model selectively shares parameters across domains in shared nodes while keeping domain-specific parameters within non-shared nodes in individual domain paths. This approach effectively reduces the total number of parameters and FLOPS, encouraging positive knowledge transfer while mitigating negative interference across domains. Extensive evaluations on the Visual Decathlon dataset demonstrate that the proposed multi-path model achieves state-of-the-art performance in terms of accuracy, model size, and FLOPS against other approaches using MobileNetV3-like architectures. Furthermore, the proposed method improves average accuracy over learning single-domain models individually, and reduces the total number of parameters and FLOPS by 78% and 32% respectively, compared to the approach that simply bundles single-domain models for multi-domain learning.

We study a new image sensor that is reminiscent of a traditional photographic film. Each pixel in the sensor has a binary response, giving only a 1-bit quantized measurement of the local light intensity. To analyze its performance, we formulate the oversampled binary sensing scheme as a parameter estimation problem based on quantized Poisson statistics. We show that, with a single-photon quantization threshold and large oversampling factors, the Cramér-Rao lower bound (CRLB) of the estimation variance approaches that of an ideal unquantized sensor, i.e., as if there were no quantization in the sensor measurements. Furthermore, the CRLB is shown to be asymptotically achievable by the maximum-likelihood estimator (MLE). By showing that the log-likelihood function of our problem is concave, we guarantee the global optimality of iterative algorithms in finding the MLE. Numerical results on both synthetic data and images taken by a prototype sensor verify our theoretical analysis and demonstrate the effectiveness of our image reconstruction algorithm. They also suggest the potential application of the oversampled binary sensing scheme in high dynamic range photography.

We present a system that automatically recommends tags for YouTube videos solely based on their audiovisual content. We also propose a novel framework for unsupervised discovery of video categories that exploits knowledge mined from the World-Wide Web text documents/searches. First, video content to tag association is learned by training classifiers that map audiovisual content-based features from millions of videos on YouTube.com to existing uploader-supplied tags for these videos. When a new video is uploaded, the labels provided by these classifiers are used to automatically suggest tags deemed relevant to the video. Our system has learned a vocabulary of over 20,000 tags. Secondly, we mined large volumes of Web pages and search queries to discover a set of possible text entity categories and a set of associated is-A relationships that map individual text entities to categories. Finally, we apply these is-A relationships mined from web text on the tags learned from audiovisual content of videos to automatically synthesize a reliable set of categories most relevant to videos – along with a mechanism to predict these categories for new uploads. We then present rigorous rating studies that establish that: (a) the average relevance of tags automatically recommended by our system matches the average relevance of the uploader-supplied tags at the same or better coverage and (b) the average precision@K of video categories discovered by our system is 70% with K=5.

We propose a new image device called gigavision camera. The main differences between a conventional and a gigavision camera are that the pixels of the gigavision camera are binary and orders of magnitude smaller. A gigavision camera can be built using standard memory chip technology, where each memory bit is designed to be light sensitive. A conventional gray level image can be obtained from the binary gigavision image by low-pass filtering and sampling. The main advantage of the gigavision camera is that its response is non-linear and similar to a logarithmic function, which makes it suitable for acquiring high dynamic range scenes. The larger the number of binary pixels considered, the higher the dynamic range of the gigavision camera will be. In addition, the binary sensor of the gigavision camera can be combined with a lens array in order to realize an extremely thin camera. Due to the small size of the pixels, this design does not require deconvolution techniques typical of similar systems based on conventional sensors.

Videos representing flames, water, smoke, etc., are often defined as dynamic textures: "textures" because they are characterized by the redundant repetition of a pattern and "dynamic" because this repetition is also in time and not only in space. Dynamic textures have been modeled as linear dynamic systems by unfolding the video frames into column vectors and describing their trajectory as time evolves. After the projection of the vectors onto a lower dimensional space by a singular value decomposition (SVD), the trajectory is modeled using system identification techniques. Synthesis is obtained by driving the system with random noise. In this paper, we show that the standard SVD can be replaced by a higher order SVD (HOSVD), originally known as Tucker decomposition. HOSVD decomposes the dynamic texture as a multidimensional signal (tensor) without unfolding the video frames on column vectors. This is a more natural and flexible decomposition, since it permits us to perform dimension reduction in the spatial, temporal, and chromatic domain, while standard SVD allows for temporal reduction only. We show that for a comparable synthesis quality, the HOSVD approach requires, on average, five times less parameters than the standard SVD approach. The analysis part is more expensive, but the synthesis has the same cost as existing algorithms. Our technique is, thus, well suited to dynamic texture synthesis on devices limited by memory and computational power, such as PDAs or mobile phones.

We consider localized instantaneous sources that reside in a 2D diffusive environment. Our goal is to reconstruct the induced field from the measurements obtained by distributed sensors. Although the field is non-bandlimited, we capitalize on the fact that it is completely determined by a finite number of parameters to develop a method that allows perfect reconstruction. We demonstrate how these results can be applied in practice in the particular case of heat diffusion. Simulation results confirm the effectiveness of the method.

In this paper, we model a spatially varying channel where a source is moving along a random trajectory with respect to a fixed receiver. The aim is to compute the power spectral density corresponding to the channel impulse response as a function of temporal and spatial frequencies. The trajectory of the source follows an auto regressive model where the poles of the system control the smoothness of the path. Theoretical results are presented for the AR2 case and generalized to any ARn systems. Simulations results are shown and compared to the presented theory. The stochastic plenacoustic function analyzed in this paper also models time-varying channels governed by the wave equation, like they appear in acoustic echo cancellation or ultra wide band communication channels

We consider the problem of reconstructing superimposed temperature and wind flow fields from acoustic measurements. A new technique based solely on acoustic waves propagation is presented. In contrast to the usual straight ray assumption, a bent ray model is considered in order to achieve higher accuracy. Under this assumption, we propose an iterative reconstruction algorithm that allows to entirely recover the considered fields. It alternates between estimating the fields of interest and the corresponding acoustic ray trajectories. Simulation results confirm the effectiveness and fast convergence of our scheme.

We present a new method for signal reconstruction from multiple sets of samples with unknown offsets. We rewrite the reconstruction problem as a set of polynomial equations in the unknown signal parameters and the offsets between the sets of samples. Then, we construct a Grobner basis for the corresponding affine variety. The signal parameters can then easily be derived from this Grobner basis. This provides us with an elegant solution method for the initial nonlinear problem. We show two examples for the reconstruction of polynomial signals and Fourier series

Dynamic textures are sequences of images showing temporal regularity. Examples can be found in videos representing smoke, flames, ocean waves, wind-shaken forests, etc. Dynamic texture modelling and synthesis has usually been done considering RGB color images. In this paper, we analyze the use of different color encodings, which permit to model luminance and chrominance information separately. We find that this separation is more appropriate, since it takes advantage of the spatial and temporal characteristics of the color channels and leads to more flexible and compact representations. We show that compared to RGB, similar synthesis performance can be achieved using YCbCror Lab color encodings, using half of the model coefficients and less computational power.

The sampling of a sound field using circular microphone arrays is studied. We give a new representation of the sound field sampled on such an array using the two-dimensional Fourier transform. We then show how we can reconstruct the sound field at all possible positions on the array from a finite number of measurements. We give the necessary azimuthal angular sampling frequency necessary to reconstruct the data up to some temporal frequency. This new technique is applied to interpolate head related transfer functions (HRTFs) where we also derive an angular sampling theorem.

Aliasing artifacts in images are visually very disturbing. Therefore, most imaging devices apply a low-pass filter before sampling. This removes all aliasing from the image, but it also creates a blurred image. Actually, all the image information above half the sampling frequency is removed. In this paper, we present a new method for the reconstruction of a high resolution image from a set of highly undersampled and thus aliased images. We use the information in the entire frequency spectrum, including the aliased part, to create a sharp, high resolution image. The unknown relative shifts between the images are computed using a subspace projection approach. We show that the projection can be decomposed into multiple projections onto smaller subspaces. This allows for a considerable reduction of the overall computational complexity of the algorithm. A high resolution image can then be reconstructed from the registered low resolution images. Simulation results show the validity of our algorithm.

In signal processing systems, aliasing is normally treated as a disturbing signal. That motivates the need for effective analog, optical and digital anti-aliasing filters. However, aliasing also conveys valuable information on the signal above the Nyquist frequency. Hence, an effective processing of the samples, based on a model of the input signal, would virtually allow the sampling frequency to be increased using slower and cheaper converters. We present such an algorithm for bandlimited signals that are sampled below twice the maximum signal frequency. Using a subspace method in the frequency domain, we show that these signals can be reconstructed from multiple sets of samples. The offset between the sets is unknown and can have arbitrary values. This approach can be applied to the creation of super-resolution images from sets of low resolution images. In this application, registration parameters have to be computed from aliased images. We show that parameters and high resolution images can be computed precisely, even when high levels of aliasing are present on the low resolution images.

In this paper, we present reconstruction schemes for the plenoptic function. Using new sampling methods, we show that, for some particular scenes, it is possible to perfectly reconstruct the plenoptic function from a finite number of cameras with finite resolution (Theorem 1 and Corollary 1). In more general cases, we demonstrate new ways of interpolating exactly the plenoptic function (Theorem 2). Finally, we show that it is possible to infer camera locations from a finite set of images. In all cases, we have perfect solutions due to the super-resolution property of the sampling. First numerical experiments on noisy observations show the potentiality of this new theoretical developments.

We address the question of how to characterize the outliers that may appear when matching two views of the same scene. The match is performed by comparing the difference of the two views at a pixel level aiming at a better registration of the images. When using digital photographs as input, we notice that an outlier is often a region that has been occluded, an object that suddenly appears in one of the images, or a region that undergoes an unexpected motion. By assuming that the error in pixel intensity generated by the outlier is similar to an error generated by comparing two random regions in the scene, we can build a model for the outliers based on the content of the two views. We illustrate our model by solving a pose estimation problem: the goal is to compute the camera motion between two views. The matching is expressed as a mixture of inliers versus outliers, and defines a function to minimize for improving the pose estimation. Our model has two benefits: First, it delivers a probability for each pixel to belong to the outliers. Second, our tests show that the method is substantially more robust than traditional robust estimators (M-estimators) used in image stitching applications, with only a slightly higher computational complexity.

This paper addresses a key issue in the problem of reconstructing a panoramic view from several pictures taken with a hand-held camera, namely the estimation of some ill-posed parameters using an external constraint. For many practical reasons, a panoramic reconstruction has to be performed in several independent steps, resulting in a set of different measurements of the same reality. For example, the focal length can be estimated with each pair of overlapping images. The idea is to introduce some a priori knowledge about the world by means of a constraint on the parameter set. In the former example, the constraint would impose equality on all the focal length estimates. This paper describes the appropriate correction that needs to be applied to the parameters in order to obtain a coherent result. It also suggests a way to evaluate if a constraint is plausible given a set of initial estimates. The basic idea behind the method is to modify the parameters without significantly changing the overlapping part of the images. The method is evaluated using two different experimental setups. The first aims at improving the quality of a full panoramic image. The second measures independently the positions in space of two planes using two pictures. The latter experiment shows that the two computed motions can be considered as a single one with two different planes in space.

Three-dimensional (3-D) motion estimation is applied to the problem of motion compensation for video coding. We suppose that the video sequence consists of the perspective projections of a collection of rigid bodies which undergo a rototranslational motion. Motion compensation can be performed on the sequence once the shape of the objects and the motion parameters are determined. We show that the motion equations of a rigid body can be formulated as a nonlinear dynamic system whose state is represented by the motion parameters and by the scaled depths of the object feature points. An extended Kalman filter is used to estimate both the motion and the object shape parameters simultaneously. The inclusion of the shape parameters in the estimation procedure adds a set of constraints to the filter equations that appear to be essential for reliable motion estimation. Our experiments show that the proposed approach gives two advantages. First, the filter can give more reliable estimates in the presence of measurement noise in comparison with other motion estimators that separately compute motion and structure. Second, the filter can efficiently track abrupt motion changes. Moreover, the structure imposed by the model implies that the reconstructed motion is very natural as opposed to more common block-based schemes. Also, the parameterization of the model allows for a very efficient coding of the motion information.

In this work, statistical based motion estimation is applied to the problem of motion estimation for video coding. We show that the motion equations of a rigid body can be formulated as a nonlinear dynamic system whose state is represented by the motion parameters and by the scaled depths of the object feature points. An extended Kalman filter is used to estimate the global motion, from which successive frames can be predicted in a motion compensated video coding system. The structure imposed by the model implies that the reconstructed motion is very natural in comparison to more common block-based schemes. Moreover, the parametrization of the model allows for a very efficient coding of motion information.