Hyperspectral images record the electromagnetic spectrum, and each hyperspectral pixel often stores hundreds of channels. Consequently, a hyperspectral image contains an order of magnitude more information than a similar-sized RGB color image. Concomitant with the decreasing cost of capturing these images, there is a need to develop efficient techniques for storing, transmitting, and analyzing hyperspectral images. This article develops a method for hyperspectral image compression using implicit neural representations (INRs) where a multilayer perceptron (MLP) network with sinusoidal activation functions “learns” to map pixel locations to pixel spectrum for a given hyperspectral image. This representation, thus, acts as a compressed encoding of this image, and the original image is reconstructed by evaluating this network at each pixel location. We introduce a sampling scheme to achieve better compression times while keeping decoding errors low. The proposed method is evaluated on four benchmarks against 16 other schemes for hyperspectral compression, and according to the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics, the method developed in this article achieves state-of-the-art compression rates at low-bit rates. In addition, we show that the proposed sampling technique reduces encoding times.

The hyperspectral pixel unmixing aims to find the underlying materials (endmembers) and their proportions (abundances) in pixels of a hyperspectral image. This work extends the Latent Dirichlet Variational Autoencoder (LDVAE) pixel unmixing scheme by taking into account local spatial context while performing pixel unmixing. The proposed method uses an isotropic convolutional neural network with spatial attention to encode pixels as a dirichlet distribution over endmembers. We have evaluated our model on Samson, Hydice Urban, Cuprite, and OnTech-HSI-Syn-21 datasets. Our model also leverages the transfer learning paradigm for Cuprite Dataset, where we train the model on synthetic data and evaluate it on the real-world data. The results suggest that incorporating spatial context improves both endmember extraction and abundance estimation.

Methods for estimating the remaining useful life of bearings are of great value in preventing them from failing. Because of this, different types of methods have been suggested. One of the most prominent is transfer learning since it achieves high performance despite the discrepancies in data expected in industrial applications. However, most current research ignores aspects such as the influence of non-bearing related faults, the non-existence of historical run-to-failure data from the target domain and knowledge transfer between machines. Considering this, how existing methods will cope with these scenarios is not established. Therefore, this study examines the performance of transfer learning methods based on current studies methodolo-gies. The result suggests that there are limitations to the assumptions made in the current research and that more considerations for scenarios found in industrial applications are needed both in the development and evaluation phases to increase the possibility of applying transfer learning methods in practice.

We present a method for hyperspectral pixel unmixing. The proposed method assumes that (1) abundances can be encoded as Dirichlet distributions and (2) spectra of endmembers can be represented as multivariate Normal distributions. The method solves the problem of abundance estimation and endmember extraction within a variational autoencoder setting where a Dirichlet bottleneck layer models the abundances, and the decoder performs endmember extraction. The proposed method can also leverage transfer learning paradigm, where the model is only trained on synthetic data containing pixels that are linear combinations of one or more endmembers of interest. In this case, we retrieve endmembers (spectra) from the United States Geological Survey Spectral Library. The model thus trained can be subsequently used to perform pixel unmixing on “real data” that contains a subset of the endmembers used to generated the synthetic data. The model achieves state-of-the-art results on several benchmarks: Cuprite, Urban Hydice and Samson. We also present new synthetic dataset, OnTech-HSI-Syn-21, that can be used to study hyperspectral pixel unmixing methods. We showcase the transfer learning capabilities of the proposed model on Cuprite and OnTech-HSI-Syn-21 datasets. In summary, the proposed method can be applied for pixel unmixing a variety of domains, including agriculture, forestry, mineralogy, analysis of materials, healthcare, etc. Additionally, the proposed method eschews the need for labelled data for training by leveraging the transfer learning paradigm, where the model is trained on synthetic data generated using the endmembers present in the “real” data.

We develop a hyperspectral pixel unmixing method that uses a Latent Variational Autoencoder within an analysis-synthesis loop to (1) construct pure spectra of the materials present in an image and (2) infer the mixing ratios of these materials in hyperspectral pixels without the need of labelled data. On OnTech-HIS-Syn-6em synthetic dataset that contains pixel unmixing groundtruth, the proposed method achieves acc = 100%, SAD = 0.0582 and RMSE = 0.0695 for segmentation, endmember extraction and abundance estimation, respectively. On HYDICE Urban benchmark, the proposed method achieves acc = 72.4%, SAD = 0.1669 and RMSE = 0.1984 for segmentation, endmember extraction and abundance estimation, respectively. Additionally, we applied this technique for crop analysis on hyperspectral data collected by the United States Department of Agriculture and achieved a coefficient of determination R2= 0.7 with respect to the ground truth. These results confirm that the proposed method is able to perform pixel unmixing without using labelled data.

There is a large growth in hardware and software systems capable of producing vast amounts of image and video data. These systems are rich sources of continuous image and video streams. This motivates researchers to build scalable computer vision systems that utilize data-streaming concepts for processing of visual data streams. However, several challenges exist in building large-scale computer vision systems. For example, computer vision algorithms have different accuracy and speed profiles depending on the content, type, and speed of incoming data. Also, it is not clear how to adaptively tune these algorithms in large-scale systems. These challenges exist because we lack formal frameworks for building and optimizing large-scale visual processing. This paper presents formal methods and algorithms that aim to overcome these challenges and improve building and optimizing large-scale computer vision systems. We describe a formal algebra framework for the mathematical description of computer vision pipelines for processing image and video streams. The algebra naturally describes feedback control and provides a formal and abstract method for optimizing computer vision pipelines. We then show that a general optimizer can be used with the feedback-control mechanisms of our stream algebra to provide a common online parameter optimization method for computer vision pipelines.

We propose a new method for remote sensing image matching. The proposed method uses an encoder subnetwork of an autoencoder pretrained on the GTCrossView data to construct image features. A discriminator network trained on the University of California Merced land-use/land-cover data set (LandUse) and the high-resolution satellite scene data set (SatScene) computes a match score between a pair of computed image features. We also propose a new network unit, called residual-dyad, and empirically demonstrate that networks that use residual-dyad units outperform those that do not. We compare our approach with both traditional and more recent learning-based schemes on the LandUse and SatScene data sets, and the proposed method achieves the state-of-the-art result in terms of mean average precision and average normalized modified retrieval rank (ANMRR) metrics. Specifically, our method achieves an overall improvement in performance of 11.26% and 22.41%, respectively, for LandUse and SatScene benchmark data sets.

We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layers are general, and that they learn generalized coordinates over the input domain. Deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we also apply our method to networks trained on MNIST, and show it is consistent with, and complimentary to, another study of intrinsic dimensionality.

Image alignment techniques have gained popularity for constructing image mosaics from video sequences. These image alignment techniques, however, have a hard time dealing with motion parallax, which limits their applicability. This paper studies image mosaicing in the presence of motion parallax and develops a new algorithm for generating view dependent image mosaics from low-flying aerial video sequences exhibiting strong parallax effects. Specifically we develop an energy minimization framework that computes a dense depth map of the scene from a sequence of images (captured by an uncalibrated camera following an unknown trajectory), which in turn can be used to generate a panoramic mosaic through view interpolation. We evaluate our algorithm on real and synthetic aerial video sequences and show that the proposed algorithm can construct high quality image mosaics even in the presence of strong parallax.

We present a new framework for capturing videos using sensor-rich mobile devices, such as smartphones, tablets, etc. Many of today's mobile devices are equipped with a variety of sensors, including accelerometers, magnetometers and gyroscopes, which are rarely used during video capture for anything more than video stabilization. We demonstrate that these sensors, together with the information that can be extracted from the recorded video via computer vision techniques, provide a rich source of data that can be leveraged to automatically edit and "clean up" the captured video. Sensor data, for example, can be used to identify undesirable video segments that are then hidden from view. We showcase an Android video recording app that captures sensor data during video recording and is capable of automatically constructing final-cuts from the recorded video. The app uses the captured sensor data plus computer vision algorithms, such as focus analysis, face detection, etc., to filter out undesirable segments and keep visually appealing portions of the captured video to create a final cut. We also show how information from various sensors and computer vision routines can be combined to create different final cuts with little or no user input.

We developed a new method for extracting 3D flight trajectories of droplets using high-speed stereo capture. We noticed that traditional multi-camera tracking techniques fare poorly on our problem, in part due to the fact that all droplets have very similar shapes, sizes and appearances. Our method uses local motion models to track individual droplets in each frame. 2D tracks are used to learn a global, non-linear motion model, which in turn can be used to estimate the 3D locations of individual droplets even when these are not visible in any camera. We have evaluated the proposed method on both synthetic and real data and our method is able to reconstruct 3D flight trajectories of hundreds of droplets. The proposed technique solves for both the 3D trajectory of a droplet and its motion model concomitantly, and we have found it to be superior to 3D reconstruction via triangulation. Furthermore, the learned global motion model allows us to relax the simultaneity assumptions of stereo camera systems. Our results suggest that, even when full stereo information is available, our unsynchronized reconstruction using the global motion model can significantly improve the 3D estimation accuracy.

This paper presents a new approach for automatic road detection in traffic cameras. The technique proposed here detects the dominant road boundary and estimates the vanishing point in images captured by traffic cameras under a wide range of lighting and environmental conditions, e.g., in images of unlit highways captured at night, etc. The approach starts by segmenting the traffic scene into a number of superpixel regions. The contours of these regions are used to generate a large number of edges which are organized into clusters of co-linearly similar sets using hierarchical bottom up clustering. A confidence level is assigned to each cluster using a statistical approach and the best clusters are chosen. Pairs of clusters with high confidence levels are then ranked and filtered according to image perspective and activity. The top ranked pair is selected as the road boundary. The proposed technique is tested on a real world dataset collected from the Ontario 401 traffic surveillance system. Experimental results demonstrate a distinct speedup and improvement in accuracy of the proposed technique in detecting the dominant road boundary in challenging scenarios compared to the state of the art Gabor filter based technique.

We envision future video sensor networks comprising tether-less smart camera nodes capable of supporting a variety of applications, ranging from video surveillance to traffic management, smart environments to ecological monitoring, etc. A key difference between video sensor networks and traditional multi-camera systems is that the later typically are not concerned with power, storage, and bandwidth usage. Power requirements, especially, must be considered when designing tether-less smart camera networks, since the operational life of a camera node is closely tied to the available power. Video capture and processing performed on a camera node and the communication between nodes needed to carry out collaborative sensing tasks impact power usage of these camera nodes. Therefore, one must devise strategies to minimize video capture and processing at each node and communication between nodes in order to reduce power consumption at each node, thereby increasing the operational life of a video sensor network.

We demonstrate a video surveillance system- comprising passive and active pan/tilt/zoom (PTZ) cameras-that intelligently responds to scene complexity, automatically capturing higher resolution video when there are fewer people in the scene and capturing lower resolution video as the number of pedestrians present in the scene increases. To this end, we have developed behavior based-controllers for passive and active cameras, enabling these cameras to carry out multiple observation tasks simultaneously. The research presented herein is a step towards video surveillance systems-consisting of a heterogeneous set of sensors-that provide persistent coverage of large spaces, while optimizing surveillance data collection by tuning the sensing parameters of individual sensors (in a distributed manner) in response to scene activity.

This paper presents a framework for preserving privacy in video surveillance. Raw video is decomposed into a background and one or more object-video streams. Object-video streams can be combined to render the scene in a variety of ways: (1) The original video can be reconstructed from object-video streams without any data loss; (2) individuals in the scene can be represented as blobs, obscuring their identities; (3) foreground objects can be color coded to convey subtle scene information to the operator, again without revealing the identities of the individuals present in the scene; (4) the scene can be partially rendered, i.e., revealing the identities of some individuals, while preserving the anonymity of others. We evaluate our approach in a virtual train station environment populated by autonomous, lifelike virtual pedestrians.

We introduce a distributed camera coalition formation scheme for perceptive scene coverage and persistent surveillance by smart camera sensor networks. The proposed model supports task-dependent camera selection and grouping via a "contract net" task allocation protocol augmented with conflict resolution and error recovery mechanisms. Our technique avoids any central controller, and it is robust to node failures and imperfect communication. In the design and empirical evaluation of our camera networks, we exploit a visually and behaviorally realistic virtual environment simulator that is populated by autonomous, lifelike virtual pedestrians.

This paper presents our research towards smart camera networks capable of carrying out advanced surveillance tasks with little or no human supervision. A unique centerpiece of our work is the combination of computer graphics, artificial life, and computer vision simulation technologies to develop such networks and experiment with them. Specifically, we demonstrate a smart camera network comprising static and active simulated video surveillance cameras that provides extensive coverage of a large virtual public space, a train station populated by autonomously self-animating virtual pedestrians. The realistically simulated network of smart cameras performs persistent visual surveillance of individual pedestrians with minimal intervention. Our innovative camera control strategy naturally addresses camera aggregation and handoff, is robust against camera and communication failures, and requires no camera calibration, detailed world model, or central controller.

This paper advocates a virtual vision paradigm and demonstrates its usefulness in camera sensor network research. Virtual vision prescribes the use of a visually and behaviorally realistic virtual environment simulator in the design and evaluation of surveillance systems. Impediments to deploying and experimenting with appropriately complex camera networks makes virtual vision an attractive alternative for many vision researchers who are motivated to investigate high level multi-camera control issues within such networks. In particular, we present two prototype surveillance systems comprising passive and active pan/tilt/zoom cameras. We deploy these systems in a virtual train station environment populated by autonomous, lifelike virtual pedestrians. The easily reconfigurable virtual cameras situated throughout this environment generate synthetic video feeds that emulate those acquired by real surveillance cameras monitoring extensive public spaces. Our novel multi-camera control strategies enable the cameras to collaborate in persistently observing pedestrians of interest that move across their fields of view and in capturing close-up videos of pedestrians as they travel through designated areas. The sensor networks support task-dependent camera node selection and aggregation through local decision-making and inter-node communication. Our approach to multi-camera control is robust to node failures and message loss.

The virtual vision paradigm features a unique synergy of computer graphics, artificial life, and computer vision technologies. Virtual vision prescribes visually and behaviorally realistic virtual environments as a simulation tool in support of research on large-scale visual sensor networks. Virtual vision has facilitated our research into developing multi-camera control and scheduling algorithms for next-generation smart video surveillance systems.