High-resolution imaging, and particularly video sensors are gaining interest by the space industry during the last years, up to the point that the main space agencies have included in their roadmap the development of different processing stages for their efficient management on-board satellites. This is relevant because the high volume of data generated by these kind of sensors, which make difficult either send or store it in raw format. One of the main objectives of the European H2020-funded VIDEO project is to provide some tools to detect on-the-fly different targets during an Earth Observation mission to then smartly compress the acquired video considering different level of loses depending on the frame content. Currently, the VIDEO project is in its last phase, so a demonstrator is on-going to evaluate the viability of the proposed instrument and its associated processing chain. In this work, the main features of this demonstrator are presented and some preliminary conclusions of the project are summarized.

The use of next-generation and high-resolution imaging sensors is gaining interest for space missions, because of their properties for identification and exploration purposes. It is expected that the demand for video sensors in the space industry will increase during the next years, mainly for monitoring and exploration missions. In this context, onboard video compression techniques emerge as mandatory, because memory resources are available on space missions and the downlink bandwidth with ground prevents for the transmission of raw video, especially when near real-time capabilities are required. The complexity of the current commercial video encoders supposes a challenge for their implementation on the hardware used on space missions, because of their high architectural complexity and computational requirements, deriving in an area overhead and an unacceptable power consumption. In this context, we propose the use of the CCSDS-123.0-B-2 standard, originally focused on the near-lossless on-board compression of multispectral and hyperspectral images, for compressing panchromatic video. The proposed solution presents low complexity, because it is specifically designed for working on space missions and also has the capability of compressing both 3-D images and panchromatic video by using a single processing core. Results demonstrate that this solution achieves high compression ratios, maintaining a considerable video quality after reconstruction on ground and achieving a fine-grain control of the losses introduced in the compression chain.

The HyperLCA compressor is a transform based algorithm specifically designed for the real-time compression of hyperspectral images captured by pushbroom scanners, using limited computational resources. It is based on the HyperLCA Transform, which follows an unmixing-like strategy to independently compress each hyperspectral frame in a causal manner. A novel approach with respect to the original HyperLCA Transform is introduced in this work. By reusing the information used to compress one frame in the subsequent frames, it has been possible to increase the HyperLCA Transform compression performance and to reduce its computational burden. Additionally, the proposed approach is applicable not only to the targeted compressor but also to other causal hyperspectral analysis algorithms based on orthogonal projections and/or unmixing like strategies. The proposed solution has been tested in a real UAV-based acquisition platform, demonstrating the ability of our proposal to compress and transmit the captured hyperspectral data to a ground station in real-time.

Remote sensing has gained relevance in the last years, mainly due to the emergence of UAVs carrying airborne imagery sensors. In this regard, the on-board data processing for on-the-fly making-decision applications is also gaining momentum. Nevertheless, these flight vehicles are still limited in terms of power budget and computational capacity, which hampers the handling of the hyperspectral data. Consequently, there is an emerging trend towards the development of more hardware-friendly algorithms suitable for an efficient implementation in parallel computing devices. In this sense, the LbL-FAD algorithm arose in response to the lack of causal anomaly detectors that could be easily integrated in push-broom-based acquisition systems. In this work, we have analysed the feasibility of the performance and power needs of the LbL-FAD algorithm in a mid-range re-configurable FPGA-SoC such as the XC7Z020 chip. Concretely, a highly optimized FPGA accelerator of the LbL-FAD method has been described for the line-by-line detection of anomalous spectra.

In recent years, anomaly detection (AD) has enjoyed a growing interest in hyperspectral data analysis. However, most state-of-the-art detectors need to work with the entire hyperspectral cube, what prevents their use for applications under real-time constraints, especially when the hyperspectral data are collected by push-broom scanners that acquire the hyperspectral images (HSIs) in a line-by-line fashion. In this paper, a Line-by-Line Fast Anomaly Detector for Hyperspectral Imagery (LbL-FAD) is proposed, which is capable of processing each sensed line as soon as it is captured. The LbL-FAD works under the assumption that anomalous pixels cannot be well represented by the background distribution. It uses an orthogonal projection strategy for extracting a set of pixels from the first captured hyperspectral frames, i.e., lines of pixels, that are used for representing the background distribution. Using these pixels, the LbL-FAD proposes a hardware-friendly alternative to compute the orthogonal subspace to that spanned by the selected background samples, making the anomalous pixels better detectable. In addition, the LbL-FAD incorporates an automatic thresholding method which provides line-by-line and real-time binary maps where anomalous targets are segmented from the background. This novelty clearly differentiates the proposed LbL-FAD from the conventional anomaly detectors, which usually are not able to automatically discriminate anomalous pixels from background pixels until the entire image is processed. Several experiments have been carried out using real HSIs collected by different sensors. The obtained results clearly support the benefits of our proposal, both in terms of the accuracy of the detection performance and the computational complexity.

Anomaly detection (AD) is an important technique in hyperspectral data analysis that permits to distinguish rare objects with unknown spectral signatures that are particularly not abundant in a scene. In this paper, a novel algorithm for an accurate detection of anomalies in hyperspectral images with a low computational complexity, named ADALOC2, is proposed. It is based on two main processing stages. First, a set of characteristic pixels that best represent both anomaly and background classes are extracted applying orthogonal projection techniques. Second, the abundance maps associated to these pixels are estimated. Under the assumption that the anomaly class is composed of a scarce group of image pixels, rare targets can be identified from abundance maps characterized by a representation coefficient matrix with a large amount of almost zero elements. Unlike the other algorithms of the state of the art, the ADALOC2algorithm has been specially designed for being efficiently implemented into parallel hardware devices for applications under real-time constraints. To achieve this, the ADALOC2algorithm uses simple and highly parallelized operations, avoiding to perform complex matrix operations such as the computation of an inverse matrix or the extraction of eigenvalues and eigenvectors. An extensive set of simulations using the most representative state-of-the-art AD algorithms and both real and synthetic hyperspectral data sets have been conducted. Moreover, extra assessment metrics apart from classical receiver operating characteristic curves have been defined in order to make deeper comparisons. The obtained results clearly support the benefits of our proposal, both in terms of the accuracy of the detection results and the processing power demanded.

Remote sensing systems equipped with multispectral and hyperspectral sensors are able to capture images of the surface of the Earth at different wavelengths. In these systems, hyperspectral sensors typically provide images with a high spectral resolution but a reduced spatial resolution, while on the contrary, multispectral sensors are able to produce images with a rich spatial resolution but a poor spectral resolution. Due to this reason, different fusion algorithms have been proposed during the last years in order to obtain remotely sensed images with enriched spatial and spectral resolutions by wisely combining the data acquired for the same scene by multispectral and hyperspectral sensors. However, the algorithms so far proposed that are able to obtain fused images with a good spatial and spectral quality require a formidable amount of computationally complex operations that cannot be executed in parallel, which clearly prevent the utilization of these algorithms in applications under real-time constraints in which high-performance parallel-based computing systems are normally required for accelerating the overall process. On the other hand, there are other state-of-the-art algorithms that are capable of fusing these images with a lower computational effort but at the cost of decreasing the quality of the resultant fused image. In this paper, a new algorithm named computationally efficient algorithm for fusing multispectral and hyperspectral images (CoEf-MHI) is proposed in order to obtain a high-quality image from hyperspectral and multispectral images of the same scene with a low computational effort. The proposed CoEf-MHI algorithm is based on incorporating the spatial details of the multispectral image into the hyperspectral image, without introducing spectral distortions. To achieve this goal, the CoEf-MHI algorithm first spatially upsamples, by means of a bilinear interpolation, the input hyperspectral image to the spatial resolution of the input multispectral image, and then, it independently refines each pixel of the resulting image by linearly combining the multispectral and hyperspectral pixels in its neighborhood. The simulations performed in this work with different images demonstrate that our proposal is much more efficient than state-of-the-art approaches, being this efficiency understood as the ratio between the quality of the fused image and the computational effort required to obtain such image.

Spectral unmixing is an important task for remotely sensed hyperspectral data exploitation. It amounts the identification of pure spectral signatures (endmembers) in the data, and the estimation of the abundance of each endmember in each (possibly mixed) pixel. A challenging problem in spectral unmixing is how to determine the number of endmembers in a given scene. For this purpose, many algorithms have been proposed in the recent literature, being the estimation of the Virtual Dimensionality (VD) of the hyperspectral image and the hyperspectral signal subspace estimator (HySime) two of the most popular choices. Unfortunately, the high dimensionality of the hyperspectral data provided by modern sensors as well as the inherent computational complexity clearly make the use of these algorithms prohibitive for applications under real-time or near real-time constraints. Hence, the utilization of high performance computing platforms in order to accelerate the process of unmixing a hyperspectral image becomes mandatory for such scenarios. Reconfigurable hardware solutions such as field programmable gate arrays (FPGAs) have consolidated during the last years as one of the preferred choices for the fast processing of hyperspectral remotely sensed images due to their advantages over other high performance computing systems, such as clusters of computers, multicore processors and/or graphical processing units (GPUs). This paper uncovers two FPGA-based architectures for accelerating the process of estimating the number of endmembers that constitute a hyperspectral image according to the VD and the HySime algorithms. The proposed methods have been implemented on a Virtex-7 XC7VX690T FPGA and tested using real hyperspectral data collected by NASAs Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS) over the Cuprite mining district in Nevada and the World Trade Center in New York. Experimental results demonstrate that the VD implementation exhibits real-time performance while the HySime implementation exhibits near real-time performance. Both implementations significantly outperform a software version, which makes our reconfigurable system appealing for onboard hyperspectral data processing.

Linear spectral unmixing is nowadays an essential tool to analyze remotely sensed hyperspectral images. Although many different contributions have been uncovered during the last two decades, the majority of them are based on dividing the whole process of linearly unmixing a given hyperspectral image into three sequential steps: 1) estimation of the number of endmembers that are present in the hyperspectral image under consideration; 2) extraction of these endmembers from the hyperspectral data set; and 3) calculation of the abundances associated with the endmembers induced in the previous step per each mixed pixel of the image. Although this de facto processing chain has proven to be accurate enough for unmixing most of the images collected by hyperspectral remote sensors, it is also true that it is not exempt of drawbacks, such as the fact that all the possible combinations of algorithms in order to fully unmix a hyperspectral image according to the aforementioned processing chain demand a formidable computational effort, which tends to be higher the better the performance of the designed unmixing chain is. This troublesome issue unfortunately prevents the use of hyperspectral imaging technology in applications under real-time constraints, in which hyperspectral images have to be analyzed in a short period of time. Hence, there is a clear need to face the challenge of fully exploiting the unquestionable benefits of the hyperspectral imaging technology for these applications, but concurrently overcoming the limitations imposed by the computationally complex nature of the processes involved. For this purpose, this paper introduces a novel algorithm named fast algorithm for linearly unmixing hyperspectral images (FUN), which is capable of fully unmixing a hyperspectral image with at least the same accuracy than state-of-the-art approaches while demanding a much lower computational effort, independent of the characteristics of the image under analysis. The FUN algorithm is based on the concept of orthogonal projections and allows performing the estimation of the number of endmembers and their extraction simultaneously, using the modified Gram-Schmidt method. The operations performed by the FUN algorithm are simple and can be easily parallelized. Moreover, this algorithm is able to calculate the abundances using very similar operations, also based on orthogonal projections, which makes it easier to achieve a hardware implementation to perform the entire unmixing process. The benefits of our proposal are demonstrated with a diverse set of artificially generated hyperspectral images and with the well-known AVIRIS Cuprite image, for which the proposed FUN algorithm is able to reduce in a factor of more than 31 times the time required for processing it, while providing a better unmixing performance than traditional methods.

Spectral unmixing is a popular technique for analyzing remotely sensed hyperspectral data sets with subpixel precision. Over the last few years, many algorithms have been developed for each of the main processing steps involved in spectral unmixing (SU) under the LMM assumption: 1) estimation of the number of endmembers; 2) identification of the spectral signatures of the endmembers; and 3) estimation of the abundance of endmembers in the scene. Although this general processing chain has proven to be effective for unmixing certain types of hyperspectral images, it also has some drawbacks. The first one comes from the fact that the output of each stage is the input of the following one, which favors the propagation of errors within the unmixing chain. A second problem is the huge variability of the results obtained when estimating the number of endmembers of a hyperspectral scene with different state-of-the-art algorithms, which influences the rest of the process. A third issue is the computational complexity of the whole process. To address the aforementioned issues, this paper develops a novel negative abundance-oriented SU algorithm that covers, for the first time in the literature, the main steps involved in traditional hyperspectral unmixing chains. The proposed algorithm can also be easily adapted to a scenario in which the number of endmembers is known in advance and two additional variations of the algorithm are provided to deal with high-noise scenarios and to significantly reduce its execution time, respectively. Our experimental results, conducted using both synthetic and real hyperspectral scenes, indicate that the presented method is highly competitive (in terms of both unmixing accuracy and computational performance) with regard to other SU techniques with similar requirements, while providing a fully self-contained unmixing chain without the need for any input parameters.

FPGA-based embedded systems are gaining relevance for implementing a wide range of applications. Part of their success is due to their balanced compromise between performance and flexibility, but also because of their capability for exploiting the dynamic reconfiguration. However, the costly reconfiguration process and the lack of management support have prevented a broader use of the FPGAs. In order to contribute to solve these issues, in this paper we propose a software/hardware dynamic resource management system that combines scheduling and placement tasks, providing a complete management flow for supporting dynamically reconfigurable hardware designs. One of the advantages of the proposed model is the capability for running its scheduling and placement tasks in different nodes, as part of a distributed network. The results of our experiments demonstrate that our placement policy, specially designed for reconfigurable systems, achieves good results, in terms of reusability and performance, compared to other management approaches.

The detection of (moving or static) targets in remotely sensed hyperspectral images often requires real-time responses for swift decisions that depend upon high computing performance of algorithm analysis. The automatic target detection and classification algorithm (ATDCA) has been widely used for this purpose. In this letter, we develop several optimizations for accelerating the computational performance of ATDCA. The first one focuses on the use of the Gram-Schmidt orthogonalization method instead of the orthogonal projection process adopted by the classic algorithm. The second one is focused on the development of a new implementation of the algorithm on commodity graphics processing units (GPUs). The proposed GPU implementation properly exploits the GPU architecture at low level, including shared memory, and provides coalesced accesses to memory that lead to very significant speedup factors, thus taking full advantage of the computational power of GPUs. The GPU implementation is specifically tailored to hyperspectral imagery and the special characteristics of this kind of data, achieving real-time performance of ATDCA for the first time in the literature. The proposed optimizations are evaluated not only in terms of target detection accuracy but also in terms of computational performance using two different GPU architectures by NVIDIA: Tesla C1060 and GeForce GTX 580, taking advantage of the performance of operations in single-precision floating point. Experiments are conducted using hyperspectral data sets collected by three different hyperspectral imaging instruments. These results reveal considerable acceleration factors while retaining the same target detection accuracy for the algorithm.

Hyperspectral image processing represents a valuable tool for remote sensing of the Earth. This fact has led to the inclusion of hyperspectral sensors in different airborne and satellite missions for Earth observation. However, one of the main drawbacks encountered when dealing with hyperspectral images is the huge amount of data to be processed, in particular, when advanced analysis techniques such as spectral unmixing are used. The main contribution of this letter is the introduction of a novel preprocessing (PP) module, called SE2PP, which is based on the integration of spatial and spectral information. The proposed approach can be combined with existing algorithms for endmember extraction, reducing the computational complexity of those algorithms while providing similar figures of accuracy. The key idea behind SE2PP is to identify and select a reduced set of pixels in the hyperspectral image, so that there is no need to process a large amount of them to get accurate spectral unmixing results. Compared to previous approaches based on similar spatial and spatial-spectral PP strategies, SE2PP clearly outperforms their results in terms of accuracy and computation speed, as it is demonstrated with artificial and real hyperspectral images.

Hyperspectral imaging is an important technique in remote sensing which is characterized by high spectral resolutions. With the advent of new hyperspectral remote sensing missions and their increased temporal resolutions, the availability and dimensionality of hyperspectral data is continuously increasing. This demands fast processing solutions that can be used to compress and/or interpret hyperspectral data onboard spacecraft imaging platforms in order to reduce downlink connection requirements and perform a more efficient exploitation of hyperspectral data sets in various applications. Over the last few years, reconfigurable hardware solutions such as field-programmable gate arrays (FPGAs) have been consolidated as the standard choice for onboard remote sensing processing due to their smaller size, weight, and power consumption when compared with other high-performance computing systems, as well as to the availability of more FPGAs with increased tolerance to ionizing radiation in space. Although there have been many literature sources on the use of FPGAs in remote sensing in general and in hyperspectral remote sensing in particular, there is no specific reference discussing the state-of-the-art and future trends of applying this flexible and dynamic technology to such missions. In this work, a necessary first step in this direction is taken by providing an extensive review and discussion of the (current and future) capabilities of reconfigurable hardware and FPGAs in the context of hyperspectral remote sensing missions. The review covers both technological aspects of FPGA hardware and implementation issues, providing two specific case studies in which FPGAs are successfully used to improve the compression and interpretation (through spectral unmixing concepts) of remotely sensed hyperspectral data. Based on the two considered case studies, we also highlight the major challenges to be addressed in the near future in this emerging and fast growing research area.

Spectral unmixing is an important tast for remotely sensed hyperspectral imaging. Spectral unmixing algorithms can be computationally expensive. In this paper, we develop two efficient implementations of a full hyperspectral unmixing chain on two different kinds of high performance computing architectures: graphics processing units (GPUs) and multi-core processors. These platforms are inter-compared in the context of hyperspectral unmixing. Our experimental results reveal that both GPUs and multi-core processors can provide real-time unmixing performance when properly implemented in this kind of parallel platforms.

Endmember extraction represents one of the most challenging aspects of hyperspectral image processing. In this letter, a new algorithm for endmember extraction, named modified vertex component analysis (MVCA), is presented. This new technique outperforms the popular vertex component analysis (VCA) by applying a low-complexity orthogonalization method and by utilizing integer instead of floating-point arithmetic when dealing with hyperspectral data. The feasibility of this technique is demonstrated by comparing its performance with VCA on synthetic mixtures as well as on the well-known Cuprite hyperspectral image. MVCA shows promising results in terms of much lower computational complexity, still reproducing similar endmember accuracy than its original counterpart. Moreover, the features of this algorithm combined with state-of-the-art hardware implementations qualify MVCA as a good potential candidate for all those applications in which real time is a must.

Systems relying on fixed hardware components with a static level of parallelism can suffer from an under use of logical resources, since they have to be designed for the worst-case scenario. This problem is especially important in video applications due to the emergence of new flexible standards, like Scalable Video Coding (SVC), which offer several levels of scalability. In this paper, Dynamic and Partial Reconfiguration (DPR) of modern FPGAs is used to achieve run-time variable parallelism, by using scalable architectures where the size can be adapted at run-time. Based on this proposal, a scalable Deblocking Filter core (DF), compliant with the H.264/AVC and SVC standards has been designed. This scalable DF allows run-time addition or removal of computational units working in parallel. Scalability is offered together with a scalable parallelization strategy at the macro block (MB) level, such that when the size of the architecture changes, MB filtering order is modified accordingly.

A highly parallel and scalable Deblocking Filter (DF) hardware architecture for H.264/AVC and SVC video codecs is presented in this paper. The proposed architecture mainly consists on a coarse grain systolic array obtained by replicating a unique and homogeneous Functional Unit (FU), in which a whole Deblocking-Filter unit is implemented. The proposal is also based on a novel macroblock-level parallelization strategy of the filtering algorithm which improves the final performance by exploiting specific data dependences. This way communication overhead is reduced and a more intensive parallelism in comparison with the existing state-of-the-art solutions is obtained. Furthermore, the architecture is completely flexible, since the level of parallelism can be changed, according to the application requirements. The design has been implemented in a Virtex-5 FPGA, and it allows filtering 4CIF (704 × 576 pixels @30 fps) video sequences in real-time at frequencies lower than 10.16 Mhz.

Onboard processing of remotely sensed hyper spectral data is a highly desirable goal in many applications. For this purpose, compact reconfigurable hardware modules such as field programmable gate arrays (FPGAs) are widely used. In this paper, we develop a new implementation of an automatic target generation process (ATGP) for hyper spectral images. Our implementation is based on a design methodology that starts from a high-level description in Matlab (or alternative C/C++) and obtains a register transfer level (RTL) description that can be ported to FPGAs. In order to validate our new implementation, we develop a quantitative and comparative study using two different FPGA architectures: Xilinx Virtex-5 and Altera Stratix-III Altera. Experimental results have been obtained in the context of a real application focused on the detection of mineral components over the Cup rite mining district (Nevada), using hyper spectral data collected by NASA's Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS). Our experimental results indicate that the proposed implementation can achieve peak frequency designs above 200MHz in the considered FPGAs, in addition to satisfactory results in terms of target detection accuracy and parallel performance. This represents a step forward towards the design of real-time onboard implementations of hyper spectral image analysis algorithms.

Hardware sharing can be used to reduce the area and the power dissipation of a design. This is of particular interest in the field of image and video compression, where an encoder must deal with different design tradeoffs depending on the characteristics of the signal to be encoded and the constraints imposed by the users. This paper introduces a novel methodology for exploring the design space based on the amount of hardware sharing between different functional blocks, giving as a result a set of feasible solutions which are broad in terms of hardware cost and throughput capabilities. The proposed approach, inspired by the notion of a partition in set theory, has been applied to optimize and to evaluate the sharing alternatives of a group of image and video compression key computational kernels when mapped onto a Xilinx Virtex-5 FPGA.

Nowadays, images are employed in several areas of medicine for early diagnosis. In this sense, the industry provides accurate models to obtain, for example, X-ray and cardiology images of high resolution. However, other images, such as those related to pathological anatomy present in many situations poor quality, which complicates the diagnostic process. This work is focused on the quality enhancement of this type of images through a system based on super-resolution techniques. The results show that the proposed methodology can help medical specialists in the diagnostic of several pathologies.

Deblocking filtering represents one of the most compute intensive tasks in an H.264/AVC standard video decoder due to its demanding memory accesses and irregular data flow. For these reasons, an efficient implementation poses big challenges, especially for programmable platforms. In this sense, the mapping of this decoder's functionality onto a C-programmable coarse-grained reconfigurable architecture named ADRES (architecture for dynamically reconfigurable embedded systems) is presented in this paper, including results from the evaluation of different topologies. The results obtained show a considerable reduction in the number of cycles and memory accesses needed to perform the filtering as well as an increase in the degree of instruction parallelism (ILP) when compared with an implementation on a very long instruction word (VLIW) dedicated processor. This demonstrates that high ILP is achievable on the ADRES even for irregular, data-dependent kernels

Motion estimation is the most critical process in video coding systems. First of all, it has a definitive impact on the rate-distortion performance given by the video encoder. Secondly, it is the most computationally intensive process within the encoding loop. For these reasons, the design of high-performance low-cost motion estimators is a crucial task in the video compression field. An adaptive cost block matching (ACBM) motion estimation technique is presented in this paper, featuring an excellent tradeoff between the quality of the reconstructed video sequences and the computational effort. Simulation results demonstrate that the ACBM algorithm achieves a slightly better rate-distortion performance than the one given by the well-known full search algorithm block matching algorithm with reductions of up to 95% in the computational load.