In the past few years, there has been a rising demand for monocular vehicle 6D pose estimation in traffic video, due to its applicability in emerging domains such as smart mobility and intelligent transportation systems. However, most of the existing approaches are image-based, which results in undesirable temporal pose inconsistencies of vehicles between consecutive frames when applied to video sequences. In this work, we present a Kalman filter-based post-processing method, named MonoKalman, that enhances the temporal consistencies of the 6D pose estimates of vehicles in traffic video. We compare our method with a state-of-the-art 6D pose estimation method on synthetic video data of traffic scenery. The experimental results indicate that our MonoKalman significantly outperforms the image-based baseline method and effectively reduces temporal pose artifacts, ensuring a more coherent and stable representation of 6D vehicle temporal poses in traffic video. To more effectively demonstrate MonoKalman’s enhancements over the baseline model, we design a graphical user interface. This interface offers users insights through detailed quantitative metrics and dynamic visualizations, allowing them to conduct and customize their experiments.

Efficiently visualizing Spatial-Temporal traffic data plays an important role nowadays in traffic monitoring. Interactive dashboards offering effective visualizations of spatial-temporal traffic data play a more prominent role in traffic monitoring. In this paper, we introduce a dashboard for visualizing traffic data. Specifically, our dashboard integrates spatial-temporal components for the time-series traffic data of Brussels, which is the first GNN-based traffic demonstration tool for Brussels. Furthermore, we provide an interface for displaying traffic prediction of deep-learning-based Spatial-Temporal Graph Neural Networks (STGNNs), which have demonstrated state of the art performance in Intelligent Transpiration Systems (ITS). In addition, we demonstrate two real-world use cases by using the proposed dashboard which provides the potential for a future tool to achieve intelligent transportation management.

Multimodal, alias, guided, image restoration is the reconstruction of a degraded image from a target modality with the aid of a high quality image from another modality. A similar task is image fusion; it refers to merging images from different modalities into a composite image. Traditional approaches for multimodal image restoration and fusion include analytical methods that are computationally expensive at inference time. Recently developed deep learning methods have shown a great performance at a reduced computational cost; however, since these methods do not incorporate prior knowledge about the problem at hand, they result in a “black box” model, that is, one can hardly say what the model has learned. In this paper, we formulate multimodal image restoration and fusion as a coupled convolutional sparse coding problem, and adopt the Method of Multipliers (MM) for its solution. Then, we use the MM-based solution to design a convolutional neural network (CNN) encoder that follows the principle of deep unfolding. To address multimodal image restoration and fusion, we design two multimodal models which employ the proposed encoder followed by an appropriately designed decoder that maps the learned representations to the desired output. Unlike most existing deep learning designs comprising multiple encoding branches followed by a concatenation or a linear combination fusion block, the proposed design provides an efficient and structured way to fuse information at different stages of the network, providing representations that can lead to accurate image reconstruction. The proposed models are applied to three image restoration tasks, as well as two image fusion tasks. Quantitative and qualitative comparisons against various state-of-the-art analytical and deep learning methods corroborate the superior performance of the proposed framework.

Abstract Deep learning has transformed the field of data analysis by dramatically improving the state of the art in various classification and prediction tasks, especially in the area of computer vision. In biomedical engineering, a lot of new work is directed toward surface electromyography (sEMG)-based gesture recognition, often addressed as an image classification problem using convolutional neural networks (CNNs). In this paper, we utilize the Hilbert space-filling curve for the generation of image representations of sEMG signals, which allows the application of typical image processing pipelines such as CNNs on sequence data. The proposed method is evaluated on different state-of-the-art network architectures and yields a significant classification improvement over the approach without the Hilbert curve. Additionally, we develop a new network architecture (MSHilbNet) that takes advantage of multiple scales of an initial Hilbert curve representation and achieves equal performance with fewer convolutional layers.

Joint image super-resolution (SR) refers to the reconstruction of a high-resolution image from its low-resolution version with the aid of a high-resolution image from another modality. Inspired by the recent success of recurrent neural networks in single image SR, we propose a novel multimodal recurrent convolutional neural network with coupled sparse priors for joint image SR. Our network fuses representations of the two image modalities at input layers using a learned multimodal convolutional sparse coding network. Additional recurrent convolutional stages are performed to further learn the mapping between the input modalities and the desired highresolution estimate. We apply the proposed network to the tasks of near-infrared image SR and multi-spectral image SR using RGB images as the guidance modality. Experimental results show the superior performance of the proposed multimodal recurrent convolutional network against several state-of-the-art single-modal and multimodal image SR methods.

Sparse representation based methods have demonstrated their superior performance in target detection tasks compared to more traditional approaches such as matched subspace detectors and adaptive subspace detectors. However, the existing sparsity-based target detection methods were mostly formulated for and validated on a single imaging modality (sometimes with multiple spectral bands). In many application domains, including art investigation, multimodal data, acquired by different sensors are readily available, and yet, efficient processing techniques for such data are still scarce. In this paper, we propose a sparsity-based multimodal target detection method that processes jointly the information from multiple imaging modalities in a kernel feature space, and making use of the spatial context. We develop our target detector such to be robust to errors in labelled data, which is especially important in applications like digital painting analysis, where pixel-wise manual annotations are unreliable. We apply the proposed method to a challenging application of paint loss detection in master paintings and we demonstrate its effectiveness on a case study with multimodal acquisitions of the Ghent Altarpiece.

The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-of-the-art methods.

In recent years, the successful application of Deep Learning methods to classification problems has had a huge impact in many domains. In biomedical engineering, the problem of gesture recognition based on electromyography is often addressed as an image classification problem using Convolutional Neural Networks. In this paper, we approach electromyography-based hand gesture recognition as a sequence classification problem using Temporal Convolutional Networks. The proposed network yields an improvement in gesture recognition of almost 5% to the state of the art reported in the literature, whereas the analysis helps in understanding the limitations of the model and exploring new ways to improve its performance.

The success of deep learning in various tasks, including solving inverse problems, has triggered the need for designing deep neural networks that incorporate domain knowledge. In this paper, we design a multimodal deep learning architecture for guided image super-resolution, which refers to the problem of super-resolving a low-resolution image with the aid of a high-resolution image of another modality. The proposed architecture is based on a novel deep learning model, obtained by unfolding a proximal method that solves the problem of convolutional sparse coding with side information. We applied the proposed architecture to super-resolve near-infrared images using RGB images as side information. Experimental results report average PSNR gains of up to 2.85 dB against state-of-the-art multimodal deep learning and sparse coding models.

In this paper, a novel approach to encode lenslet (LL) images is proposed. The method departs from traditional block-based coding structures and employs a hexagonal-shaped pixel cluster, called macro-pixel, as an elementary coding unit. A novel prediction mode based on dictionary learning is proposed, whereby macro-pixels are represented by a sparse linear combination of atoms from a generic dictionary. Additionally, an optimized linear prediction mode and a directional prediction mode specifically designed for macro-pixels are proposed. Rate-distortion optimization is utilized to select the best intra prediction mode for each macro-pixel. Experimental results on the light field image data set show that the proposed coding system outperforms HEVC and the state-of-the-art in LL image coding with an average peak signal to noise ratio gain of 3.33 and 1.41 dB, respectively, and with rate savings of 67.13% and 34.30%, respectively.

Microcalcifications (MCs) are small calcium deposits which are often present in breast tissue. Although they may be an early indicator of breast cancer, they are often present in healthy women. Judging on the malignancy of MCs based on their appearance on mammograms is a challenging task for radiologists as mammography projects a 3D object onto a 2D plane, as a consequence, superposition of tissue may alter their appearance. Although Computer Aided Detection and Diagnosis (CAD) systems have been developed to assist radiologists in detecting and characterizing MCs, they mainly focus on analyzing MCs’ spatial configuration rather than their individual characteristics. In this study, we propose a CAD system for the characterization of individual MCs, whose shape is visualized in 3D by scanning breast tissue with micro-CT, a high resolution 3D imaging modality. By calculating a large amount of radiomic features we are able to classify MCs as benign or malignant with 75.88% accuracy, 62.13% sensitivity and 86.39% specificity, outperforming the state of the art and proving that there is a difference regarding the shape and the texture of individual MCs. Although this methodology is not applicable in vivo, its potential for use in intra operative imaging is high, as it can reduce the waiting time between tissue extraction and result of the anatomopathological investigation, facilitating and accelerating the procedure.

Predicting the geographical location of users on social networks like Twitter is an active research topic with plenty of methods proposed so far. Most of the existing work follows either a content-based or a network-based approach. The former is based on user-generated content while the latter exploits the structure of the network of users. In this paper, we propose a more generic approach, which incorporates not only both content-based and network-based features, but also other available information into a unified model. Our approach, named Multi-Entry Neural Network (MENET), leverages the latest advances in deep learning and multiview learning. A realization of MENET with textual, network and metadata features results in an effective method for Twitter user geolocation, achieving the state of the art on two well-known datasets.

Light field images can be conveniently captured by consumer-level plenoptic cameras. However, as the resulting data rates are very high, providing efficient compression for this type of data is of critical importance. This remains an open problem which has recently attracted a lot of attention from the coding community. State-of-the-art compression systems prove to be inefficient when directly applied on this type of data due to the inherent spatial discontinuities in light field images. In this paper, a novel intra-prediction method for disk-shaped pixel clusters is proposed. An L1 minimization of the prediction residuals is performed followed by clustering of the predictors, leading to an optimized set of predictors for the macro-pixels. Furthermore, directional intra-prediction modes based on HEVC are devised for the macro-pixels. Experimental results obtained on the EPFL light field image dataset demonstrate that the proposed coding scheme yields an average of 3.22 dB and 1.45 dB gain in PSNR, and 59.6% and 30.88% average rate savings compared to HEVC and the state-of-the-art in light field image coding respectively.

We address the removal of canvas artifacts from high-resolution digital photographs and X-ray images of paintings on canvas. Both imaging modalities are common investigative tools in art history and art conservation. Canvas artifacts manifest themselves very differently according to the acquisition modality; they can hamper the visual reading of the painting by art experts, for instance, in preparing a restoration campaign. Computer-aided canvas removal is desirable for restorers when the painting on canvas they are preparing to restore has acquired over the years a much more salient texture. We propose a new algorithm that combines a cartoon-texture decomposition method with adaptive multiscale thresholding in the frequency domain to isolate and suppress the canvas components. To illustrate the strength of the proposed method, we provide various examples, for acquisitions in both imaging modalities, for paintings with different types of canvas and from different periods. The proposed algorithm outperforms previous methods proposed for visual photographs such as morphological component analysis and Wiener filtering and it also works for the digital removal of canvas artifacts in X-ray images.

In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. In this problem, the X-ray signals to be separated have similar morphological characteristics, which brings previous source separation methods to their limits. Our solution is to use photographs taken from the front-and back-side of the panel to drive the separation process. The crux of our approach relies on the coupling of the two imaging modalities (photographs and X-rays) using a novel coupled dictionary learning framework able to capture both common and disparate features across the modalities using parsimonious representations; the common component captures features shared by the multi-modal images, whereas the innovation component captures modality-specific information. As such, our model enables the formulation of appropriately regularized convex optimization procedures that lead to the accurate separation of the X-rays. Our dictionary learning framework can be tailored both to a single- and a multi-scale framework, with the latter leading to a significant performance improvement. Moreover, to improve further on the visual quality of the separated images, we propose to train coupled dictionaries that ignore certain parts of the painting corresponding to craquelure. Experimentation on synthetic and real data - taken from digital acquisition of the Ghent Altarpiece (1432) - confirms the superiority of our method against the state-of-the-art morphological component analysis technique that uses either fixed or trained dictionaries to perform image separation.

In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. Unlike prior source separation methods, which are based on statistical or structural incoherence of the sources, we use visual images taken from the front- and back-side of the panel to drive the separation process. The coupling of the two imaging modalities is achieved via a new multi-scale dictionary learning method. Experimental results demonstrate that our method succeeds in the discrimination of the sources, while state-of-the-art methods fail to do so.

The advent of consumer-level plenoptic cameras has sparkled the interest towards the design of efficient compression techniques for light field images. State-of-the-art compression systems such as HEVC prove to be inefficient when directly applied on this type of data due to the inherent spatial discontinuities among neighboring microlens images. In this paper, a novel light field image compression system is proposed. The disk-shaped pixel clusters corresponding to each microlens in the light field image are efficiently predicted based on the neighboring disks. In this context, an optimized linear prediction design based on L1 minimization of the residuals is proposed. K-means clustering is employed on training data in order to determine the optimized set of predictors. The experimental results on an extensive set of light field images demonstrate that the proposed coding scheme yields an average of 2.93 dB and 3.22 dB gain in PSNR, and 52.67% and 57.27% average rate savings compared to HEVC and JPEG2000 respectively.

We propose an algorithm that removes the visually unpleasant effects of cradling in X-ray images of panel paintings, with the goal of improving the X-ray image readability by art experts. The algorithm consists of three stages. In the first stage the location of the cradle is detected automatically and the grayscale inconsistency, caused by the thickness of the cradle, is corrected. In a second stage we use a method called morphological component analysis to separate the X-ray image into a so-called cartoon part and a texture part, where the latter contains mostly the wood grain from both the panel and the cradling. The algorithm next learns a Bayesian factor model that distinguishes between the texture patterns that originate from the cradle and those from other components such as the panel and/or the painting on the panel surface, and finally uses this to remove the textures associated with the cradle. We apply the algorithm to a number of historically important paintings on panel. We also show how it can be used to digitally remove stretcher artifacts from X-rays of paintings on canvas. We compare our results with those obtained manually by best current practices in art conservation as well as on a ground truth dataset, consisting of X-ray images of a painting before and after removal of the physically attached cradle.

We introduce an algorithm that removes the deleterious effect of cradling on X-ray images of paintings on wooden panels. The algorithm consists of a three stage procedure. Firstly, the cradled regions are located automatically. The second step consists of separating the X-ray image into a textural and image component. In the last step the algorithm learns to distinguish between the texture caused by the wooden cradle and the texture belonging to the original painted wooden panel. The results obtained with our method are compared with those obtained manually by best current practice.

Objective characterization of jewels in paintings, especially pearls, has been a long lasting challenge for art historians. The way an artist painted pearls reflects his ability to observing nature and his knowledge of contemporary optical theory. Moreover, the painterly execution may also be considered as an individual characteristic useful in distinguishing hands. In this work, we propose a set of image analysis techniques to analyze and measure spatial characteristics of the digital images of pearls, all relying on the so called spatiogram image representation. Our experimental results demonstrate good correlation between the new metrics and the visually observed image features, and also capture the degree of realism of the visual appearance in the painting. In that sense, these results set the basis in creating a practical tool for art historical attribution and give strong motivation for further investigations in this direction.