Podcast transcripts are long spoken documents of conversational dialogue. Challenging to summarize, podcasts cover a diverse range of topics, vary in length, and have uniquely different linguistic styles. Previous studies in podcast summarization have generated short, concise dialogue summaries. In contrast, we propose a method to generate long fine-grained summaries, which describe details of sub-topic narratives. Leveraging a readability formula, we curate a data subset to train a long sequence transformer for abstractive summarization. Through text segmentation, we filter the evaluation data and exclude specific segments of text. We apply the model to segmented data, producing different types of fine grained summaries. We show that appropriate filtering creates comparable results on ROUGE and serves as an alternative method to truncation. Experiments show our model outperforms previous studies on the Spotify podcast dataset when tasked with generating longer sequences of text.

Convolutional neural networks (CNNs) learn filters in order to capture local correlation patterns in feature space. We propose to learn these filters as combinations of preset spectral filters defined by the Discrete Cosine Transform (DCT). Our proposed DCT-based harmonic blocks replace conventional convolutional layers to produce partially or fully harmonic versions of new or existing CNN architectures. Using DCT energy compaction properties, we demonstrate how the harmonic networks can be efficiently compressed by truncating high-frequency information in harmonic blocks thanks to the redundancies in the spectral domain. We report extensive experimental validation demonstrating benefits of the introduction of harmonic blocks into state-of-the-art CNN models in image classification, object detection and semantic segmentation applications.

We show how parameter redundancy in Convolutional Neural Network (CNN) filters can be effectively reduced by pruning in spectral domain. Specifically, the representation extracted via Discrete Co-sine Transform (DCT) is more conducive for pruning than the original space. By relying on a combination of weight tensor reshaping and reordering we achieve high levels of layer compression with just minor accuracy loss. Our approach is applied to compress pre-trained CNNs and we show that minor additional fine-tuning allows our method to recover the original model performance after a significant parameter reduction. We validate our approach on ResNet-50 and MobileNet-V2 architectures for ImageNet classification task.

Blendshape facial rigs are used extensively in the industry for facial animation of virtual humans. However, storing and manipulating large numbers of facial meshes (blendshapes) is costly in terms of memory and computation for gaming applications. Blendshape rigs are comprised of sets of semantically-meaningful expressions, which govern how expressive the character will be, often based on Action Units from the Facial Action Coding System (FACS). However, the relative perceptual importance of blendshapes has not yet been investigated. Research in Psychology and Neuroscience has shown that our brains process faces differently than other objects so we postulate that the perception of facial expressions will be feature-dependent rather than based purely on the amount of movement required to make the expression. Therefore, we believe that perception of blendshape visibility will not be reliably predicted by numerical calculations of the difference between the expression and the neutral mesh. In this paper, we explore the noticeability of blendshapes under different activation levels, and present new perceptually-based models to predict perceptual importance of blendshapes. The models predict visibility based on commonly-used geometry and image-based metrics.

Blendshape facial rigs are used extensively in the industry for facial animation of virtual humans. However, storing and manipulating large numbers of facial meshes is costly in terms of memory and computation for gaming applications, yet the relative perceptual importance of blendshapes has not yet been investigated. Research in Psychology and Neuroscience has shown that our brains process faces differently than other objects, so we postulate that the perception of facial expressions will be feature-dependent rather than based purely on the amount of movement required to make the expression. In this paper, we explore the noticeability of blendshapes under different activation levels, and present new perceptually based models to predict perceptual importance of blendshapes. The models predict visibility based on commonly-used geometry and image-based metrics.

We propose a new method with Nadaraya-Watson that maps one N-dimensional distribution to another, taking into account available information about correspondences. We extend the 2D/3D problem to higher dimensions by encoding overlapping neighborhoods of data points and solve the high-dimensional problem in 1D space using an iterative projection approach. To show the potentials of this mapping, we apply it to colour transfer between two images that exhibit overlapped scenes. Experiments show quantitative and qualitative improvements over the previous state of the art colour transfer methods.

Next generation of embedded Information and Communication Technology (ICT) systems are interconnected and collaborative systems able to perform autonomous tasks. The remarkable expansion of the embedded ICT market, together with the rise and breakthroughs of Artificial Intelligence (AI), have put the focus on the Edge as it stands as one of the keys for the next technological revolution: the seamless integration of AI in our daily life. However, training and deployment of custom AI solutions on embedded devices require a fine-grained integration of data, algorithms, and tools to achieve high accuracy and overcome functional and non-functional requirements. Such integration requires a high level of expertise that becomes a real bottleneck for small and medium enterprises wanting to deploy AI solutions on the Edge , which, ultimately, slows down the adoption of AI on applications in our daily life. In this work, we present a modular AI pipeline as an integrating framework to bring data, algorithms, and deployment tools together. By removing the integration barriers and lowering the required expertise, we can interconnect the different stages of particular tools and provide a modular end-to-end development of AI products for embedded devices. Our AI pipeline consists of four modular main steps: (i) data ingestion, (ii) model training, (iii) deployment optimization, and (iv) the IoT hub integration. To show the effectiveness of our pipeline, we provide examples of different AI applications during each of the steps. Besides, we integrate our deployment framework, Low-Power Deep Neural Network (LPDNN), into the AI pipeline and present its lightweight architecture and deployment capabilities for embedded devices. Finally, we demonstrate the results of the AI pipeline by showing the deployment of several AI applications such as keyword spotting, image classification, and object detection on a set of well-known embedded platforms, where LPDNN consistently outperforms all other popular deployment frameworks.

Abundant image and sensory data collected over the last decades represents an invaluable source of information for cataloging and monitoring of the environment. Fusion of heterogeneous data sources is a challenging but promising tool to efficiently leverage such information. In this work we propose a pipeline for automatic detection and geolocation of recurring stationary objects deployed on fusion scenario of street level imagery and LiDAR point cloud data. The objects are geolocated coherently using a fusion procedure formalized as a Markov random field problem. This allows us to efficiently combine information from object segmentation, triangulation, monocular depth estimation and position matching with LiDAR data. The proposed fusion approach produces object mappings robust to scenes reporting multiple object instances. We introduce a new challenging dataset of over 200 traffic lights in Dublin city centre and demonstrate high performance of the proposed methodology and its capacity to perform multi -sensor data fusion.

Abstract We propose to automatically populate geo‐located virtual cities by harvesting and analyzing online contents shared on social networks and websites. We show how pose and motion paths of agents can be realistically rendered using information gathered from social media. 3D cities are automatically generated using open‐source information available online. To provide our final rendering of both static and dynamic urban scenes, we use Unreal game engine.

We compare the objective functions used by GR2T [1] and the L2E estimator [2] that have both been proposed for robust parameter estimation. We show their similarity when estimating location parameters. Of particular interest is their ability for dealing with the scale parameter that is often unknown and acts as a nuisance parameter. Both techniques are tested experimentally for regression (e.g. to find patterns such as line and circle in noisy datasets) and for registration between datasets.

Visual saliency (VS), which refers to the study of the behaviour or perception of the human vision system, is usually captured using eye tracking technologies on a (statistically) representative set of participants watching visual media on a screen. Using eye tracking technologies to capture the visual behaviour of a set of candidates is a long, expensive and tedious experiment to set. Artificial intelligence can be used to replicate this human behaviour and several 2D and 3D visual saliency algorithms (VSAs) have been proposed [Wang et al. 2013; Zdziarski and Dahyot 2013]. These algorithms attempt to reproduce human visual perception behaviour and have assisted in applications such as video content creation, retargetting and summarisation. Automatically replicating viewing behaviour is easier, less cumbersome and cheaper than the manual option presented by eye trackers.

We introduce a novel skeleton extraction algorithm in binary and gray-scale images, based on the anisotropic heat diffusion analogy. We propose to diffuse image in the dominance of direction normal to the feature boundaries and also allow tangential diffusion to contribute slightly. The proposed anisotropic diffusion provides a high quality medial function in the image, since it removes noise and preserves prominent curvatures of the shape along the level-sets (skeleton locations). Then the skeleton strength map, which provides the likelihood to be a skeleton point, is obtained by computing the mean curvature of level-sets. The overall process is completed by non-maxima suppression and hysteresis thresholding to obtain thin and binary skeleton. Results indicate that this approach has advantages in handling noise in the image and in obtaining smooth shape skeleton because of the directional averaging inherent of our new anisotropic heat flow.

In this article, a novel method to accurately estimate 3D surface of objects of interest is proposed. Each ray projected from 2D image plane to 3D space is modelled with the Gaussian kernel function. Then a mean shift algorithm with an annealing scheme is used to find maximums of the probability density function and recovers the 3D surface. Experimental results show that our method is more accurate to estimate 3D surface than the Radon transform-based approach.

Example-based texture synthesis algorithms generate novel texture images from example data. A popular hierarchical pixel-based approach uses spatial jitter to introduce diversity, at the risk of breaking coarse structure beyond repair. We propose a multiscale descriptor that enables appearance-space jitter, which retains structure. This idea enables repurposing of existing texture synthesis implementations for a qualitatively different problem statement and class of inputs: generating hybrids of structured images.

The Standard Hough Transform is a popular method in image processing and is traditionally estimated using histograms. Densities modeled with histograms in high dimensional space and/or with few observations, can be very sparse and highly demanding in memory. In this paper, we propose first to extend the formulation to continuous kernel estimates. Second, when dependencies in between variables are well taken into account, the estimated density is also robust to noise and insensitive to the choice of the origin of the spatial coordinates. Finally, our new statistical framework is unsupervised (all needed parameters are automatically estimated) and flexible (priors can easily be attached to the observations). We show experimentally that our new modeling encodes better the alignment content of images.

We have introduced the statistical Hough transform that extends the standard Hough transform by using a kernel mixture as a robust alternative to the 2 dimensional accumulator histogram. This work develops further this framework by proposing a Bayesian classification scheme to associate the spatial coordinates (x, y) to one particular class defined in the Hough space (¿, ¿). In a first step, we segment the Hough space into meaningful classes. Then using the inverse Radon transform, we backproject the different classes into the image space. We illustrate our approach on a synthetic image and on real images.

This article proposes an original method to estimate a continuous transformation that maps one N-dimensional distribution to another. The method is iterative, non-linear, and is shown to converge. Only 1D marginal distribution is used in the estimation process, hence involving low computation costs. As an illustration this mapping is applied to color transfer between two images of different contents. The paper also serves as a central focal point for collecting together the research activity in this area and relating it to the important problem of automated color grading

This paper presents a probabilistic framework for off-line multiple object tracking. At each timestep, a small set of deterministic candidates is generated which is guaranteed to contain the correct solution. Tracking an object within video then becomes possible using the Viterbi algorithm. In contrast with particle filter methods where candidates are numerous and random, the proposed algorithm involves a few candidates and results in a deterministic solution. Moreover, we consider here off-line applications where past and future information is exploited. This paper shows that, although basic and very simple, this candidate selection allows the solution of many tracking problems in different real-world applications and offers a good alternative to particle filter methods for off-line applications.

This paper investigates the semantic analysis of broadcast tennis footage. We consider the spatio-temporal behaviour of an object in the footage as being the embodiment of a semantic event. This object is tracked using a colour based particle filter. The video syntax and audio features are used to help delineate the temporal boundaries of these events. For broadcast tennis footage, the system firstly parses the video sequence based on the geometry of the content in view and classifies the clip as a particular view type. The temporal behaviour of the serving player is modelled using a HMM. As a result, each model is representative of a particular semantic episode. Events are then summarised using a number of synthesised keyframes.

In this paper, we investigate the retrieval of dynamic events that occur in broadcast sports footage. Dynamic events in sports are important in so far as they are related to the game semantics. Thus far, the temporal interleaving of camera views has been used to infer these types of events. We propose the use of the spatio-temporal behaviour of an object in the footage as an embodiment of a semantic event. This is accomplished by modeling the evolution of the position of the object with a hidden Markov model (HMM). Snooker is used as an example for the purpose of this research. The system firstly parses the video sequence based on the geometry of the content in the camera view and classifies the footage as a particular view type. Secondly, we consider the relative position of the white ball on the snooker table over the duration of a clip to embody semantic events. A colour based particle filter is employed to robustly track the snooker balls. The temporal behaviour of the white ball is modeled using a HMM where each model is representative of a particular semantic episode. Upon collision of the white ball with another coloured ball, a separate track is instantiated.

This paper presents a robust method to estimate the unknown standard deviation of a centred normal distribution from a mixture density. This method is applied to different signal processing problems. The first one concerns silence segmentation from audio data. The second application deals with colour class parameter extraction. In this later case, the mean is also estimated from the observations.

In recent years, there has been increasing work in the area of content retrieval for sports. The idea is generally to extract important events or create summaries to allow personalisation of the media stream. While previous work in sports analysis has employed either the audio or video stream to achieve some goal, there is little work that explores how much can be achieved by combining the two streams. This paper combines both audio and image features to identify the key episode in tennis broadcasts. The image feature is based on image moments and is able to capture the essence of scene geometry without recourse to 3D modelling. The audio feature uses PCA to identify the sound of the ball hitting the racket. The features are modelled as stochastic processes and the work combines the features using a likelihood approach. The results show that combining the features yields a much more robust system than using the features separately.

Change detection in image sequences has mainly focused on the recovery of moving objects when the viewing system is static, or on the detection of simple production effects such as video shot boundaries or scene transitions. Camera motion is usually handled by the compensation of dominant motion, using motion estimation and segmentation schemes. We propose a novel statistical change detection method able to handle more complex events such as entering or exiting objects, or changes in object appearance, when the camera is moving. Temporal changes of objects are captured by analyzing the statistics of successive images. Considering an appropriate choice of image features, we show how it is possible to extract the statistics of changing objects from a pair of successive image histograms. Changing objects are then located by statistical backprojection techniques. The method is completely unsupervised and does not require any motion estimation or motion compensation. It is illustrated here on real world road scenes exhibiting large camera motion.