We propose a method to address audio-visual target speaker enhancement in multi-talker environments using eventdriven cameras. State of the art audio-visual speech separation methods shows that crucial information is the movement of the facial landmarks related to speech production. However, all approaches proposed so far work offline, using frame-based video input, making it difficult to process an audio-visual signal with low latency, for online applications. In order to overcome this limitation, we propose the use of event-driven cameras and exploit compression, high temporal resolution and low latency, for low cost and low latency motion feature extraction, going towards online embedded audio-visual speech processing. We use the event-driven optical flow estimation of the facial landmarks as input to a stacked Bidirectional LSTM trained to predict an Ideal Amplitude Mask that is then used to filter the noisy audio, to obtain the audio signal of the target speaker. The presented approach performs almost on pair with the frame- based approach, with very low latency and computational cost.

Fast, non-linear trajectories have been shown to be more accurately visually measured, and hence predicted, when sampled spatially (that is when the target position changes) rather than temporally, i.e. at a fixed-rate as in traditional frame-based cameras. Event-cameras, with their asynchronous, low latency information stream, allow for spatial sampling with very high temporal resolution, improving the quality of the data and the accuracy of post-processing operations. This paper investigates the use of Long Short-Term Memory (LSTM) networks with event-cameras spatial sampling for trajectory prediction. We show the benefit of using an Encoder-Decoder architecture over parameterised models for regression on event-based human-to-robot handover trajectories. In particular, we exploit the temporal information associated to the events stream to predict not only the incoming spatial trajectory points, but also when these will occur in time. After having studied the proper LSTM input/output sequence length, the network performance are compared to other regression models. Then, prediction behavior and computational time are analysed for the proposed method. We carry out the experiment using an iCub robot equipped with event-cameras, addressing the problem from the robot perspective.

This article proposes a new spike encoding and decoding algorithm for analog data. The algorithm uses the pulsewidth modulation principles to achieve a high reconstruction accuracy of the signal, along with a high level of data compression. Two benchmark data sets are used to illustrate the method: stock index time series and human voice data. Applications of the method for spiking neural network (SNN) modeling and neuromorphic implementations are discussed. The proposed method would allow the development of new applications of SNNs as regression techniques for predictive time-series modeling.