Real-time monitoring of ECG signals in wearable and implanted edge devices, such as smart watches, ILRs, pace-makers etc. is essential for early clinical intervention in Cardio-Vascular illnesses. Existing deep learning-based algorithms are not suitable for such battery-powered, low-power, low-memory devices. In this paper, a new peak-based Spiking Hamilton (SH) encoder is proposed and its performance for classifying Atrial Fibrillation (AF) patterns in ECG signal is tested and compared with three existing encoders by deploying reservoir-based SNN and a feed-forward SNN and on five well known ECG dataset. Feed-forward SNN, coupled with proposed SH encoder, achieves best performance while requiring the least amount of computational effort and power. This system running on Neuromorphic Computing (NC) platforms could therefore be a good option for classifying ECG patterns at the wearable edge.

Emergence of small satellites for earth observation missions has opened up new horizons for space research but at the same time posed newer challenges of limited power and compute resource arising out of the size & weight constraints imposed by these satellites. The currently evolving neuromorphic computing paradigm shows promise in terms of energy efficiency and may possibly be exploited here. In this paper, we try to prove the applicability of neuromorphic computing for on-board data processing in satellites by creating a 2-stage hierarchical cloud cover detection application for multi-spectral earth observation images. We design and train a CNN and convert it into SNN using the CNN2SNN conversion toolkit of Brainchip Akida neuromorphic platform. We achieve 95.46% accuracy while power consumption and latency are at least 35x and 3.4x more efficient respectively in stage-1 (and 230x & 7x in stage-2) compared to the equivalent CNN running on Jetson TX2.

The emerging trend of small satellites for earth observation missions has enabled commercial organisations to exploit the horizon for various business applications related to weather forecasting/monitoring, Land Use Land Cover (LULC) classifications, disaster (such as oil spill or forest fire) monitoring etc. However, the limited power and computational capacity of these small satellites arising out of the size and weight restrictions have posed newer challenges, primarily related to low-power on-board data processing and transmission. One possible approach is to harness the capabilities of the evolving neuromorphic computing paradigm for such low-power computing requirements. One possible application can be lossless compression of high-resolution earth observation images before sending those downstream to ground stations for further analysis. In this paper, we propose a novel method of lossless image compression based on classical Arithmetic Encoding that exploits the low power computing capability of Spiking Neural Networks and neuromorphic platforms. We experimentally prove that our SNN approach achieves compression ratio at-par with state of the art ANN methods with an estimated 2.5× power efficiency and 50% lower latency with a much smaller model - thereby enabling on-board image compression and at the same time, saving on a corresponding amount of energy during transmission.

In this paper, we propose StereoGest-SNN which is a robust low power edge compatible ultrasound based gesture detection system. StereoGest-SNN uses easily available off-the-shelf stereo speaker and microphone setup inbuilt in most commercial devices. The stereo setup mimics a$2\times 2$Multiple Input Multiple Output (MIMO) system which provides the requisite diversity to effectively address fading. It also makes use of distinctive Channel Impulse Response (CIR) estimated by imposing sparsity prior for robust gesture detection. A 5-layer CNN has been trained on these distinctive CIRs and the trained model is converted into an equivalent Spiking Neural Network (SNN) via an ANN-to-SNN conversion mechanism. The StereoGest-SNN with converted SNN is found to classify gestures with an accuracy of around 92% (an improvement of over 6% compared to state-of-the-art) and shows$3\times$reduction in number of operations compared to CNN resulting in further energy benefit when run on neuromorphic computation platforms.

In recent times, in many industrial domains, the need for in-situ processing of time series data from sensors have grown extensively, to ensure low-latency real-time responses, making embedded edge AI an important area of work. Most current techniques, being computationally intensive, are not suited for edge implementation; and in real world scenarios, connectivity to the cloud for processing sensed data leads to higher latency and often less reliability. Neuromorphic systems, coupled with spiking neural networks (SNN) offer a solution to such problems. This paper explores this evolving paradigm to address the need for low-footprint efficient time series classifiers implementable on edge, and targeted for predictive maintenance scenarios. A reservoir-based SNN architecture is designed and tried for classification of different vibration time series datasets. While the system is found to obtain at par classification accuracy for each of the datasets compared to prior arts, it is also observed to be more efficient in terms of synaptic operations per timestep (13% to 38%) using Gaussian temporal spike encoding scheme compared to Poisson rate encoding. Moreover, the system is found to be robust with respect to learning with reduction in training data (upto 20%).

In this paper, we propose EchoWrite-SNN, a robust edge compatible air-writing recognition system (used in applications such as AR/VR, HRI etc.) based on principles of SONAR and neuromorphic computing. The bare finger movements in air are captured by a pair of commonly available speaker-microphone pair. A new tracking algorithm based on windowed difference cross-correlation and ESPRIT is employed which shows better tracking accuracy compared to state-of-the-art methods with a median tracking error of only 3.31mm. To classify these air-written shapes, a 5-layer CNN is trained and then converted to a Spiking Neural Network (SNN) using ANN-to-SNN conversion technique to reap the benefits of low power neuromorphic computing on edge. Experimental results show that the converted SNN achieves 92% accuracy (a mere 3% less than the CNN) while showing 4.4 × reduction in number of operations compared to CNN resulting in further energy benefit when run on actual neuromorphic computation platforms.

Low resource Handwritten Text Recognition (HTR) is a hard problem due to the scarce annotated data and the very limited linguistic information (dictionaries and language models). For example, in the case of historical ciphered manuscripts, which are usually written with invented alphabets to hide the message contents. Thus, in this paper we address this problem through a data generation technique based on Bayesian Program Learning (BPL). Contrary to traditional generation approaches, which require a huge amount of annotated images, our method is able to generate human-like handwriting using only one sample of each symbol in the alphabet. After generating symbols, we create synthetic lines to train state-of-the-art HTR architectures in a segmentation free fashion. Quantitative and qualitative analyses were carried out and confirm the effectiveness of the proposed method.

Scene text instances found in natural images carry explicit semantic information that can provide important cues to solve a wide array of computer vision problems. In this paper, we focus on leveraging multi-modal content in the form of visual and textual cues to tackle the task of fine-grained image classification and retrieval. First, we obtain the text instances from images by employing a text reading system. Then, we combine textual features with salient image regions to exploit the complementary information carried by the two sources. Specifically, we employ a Graph Convolutional Network to perform multi-modal reasoning and obtain relationship-enhanced features by learning a common semantic space between salient objects and text found in an image. By obtaining an enhanced set of visual and textual features, the proposed model greatly outperforms previous state-of-the-art in two different tasks, fine-grained classification and image retrieval in the Con-Text[23] and Drink Bottle[4] datasets.

In the past two decades, radar-based human sensing has become a topic of intense research. Unlike vision-based techniques which require the use of camera, radars are unobtrusive and privacy preserving in nature. Further, radars are agnostic of the lighting conditions and can be used for through-the-wall imaging thereby making them hugely effective in many situations. Compact, affordable radars have been designed that can be easily integrated with remote monitoring systems. However, the classical machine learning techniques currently used for learning and inferring human actions from radar images are compute intensive, and require large volume of training data, making them unsuitable for deployment on the network edge. In this paper, we propose to use the concepts of neuromorphic computing and Spiking Neural Networks (SNN) to learn human actions from data captured by the radar. To the best our knowledge, this is the first attempt of using SNNs on micro-Doppler data from radars. Our SNN model is capable of learning spatial as well as temporal features from the data and our experiments have resulted in 85% accuracy which is comparable with the classical machine learning approaches that are typically used on similar data. Further, the use of neuromorphic and SNN concepts make our model deployable over evolving neuromorphic edge devices thereby making the entire approach more efficient in terms of data, computation and energy consumption.

Mammalian neural circuits respond to different sensory stimuli by firing spikes at particular times. Closely mimicking this phenomenon, the evolving 3rd generation neural networks, known as Spiking Neural Networks (SNNs), are found to be capable of memorizing and learning from the spatio-temporal spike patterns. This makes SNN applicable in identification of human actions and gestures, especially in the robotics domain. The paradigm is also suited for Neuromorphic Systems leading to less energy intensive applications. In this work, we present a novel spiking neural network constituting multiple convolutional layers and a reservoir layer to extract spatial and temporal features respectively from human gesture videos captured with DVS camera. We achieved more than 95% Top-3 accuracy on IBM DVS dataset and we claim that the performance of our network is better in terms of accuracy vs. learning parameters ratio when compared to other networks.

Text contained in an image carries high-level semantics that can be exploited to achieve richer image understanding. In particular, the mere presence of text provides strong guiding content that should be employed to tackle a diversity of computer vision tasks such as image retrieval, fine-grained classification, and visual question answering. In this paper, we address the problem of fine-grained classification and image retrieval by leveraging textual information along with visual cues to comprehend the existing intrinsic relation between the two modalities. The novelty of the proposed model consists of the usage of a PHOC descriptor to construct a bag of textual words along with a Fisher Vector Encoding that captures the morphology of text. This approach provides a stronger multimodal representation for this task and as our experiments demonstrate, it achieves state-of-the-art results on two different tasks, fine-grained classification and image retrieval. The code of this model will be publicly available at1.

In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research.

In this work we introduce a cross modal image retrieval system that allows both text and sketch as input modalities for the query. A cross-modal deep network architecture is formulated to jointly model the sketch and text input modalities as well as the the image output modality, learning a common embedding between text and images and between sketches and images. In addition, an attention model is used to selectively focus the attention on the different objects of the image, allowing for retrieval with multiple objects in the query. Experiments show that the proposed method performs the best in both single and multiple object image retrieval in standard datasets.

Robotic automation has made significant inroads into industrial manufacturing and supply chains. With Industry 4.0 requirements proposing further autonomy to robotic participants, it is necessary to reason about robotic tasks within a knowledge dependent software framework. In this work, we model robotic automation tasks using hierarchical decomposition models, that are used to extract action plans to satisfy end goals. By abstracting components as intelligent agents that have perception, action, goal and knowledge base elements, we provide a reusable model to abstract robotic automation behavior. Through the use of the formal typed specification language Orc, implementations that confirm to the goal decomposition process are formulated. We demonstrate our techniques over Smart Warehouse deployments, with domain specific ontologies to ensure accurate type based descriptions of knowledge elements. This provides a generic framework for deploying intelligent automation systems across a host of industrial settings.

Information Retrieval (IR) is the activity of obtaining information resources relevant to a questioned information. It usually retrieves a set of objects ranked according to the relevancy to the needed fact. In document analysis, information retrieval receives a lot of attention in terms of symbol and word spotting. However, through decades the community mostly focused either on printed or on single writer scenario, where the state-of-the-art results have achieved reasonable performance on the available datasets. Nevertheless, the existing algorithms do not perform accordingly on multiwriter scenario. A graph representing relations between a set of objects is a structure where each node delineates an individual element and the similarity between them is represented as a weight on the connecting edge. In this paper, we explore different analytics of graphs constructed from words or graphical symbols, such as diffusion, shortest path, etc. to improve the performance of information retrieval methods in multiwriter scenario.

Handwriting recognition consists in obtaining the transcription of a text image. Recent word spotting methods based on attribute embedding have shown good performance when recognizing words. However, they are holistic methods in the sense that they recognize the word as a whole (i.e. they find the closest word in the lexicon to the word image). Consequently, these kinds of approaches are not able to deal with out of vocabulary words, which are common in historical manuscripts. Also, they cannot be extended to recognize text lines. In order to address these issues, in this paper we propose a handwriting recognition method that adapts the attribute embedding to sequence learning. Concretely, the method learns the attribute embedding of patches of word images with a convolutional neural network. Then, these embeddings are presented as a sequence to a recurrent neural network that produces the transcription. We obtain promising results even without the use of any kind of dictionary or language model.

Developing analytical applications for IoT based on sensor signal processing tends to be complicated as applications are executed as sequence of steps comprising of multiple alternative algorithms, including suitable feature extraction modules depending on the goal of the application. Experience shows that developers spend considerable time and effort in performing feature extraction and dimensionality reduction. In this paper we propose a framework based on a relevant case study which allows developers to drag and drop algorithms to create a workflow chain, automatically select the most relevant signal features for the particular analytic application using a training data set to generate a model and deploy the model for use. The method reduces the effort and cost of development which is deemed highly important for the analytics industry.

Numerous sensors are involved in sensing almost everything around us in a cyber-physical system. They generate heterogeneous data which are diverse in format and protocol. Prevailing standards formally define sensor data encodings and provide syntactic interoperability but failed to provide semantic capability required for searching, reasoning, querying on sensors and their observations. The paper extends IEEE SUMO and Onto Sensor to create an ontological structure that captures various sensor concepts and their relationships. It logically classifies sensor information into four major sensor knowledge representation (1) hierarchy (2) data (3) function and (4) data exchange. The proposed sensor ontology has been extended for energy meter as an energy sensor. The represented semantic can be captured through Protégé 4.3.0 and subsequently by OWLXML. Reasoning on the sensor data can be carried out using SWRL.