Affect is often expressed via non-verbal body language such as actions/gestures, which are vital indicators for human behaviors. Recent studies on recognition of fine-grained actions/gestures in monocular images have mainly focused on modeling spatial configuration of body parts representing body pose, human-objects interactions and variations in local appearance. The results show that this is a brittle approach since it relies on accurate body parts/objects detection. In this work, we argue that there exist local discriminative semantic regions, whose “informativeness” can be evaluated by the attention mechanism for inferring fine-grained gestures/actions. To this end, we propose a novel end-to-endregional attention network (RAN), which is a fully convolutional neural network (CNN) to combine multiple contextual regions through attention mechanism, focusing on parts of the images that are most relevant to a given task. Our regions consist of one or more consecutive cells and are adapted from the strategies used in computing HOG (Histogram of Oriented Gradient) descriptor. The model is extensively evaluated on ten datasets belonging to 3 different scenarios: 1) head pose recognition, 2) drivers state recognition, and 3) human action and facial expression recognition. The proposed approach outperforms the state-of-the-art by a considerable margin in different metrics.

The ability to apply a previously-learned skill (e.g., pushing) to a new task (context or object) is an important requirement for new-age robots. An attempt is made to solve this problem in this paper by proposing a deep meta-imitation learning framework comprising of an attentive-embedding net-work and a control network, capable of learning a new task in an end-to-end manner while requiring only one or a few visual demonstrations. The feature embeddings learnt by incorporating spatial attention is shown to provide higher embedding and control accuracy compared to other state-of-the-art methods such as TecNet [7] and MIL [4]. The interaction between the embedding and the control networks is improved by using multiplicative skip-connections and is shown to overcome the overfitting of the trained model. The superiority of the proposed model is established through rigorous experimentation using a publicly available dataset and a new dataset created using PyBullet [36]. Several ablation studies have been carried out to justify the design choices.

Human activity/behaviour monitoring and recognition is a key for facilitating humans robot interaction, and allows robots for a better scheduling of future operations. It is challenging and often addressed at different levels, such as human activity classification, future activity prediction and monitoring of the on-going activities. The paper proposes a novel attention-based learn-able pooling mechanism for human activity classification from RGB videos. Recently, most of the best performing human activity recognition approaches are based on 3D skeleton positions. The 3D skeleton positions are not always available in videos captured using RGB cameras, which are widely used in robotics applications. RGB videos contain rich spatio-temporal information and processing them semantically is a difficult task. Moreover, accurately capturing spatial information and long-term temporal dependencies is the key to achieving high recognition accuracy. We use an existing Convolutional Neural Network for image recognition to extract video features which are then processed using our innovative application of attention mechanism to focus the network on features that are more important for discrimination. Afterwards, we use a novel learn-able pooling mechanism to extract activity-aware spatio-temporal cues for efficient activity recognition. The proposed pooling mechanism learns the structural information from hidden states of a bidirectional Long Short-Term Memory network via Fisher Vectors.

This article proposes a novel attention-based body pose encoding for human activity recognition. Most of the existing human activity recognition approaches based on 3D pose data often enrich the input data using additional handcrafted representations such as velocity, super-normal vectors, pairwise relations, and so on. The enriched data complements the 3D body joint position data and improves model performance. In this paper, we propose a novel approach that learns enhanced feature representations from a given sequence of 3D body joints. To achieve this encoding, the approach exploits two body pose streams: 1) a spatial stream which encodes the spatial relationship between various body joints at each time point to learn spatial structure involving the spatial distribution of different body joints 2) a temporal stream that learns the temporal variation of individual body joints over the entire sequence duration to present a temporally enhanced representation. Afterwards, these two pose streams are fused with a multi-head attention mechanism. We also capture the contextual information from the RGB video stream using a deep Convolutional Neural Network (CNN) model combined with a multi-head attention and a bidirectional Long Short-Term Memory (LSTM) network. Finally, the RGB video stream is combined with the fused body pose stream to give a novel end-to-end deep model for effective human activity recognition. The proposed model is evaluated on three datasets including the challenging NTU-RGBD dataset and achieves state-of-the-art results.

This paper proposes an end-to-end self-supervised feature representation network named Attentive Task-Net or AT-Net for video-based task imitation. The proposed AT-Net incorporates a novel multi-level spatial attention module to highlight spatial features corresponding to the intended task demonstrated by the expert. The neural connections in AT-Net ensure the relevant information in the demonstration is amplified and the irrelevant information is suppressed while learning task-specific feature embeddings. This is achieved by a weighted combination of multiple intermediate feature maps of the input image at different stages of the CNN pipeline. The weights of the combination are given by the compatibility scores, predicted by the attention module for respective feature maps. The AT-Net is trained using a metric learning loss which aims to decrease the distance between the feature representations of concurrent frames from multiple view points and increase the distance between temporally consecutive frames. The AT-Net features are then used to formulate a reinforcement learning problem for task imitation. Through experiments on the publicly available Multi-view pouring dataset, it is demonstrated that the output of the attention module highlights the task-specific objects while suppressing the rest of the background. The efficacy of the proposed method is further validated by qualitative and quantitative comparison with a state-of-the-art technique along with intensive ablation studies. The proposed method is implemented to imitate a pouring task where an RL agent is learned with the AT-Net in Gazebo simulator. Our findings show that the AT-Net achieves 6.5% decrease in alignment error along with a reduction in the number of training iterations by almost 155k over the state-of-the-art while satisfactorily imitating the intended task.

Capsule Network (CapsNet) has motivated researchers to work on it due to its distinct capability of retaining spatial correlations between image features. However, its applicability is still limited because of its intensive computational cost, memory usage and bandwidth requirement. This paper proposes a computationally efficient, lightweight CapsNet which paves its way forward for deployment in constrained edge devices as well as in web based applications. The proposed framework consists of Capsule layers and a deep feature representation layer as an input for capsules. The deep feature representation layer comprises of a series of feature blocks, containing convolution with a 3 × 3 kernel followed by batch normalization and convolution with a 1 × 1 kernel. The deeper or better represented input features help to improve recognition performance even with lesser number of capsules, making the network computationally more efficient. The efficacy of the proposed framework is validated by performing rigorous experimental studies on different datasets, such as CIFAR-10, FMNIST, MNIST and SVHN which include images of object classes as well as text characters. A comparative analysis has also been done with the state-of-the-art technique CapsNet. The comparison with recognition accuracy ensures that, the proposed architecture with deep input features provides more efficient routing between the capsules as compared to CapsNet. The proposed lightweight network has scaled down the number of parameters up to 60% of CapsNet, which is another significant contribution. This is achieved by collaborative effect of deep feature generation module and parametric changes performed in the primary capsule layer.

In this paper, we propose an unsupervised deep learning framework with Bayesian inference for improving the accuracy of per-pixel depth prediction from monocular RGB images. The proposed framework predicts confidence map along with depth and pose information for a given input image. The depth hypotheses from previous frames are propagated forward and fused with the depth hypothesis of the current frame by using Bayesian inference mechanism. The ground truth information required for training the confidence map prediction is constructed using image reconstruction loss thereby obviating the need for explicit ground truth depth information used in supervised methods. The resulting unsupervised framework is shown to outperform the existing state-of-the-art methods for depth prediction on the publicly available KITTI outdoor dataset. The usefulness of the proposed framework is further established by demonstrating a real-world robotic pick-and-place application where the pose of the robot end-effector is computed using the depth predicted from an eye-in-hand monocular camera. The design choices made for the proposed framework is justified through extensive ablation studies.

Visual place recognition (VPR) - the act of recognizing a familiar visual place - becomes difficult when there is extreme environmental appearance change or viewpoint change. Particularly challenging is the scenario where both phenomena occur simultaneously, such as when returning for the first time along a road at night that was previously traversed during the day in the opposite direction. While such problems can be solved with panoramic sensors, humans solve this problem regularly with limited field-of-view vision and without needing to constantly turn around. In this paper, we present a new depth- and temporal-aware visual place recognition system that solves the opposing viewpoint, extreme appearance-change visual place recognition problem. Our system performs sequence-to-single frame matching by extracting depth-filtered keypoints using a state-of-the-art depth estimation pipeline, constructing a keypoint sequence over multiple frames from the reference dataset, and comparing these keypoints to the keypoints extracted from a single query image. We evaluate the system on a challenging benchmark dataset and show that it consistently outperforms state-of-the-art techniques. We also develop a range of diagnostic simulation experiments that characterize the contribution of depth-filtered keypoint sequences with respect to key domain parameters including the degree of appearance change and camera motion.

This paper presents a new GAN-based deep learning framework for estimating absolute scale awaredepth and ego motion from monocular images using a completely unsupervised mode of learning.The proposed architecture uses two separate generators to learn the distribution of depth and posedata for a given input image sequence. The depth and pose data, thus generated, are then evaluated bya patch-based discriminator using the reconstructed image and its corresponding actual image. Thepatch-based GAN (or PatchGAN) is shown to detect high frequency local structural defects in thereconstructed image, thereby improving the accuracy of overall depth and pose estimation. Unlikeconventional GANs, the proposed architecture uses a conditioned version of input and output of thegenerator for training the whole network. The resulting framework is shown to outperform all existing deep networks in this field and beating the current state-of-the-art method by 8.7% in absoluteerror and 5.2% in RMSE metric. To the best of our knowledge, this is first deep network based modelto estimate both depth and pose simultaneously using a conditional patch-based GAN paradigm.The efficacy of the proposed approach is demonstrated through rigorous ablation studies and exhaustive performance comparison on the popular KITTI outdoor driving dataset.

Automatic recognition of text, such as a batch code printed on a box placed on a moving conveyor belt, is still a challenging problem. This paper proposes an end-to-end character recognition technique while addressing the major challenges encountered in a real environment, such as motion blur in the acquired images, slanted or oriented characters, creased batch codes due to wear and tear of boxes, variations in label formats, and variations in printing styles. The major contribution of this work lies in development of three sequential modules: text localization using Connectionist Text Proposal Network(CTPN), character detection and character recognition using a modified version of the capsule network (CapsNet). In contrast to CapsNet, where only a standard single convolution is used, the proposed method uses a series of feature blocks, making it a deep CapsNet which is later proven to generate more comprehensive and better separable feature vectors over its counterpart. The feature generation module is further enhanced by setting a smaller kernel size than CapsNet. The proposed system is validated on a real-world box / packet dataset generated in a retail manufacturing industry. The proposed recognition network architecture is also validated on a standard public dataset (ICDAR 2013). The comparative results are presented with statistical analysis in the experimental results section.

This paper looks into the problem of online 2D bin packing where the objective is to place an incoming object in a way so as to maximize the overall packing density inside the bin. Unlike off-line methods, the online methods do not make use of information about the sequence of future objects that are going to arrive and hence, are comparatively difficult to solve. A deep reinforcement learning framework based on Double DQN is proposed to solve this problem that takes an image showing the current state of the bin as input and gives out the pixel location where the incoming object needs to be placed as the output. The reward function is defined in such a way so that the system learns to place an incoming object adjacent to the already placed items so that the maximum grouped empty area is retained for future placement. The resulting approach is shown to outperform existing state-of-the-art-method for 2D online packing and can easily be extended to 3D online bin packing problems.

This paper looks into the problem of grasping unknown objects in a cluttered environment using a 3D point cloud data obtained from a range sensor or an RGBD sensor. The objective is to identify graspable regions and detect suitable grasp poses from a single view, possibly, partial 3D point cloud without any apriori knowledge of the object geometry. The problem is solved in two steps - first, identifying and segmenting various object surfaces and second, searching for suitable grasping handles on these surfaces by applying geometric constraints of the physical gripper. The first step is solved by using a modified version of region growing algorithm that uses a pair of thresholds for the smoothness constraint on local surface normals to find natural boundaries of object surfaces. In this process, a novel concept of edge point is introduced that allows us to segment between different surfaces of the same object. The second step is solved by converting a 6D pose detection problem into a 1D linear search problem by projecting 3D cloud points onto the principal axes of the surface segment obtained in the first step. The graspable handles are then localized by applying physical constraints of the gripper. The resulting method allows us to grasp all kinds of objects including rectangular or box-type objects with flat surfaces which is otherwise considered to be difficult in the grasping literature. The proposed method is simple and can be implemented in real-time and does not require any off-line training phase for computing these affordances. The improvements achieved is demonstrated through comparison with another state-of-the-art grasping algorithm on various publicly-available datasets. We also contribute a new grasping dataset for extreme clutter situations.

The presented work focuses on automatic recognition of object classes while ensuring near real-time training required for recognizing a new object not seen previously. This is achieved by proposing a two-stage hierarchical deep learning framework which first learns object categories using a Nearest Class Mean (NCM) classifier applied directly to CNN features and then, uses a two-layer artificial neural network to learn the object labels within each category. In order to recognize a new object not seen earlier, the category is identified first and then the second stage neural network is incrementally trained with the features of the new object without forgetting previously learnt labels. The proposed hierarchical framework is shown to provide comparable recognition accuracy with significant reduction in overall computational time in recognizing new objects compared to methods that use end-to-end re-training. The efficacy of the approach is demonstrated through comparison with existing state-of-the-art methods on the publicly available CORe50 dataset.

This paper presents a feature agnostic and model-free visual servoing (VS) technique using deep reinforcement learning (DRL) which exploits two new architectures of experience replay buffer in deep deterministic policy gradient (DDPG). The proposed architectures are significantly fast and converge in a few numbers of steps. We use the proposed method to learn an end-to-end VS with eye-in-hand configuration. In traditional DDPG, the experience replay memory is randomly sampled for training the actor-critic network. This results in a loss of useful experiences when the buffer contains very few successful examples. We solve this problem by proposing two new replay buffer architectures: (a) min-heap DDPG (mH-DDPG) and (b) dual replay buffer DDPG (dR-DDPG). The former uses a min-heap data structure to implement the replay buffer whereas the latter uses two buffers to separate “good” examples from the “bad” examples. The training data for the actor-critic network is created as a weighted combination of the two buffers. The proposed algorithms are validated in simulation with the UR5 robotic manipulator model. It is observed that as the number of good experiences increases in the training data, the convergence time decreases. We find 27.25% and 43.25% improvements in the rate of convergence respectively by mH-DDPG and dR-DDPG over state-of-the-art DDPG.

Transferring skills to roUots through human demonstrations is an interesting problem. Locally generated demonstrations of reaching motion, given by a human teacher are generally encoded in a dynamical model. Stability of this encoding system demands great attention while learning the model parameters. In that context, we present a new architecture of dynamical system to learn movement primitives from multiple demonstrations exploiting a fuzzy Lyapunov function (FLF). We assume that there exists a natural Lyapunov function (LF) that associates the demonstrations. The proposed FLF tries to approximate that LF. First, the dynamics of the demonstrations are encoded in a regressive model, learnt using Gaussian mixture regression with EM algorithm. Then the FLF is searched involving the learnt dynamics in an optimization process. The FLF in turn helps to learn a fuzzy controller. Our architecture is new in a sense that it combines the probabilistic model with a fuzzy controller to create a globally asymptotically stable motion model. The proposed algorithm can simultaneously learn position and orientation profiles in a single model. The algorithm is experimentally validated on a commercially available manipulator and also compared with a state-of-the-art technique.

The paper presents a deep learning based fully automatic object annotation technique for warehouse application usecase. One of the main challenges that is addressed in this paper is the large amount of manual labour involved in generating datasets for training a deep network. The proposed annotation model is developed by fine-tuning a deep network based object detection framework with ImageNet pre-trained models. We have used Faster RCNN network with pre-trained model VGG-16 and RFCN with ResNet-101. A small set of manually annotated images of single objects are used to automatically generate a dataset of significantly large size within a very short time duration (in real-time). The model also has the competence of precisely localizing the region of any new object that comes into the familiar background. Incorporation of techniques like color augmentation and affine transformation enables the network invariant to rotation, scale and brightness. Augmentation also enables the model to performs well even if the background is different. A clutter generation technique is introduced in the framework which makes the system capable of annotating objects even in a densely populated real-world environment. This work has another significant contribution in detection of objects those are used in Amazon Robotic Challenge (ARC) 2017 where our team was among the four finalist in both picking and stowing task. The automatically generated big dataset is further used to train multi-class detectors using Faster RCNN and RFCN networks to validate the performance of the proposed annotation model. The efficacy of the proposed model is hence demonstrated through various experimental results. The dataset is shared online for the convenience of the reader.

This paper presents a deep network based unsupervised visual odometry system for 6-DoF camera pose estimation and finding dense depth map for its monocular view. The proposed network is trained using unlabeled binocular stereo image pairs and is shown to provide superior performance in depth and ego-motion estimation compared to the existing state-of-the-art. This is achieved by introducing a novel objective function and training the network using temporally alligned sequences of monocular images. The objective function is based on the Charbonnier penalty applied to spatial and bi-directional temporal reconstruction losses. The overall novelty of the approach lies in the fact that the proposed deep framework combines a disparity-based depth estimation network with a pose estimation network to obtain absolute scale-aware 6-DoF camera pose and superior depth map. According to our knowledge, such a framework with complete unsupervised end-to-end learning has not been tried so far, making it a novel contribution in the field. The effectiveness of the approach is demonstrated through performance comparison with the state-of-the-art methods on KITTI driving dataset.

The place recognition problem comprises two distinct subproblems; recognizing a specific location in the world (“specific” or “ordinary” place recognition) and recognizing the type of place (place categorization). Both are important competencies for mobile robots and have each received significant attention in the robotics and computer vision community, but usually as separate areas of investigation. In this paper, we leverage the powerful complementary nature of place recognition and place categorization processes to create a new hybrid place recognition system that uses place context to inform place recognition. We show that semantic place categorization creates an informative natural segmenting of physical space that in turn enables significantly better place recognition performance in comparison to existing techniques. In particular, this new semantically-informed approach adds robustness to significant local changes within the environment, such as transitioning between indoor and outdoor environments or between dark and light rooms in a house, complementing the capabilities of current condition-invariant techniques that are robust to globally consistent change (such as day to night cycles). We perform experiments using 4 novel benchmark datasets and show that semantically-informed place recognition outperforms the previous state-of-the-art systems. Like it does for object recognition [1], we believe that semantics can play a key role in boosting conventional place recognition and navigation performance for robotic systems.

The ability to follow a human is an important requirement for a service robot designed to work along side humans in homes or in work places. This paper describes the development and implementation of a novel robust visual controller for the human-following robot. This visual controller consists of two parts: 1) a robust algorithm that tracks a human visible in its camera view and 2) a servo controller that generates necessary motion commands so that the robot can follow the target human. The tracking algorithm uses point-based features, like speeded up robust feature, to detect human under challenging conditions, such as, variation in illumination, pose change, full or partial occlusion, and abrupt camera motion. The novel contributions in the tracking algorithm include the following: 1) a dynamic object model that evolves over time to deal with short-term changes, while maintaining stability over long run; 2) an online K-D tree-based classifier along with a Kalman filter is used to differentiate a case of pose change from a case of partial or full occlusion; and 3) a method is proposed to detect pose change due to out-of-plane rotations, which is a difficult problem that leads to frequent tracking failures in a human following robot. An improved version of a visual servo controller is proposed that uses feedback linearization to overcome the chattering phenomenon present in sliding mode-based controllers used previously. The efficacy of the proposed approach is demonstrated through various simulations and real-life experiments with an actual mobile robot platform.

This paper looks into the problem of pedestrian tracking using a monocular, potentially moving, uncalibrated camera. The pedestrians are located in each frame using a standard human detector, which are then tracked in subsequent frames. This is a challenging problem as one has to deal with complex situations like changing background, partial or full occlusion and camera motion. In order to carry out successful tracking, it is necessary to resolve associations between the detected windows in the current frame with those obtained from the previous frame. Compared to methods that use temporal windows incorporating past as well as future information, we attempt to make decision on a frame-by-frame basis. An occlusion reasoning scheme is proposed to resolve the association problem between a pair of consecutive frames by using an affinity matrix that defines the closeness between a pair of windows and then, uses a binary integer programming to obtain unique association between them. A second stage of verification based on SURF matching is used to deal with those cases where the above optimization scheme might yield wrong associations. The efficacy of the approach is demonstrated through experiments on several standard pedestrian datasets.

Detecting and tracking a human from a mobile robot platform has several applications in service robotics where a robot is expected to assist humans. In this paper, we propose a novel interest point-based algorithm that can track a human reliably under several challenging situations like variation in illumination, pose change, scaling, camera motion and occlusion. The limitations of point-based methods are overcome using colour information and imposing a structure on the colour blobs. Whenever sufficient number of SURF matching points are not available for a given frame, the presence of human is detected using Markov random field based graph matching algorithm. Imposition of structure on coloured blobs helps in eliminating background objects having similar colour distribution. The stability-versus-plasticity dilemma inherent in tracking over long run is resolved by selecting new templates on-line and maintaining a tree of templates which is updated with new information. The performance of the algorithm is demonstrated through simulation on standard datasets and the computation time is found to be comparable with existing SURF-based tracking methods.

The interest point based tracking methods suffer from the limitation of unavailability of sufficient number of matching key points for the target in all frames of a running video. In this paper, a dynamic model is proposed for describing the object model which is used for tracking a human in a non-stationary video. This dynamic model takes into account the change in the pose as well as the motion of the human. A simple autoregression based predictor is used for dealing with the case of full occlusion. Simulation results are provided to show the efficacy of the algorithm.

This paper deals with visual-motor coordination of a 7 dof robot manipulator for pick and place applications. Three issues are dealt with in this paper - finding a feasible inverse kinematic solution without using any orientation information, resolving redundancy at position level and finally maintaining the fidelity of information during clustering process thereby increasing accuracy of inverse kinematic solution. A 3-dimensional KSOM lattice is used to locally linearize the inverse kinematic relationship. The joint angle vector is divided into two groups and their effect on end-effector position is decoupled using a concept called function decomposition. It is shown that function decomposition leads to significant improvement in accuracy of inverse kinematic solution. However, this method yields a unique inverse kinematic solution for a given target point. A concept called sub-clustering in configuration space is suggested to preserve redundancy during learning process and redundancy is resolved at position level using several criteria. Even though the training is carried out off-line, the trained network is used online to compute the required joint angle vector in only one step. The accuracy attained is better than the current state of art. The experiment is implemented in real-time and the results are found to corroborate theoretical findings.

In the above titled paper (ibid., vol. 17, no. 5, pp. 1116-1125), there were a few errors. Corrections are presented here.

This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima.