For a robot to manipulate an object, it has to understand the functions and the actions that can be subjected to the object. This set of information is known as affordance of the object. Affordances are generally defined by the geometrical structures and physical properties of the objects. In this paper, we present an affordance detection network (ADNet) for detecting object affordances using multimodal input i.e., RGB-D data. The method is based on the state-of-the-art fully convolutional network with two encoding streams and one decoding stream. In the presented formulation, the network learns powerful discriminative features independently from the RGB and depth images, which enables it to abstract rich photometrical and geometrical properties of the objects. The multimodal encoding is combined at multiple stages of the network using the late-fusion strategy and used is for predicting the potential affordances of the objects.

In this paper, we introduce the achievement of the flight motion to pass through small opening by the multilinked and transformable aerial robot. Previous works about such motion are based on under-actuated multirotors, indicating that aggressive maneuvering is necessary condition. This involves two crucial problems: i) enough free space for deceleration is necessary, otherwise the robot would collide with unknown obstacle after exiting opening; ii) the multirotor can not traverse the openings that are smaller than the robot body. The proposed transformable aerial robot in our work can solve these problems, since the multilinked model can not only guarantee the near-hover condition during the whole motion sequence, but also slowly traverse relative small openings by changing its form like a snake. We first propose an improved dynamics derivation and flight control method for this multilinked aerial robot based on our previous work. Then, we present the path planning method which takes the flight stability in the near-hover condition into account. Finally we demonstrate the experimental results of the motion to pass through a horizontal and small opening which also involves the borders (the floor and the ceiling).

In recent years, applications of visual tracking algorithms has seen a substantial growth with deployments in intelligent robots such as drones for human tracking. The algorithms for such tasks has to be efficient in terms of computational cost while been robust, accurate and fast. Object tracking algorithms based on handcrafted heuristics and constraints are widely used in uav applications. The handcrafted heuristics are mostly implemented for task-oriented applications which limits the extensions in uav's capability beyond the predefined functions. This paper considers the challenges of tracking and landing an autonomous uav on a speed high moving target, and presents a visual tracking algorithm that integrates correlation filters with deep comparison network for real-time tracking with state-of-the-art accuracy. The method first tracks the target upto translation using an online learnt model via local search technique. The changes in scale is estimated by a deep comparison network (DCN) instead of the commonly used pyramidal approach. In a single network evaluation, DCN can estimate the changes in scale as well as compensate the drifting of the tracker by refining the object region estimated by the correlation filters. The network is end-to-end trained which attempts to learn a powerful matching function for object localization using a known template. Generally, the integrated framework can be viewed as coarse-to-fine level motion estimation. Moreover, the framework can redetect the lost target without a need for a separate detector.

Visual tracking of objects subjected to non-linear motion and appearance changes has shown to be a difficult task in computer vision. While research in visual object tracking has progressed significantly in terms of robust tracking of objects subjected to non-linear motion and appearance changes, these algorithms has shown limited capability for long term tracking of handheld objects during human-object interactions. The failure in tracking is a consequence of abrupt changes in the handheld object motion resulting in tracker drifting off the optimal object space. In this paper, we present a novel 3 layer RGB-D image model formulated with Bayesian filters that tracks handheld object using near constant velocity motion model. Our method divides the image into three layers of abstraction where each encodes visual information of environment, human, object and contributes toward precise localization of the handheld object during tracking. A boundary re-alignment step is introduced during tracking such that the tracker predicted object region is re-aligned to the optimal object region, therefore reducing the likelihood of tracker drifting off the object space. This compensation of the tracker prediction offset enables our algorithm to robustly track handheld object subjected to abrupt changes in motion during manipulation.

We present a vision cognition framework for tomato harvesting humanoid robot based on geometrical and physical reasoning. Inspired from the natural human harvesting behaviour, our goal is to build a humanoid robot to pick tomatoes autonomously or with minimal human efforts. The proposed vision approach uses fusion of calibrated observation data from two RGB-D sensors installed on the head and the hand of the humanoid. We observe the natural human harvesting behaviour and equip our robot with similar grippers to follow the same picking processes for a specific fruit. In the vision approach, we mainly focus on modelling fruits in one branch and then estimating the pedicel direction of each fruit in a branch. Through pointcloud model segmentation, the primitive shape model of each fruit can be obtained and we consider a simple fact that crops in one branch should remain stable with respect to gravity and interaction forces from neighbouring crops in the branch. According to this assumption, a probabilistic model is created and the picking order in the branch is assigned under the evaluated geometrical structure. In the experiments, we tested harvesting of real tomatoes on actual branches and evaluated the successful harvesting rate.

Robots operating in human environments must have the ability to autonomously acquire object representations in order to perform object search and recognition tasks without human intervention. However, autonomous acquisition of object appearance model in an unstructured and cluttered human environment is a challenging task, since the object boundaries are unknown in prior. In this paper, we present a novel method to solve the problem of unknown object boundaries for handheld objects in an unstructured environment using robotic vision. The objective is to solve the problem of object segmentation without prior knowledge of the objects that human interacts with daily. In particular, we present a method that segments handheld objects by observing human-object interaction process, and performs incremental learning on the acquired models using SVM. The unknown object boundary is estimated using sequential back-tracking via exploitation of affine relationship of consecutive frames. The segmentation is achieved using identified optimal object boundaries, and the extracted models are used to perform future object search and recognition tasks.