This paper tackles dynamic multi-objective optimization problems (DMOPs) by proposing novel change prediction strategies within an evolutionary algorithm framework. This framework combines an adaptive dual-domain change-response strategy with a second-order derivative prediction mechanism. In real-world scenarios, some Pareto Sets (PS) and Pareto Fronts (PF) evolve dynamically with environmental changes, while others remain stationary. Our algorithm employs an adaptive dual-domain approach that simultaneously monitors changes in both the PS and PF, and dynamically adjusts the allocation of prediction efforts between the decision and objective spaces according to environmental characteristics, thereby ensuring efficient sampling when objectives change. Furthermore, we incorporate a second-order derivative prediction scheme to actively reinitialize the population, enhancing the algorithm’s responsiveness to sudden or nonlinear changes. We evaluate the proposed method on 28 standard benchmark DMOPs and compare it with six state-of-the-art algorithms. The experimental results indicate that the proposed method achieves significant advantages in convergence and diversity on most test problems.

In this paper, we investigate the impact of high-level semantics (evaluation of the environment) on Human-Robot Team (HRT) and Human-Robot Interaction (HRI) in the context of mobile robot deployments. Although semantics has been widely researched in AI, how high-level semantics can benefit the HRT paradigm is underexplored, often fuzzy, and intractable. We applied a semantics-based framework that could reveal different indicators of the environment (i.e. how much semantic information exists) in a mock-up disaster response mission. In such missions, semantics are crucial as the HRT should handle complex situations and respond quickly with correct decisions, where humans might have a high workload. Especially when human operators need to shift their attention between robots and other tasks, they will struggle to build Situational Awareness (SA) quickly. The experiment suggests that the presented semantics: 1) alleviate the perceived human operator’s workload; 2) increase the operator’s trust in the SA; and 3) help to reduce the reaction time in switching the Level of Autonomy (LoA) when needed. Additionally, we find that participants with higher trust in the system are encouraged by high-level semantics to use teleoperation mode more.

Robotic cutting, a crucial task in applications such as disassembly and decommissioning, faces challenges due to uncertainties in real-world environments. This paper presents a novel approach to enhance sim-to-real transfer of robotic cutting policies, leveraging a hybrid method integrating Gaussian process (GP) regression to model disturbance forces encountered during cutting tasks. By learning from a limited number of real-world trials, our method captures residual process dynamics, enabling effective adaptation to diverse materials without the need for fine-tuning on physical robots. Key to our approach is the utilisation of imitation learning, where expert actions in the uncorrected simulation are paired with GP-corrected observations. This pairing aligns action distributions between simulated and real-world domains, facilitating robust policy transfer. We illustrate the efficacy of our method through real world cutting trials in autonomously adapting to diverse material properties; our method surpasses re-training, while providing similar benefits to fine-tuning in real-world cutting scenarios. Notably, policies transferred using our approach exhibit enhanced resilience to noise and disturbances, while maintaining fidelity to expert behaviours from the source domain.

This paper presents a novel method to quantify Trust in HRI. It proposes an HRI framework for estimating the Robot Trust towards the Human in the context of a narrow and specified task. The framework produces a real-time estimation of an AI agent's Artificial Trust towards a Human partner interacting with a mobile teleoperation robot. The approach for the framework is based on principles drawn from Theory of Mind, including information about the human state, action, and intent. The framework creates the ATTUNE model for Artificial Trust Towards Human Operators. The model uses metrics on the operator's state of attention, navigational intent, actions, and performance to quantify the Trust towards them. The model is tested on a pre-existing dataset that includes recordings (ROSbags) of a human trial in a simulated disaster response scenario. The performance of ATTUNE is evaluated through a qualitative and quantitative analysis. The results of the analyses provide insight into the next stages of the research and help refine the proposed approach.

This paper addresses the problem of robotic cutting during disassembly of products for materials separation and recycling. Waste handling applications differ from milling in manufacturing processes, as they engender considerable variety and uncertainty in the parameters (e.g. hardness) of materials which the robot must cut. To address this challenge, we propose a learning-based approach incorporating elements of interaction control, in which the robot can adapt key parameters, such as feed rate, depth of cut, and mechanical compliance during task execution. We show how a mathematical model of cutting mechanics, embedded in a simulation environment, can be used to rapidly train the system without needing large amounts of data from physical cutting trials. The simulation approach was validated on a real robot setup based on four case study materials with varying structural and mechanical properties. We demonstrate the proposed method minimises process force and path deviations to a level similar to offline optimal planning methods, while the average time to complete a cutting task is within 25% of the optimum, at the expense of reduced volume of material removed per pass. A key advantage of our approach over similar works is that no prior knowledge about the material is required.Note to Practitioners—This work is motivated by challenges in emerging fields such as recycling of electric vehicles, where products such as batteries adopt a range of designs with varying physical geometry and materials. More generally, this applies when considering robotic disassembly of any unknown component where semi-destructive operations such as cutting are required. Product-to-product variation introduces challenges when planning cutting processes required to disassemble a component, as contemporary planning approaches typically require advance knowledge of the material properties, shape and desired path to select tool speed, feed and depth of cut. In this paper, we show a mathematical model of milling force embedded in a simulation environment can be used as a relatively inexpensive approach to simulate a broad spectrum of cutting processes the robot may encounter. This allows the robot to learn from experience a strategy that can select these key parameters of a milling task online without user assistance. We develop a framework for controlling a robot using this strategy that allows the stiffness of the robot arm to be modulated over time to best satisfy metrics of productivity (e.g. required cutting time), while maintaining safe interaction of the robot with its environment (e.g. by avoiding force limits), similarly to how a human operator can vary muscular tension to accomplish different tasks. We posit that the proposed method can substitute a trial-and-error strategy of selecting process parameters for disassembly of novel products, or integrated with existing planning approaches to adjust the parameters of milling tasks online.

Superpixel segmentation algorithms are predominently based on simple linear iterative clustering (SLIC), and treat homogeneous and complex regions equally. This can lead to suboptimal segmentation results, especially in complex images with multiple objects. We address this problem by proposing an SAR image segmentation algorithm based on complicated region-sensitive adaptive superpixel generation and hybrid edge correction (RSASGEC). First, a dynamic initialization algorithm for superpixel seeds based on region complexity is designed. Specifically, a new superpixel representation structure for superpixel seeds is constructed by combining superpixel complexity and the number of contained pixels. The algorithm gives priority to regions with high complexity, dynamically selecting the region with the highest complexity for further partitioning. This results in a dense distribution of superpixel seeds in complex regions, and sparse distributions in homogeneous regions with low complexity. Second, an iterative superpixel segmentation process based on an adaptive energy function is proposed. The Lagrange multiplier mathematical strategy is employed to optimize the adaptive energy function within an adjustable search window, resulting in more compact superpixel segmentation. Finally, a label correction method, based on edge mixture model constraints, is proposed for postprocessing. By integrating edge information from the Gaussian edge detector and the Canny algorithm as constraints, this method leverages majority voting and region growth methods to mitigate edge noise and outliers, refining the superpixel labels. The RSASGEC algorithm is verified in experiments, using one simulated image and six real SAR images. The results indicate that RSASGEC outperforms six representative algorithms, achieving more satisfactory segmentation performance.

This paper presents a spectral domain registration-based visual servoing scheme that works on 3D point clouds. Specifically, we propose a 3D model/point cloud alignment method, which works by finding a global transformation between reference and target point clouds using spectral analysis. A 3D Fast Fourier Transform (FFT) in$\mathbb{R}^{3}$is used for the translation estimation, and the real spherical harmonics in$\boldsymbol{SO}(3)$are used for the rotations estimation. Such an approach allows us to derive a decoupled 6 degrees of freedom (DoF) controller, where we use gradient ascent optimisation to minimise translation and rotational costs. We then show how this methodology can be used to regulate a robot arm to perform a positioning task. In contrast to the existing state-of-the-art depth-based visual servoing methods that either require dense depth maps or dense point clouds, our method works well with partial point clouds and can effectively handle larger transformations between the reference and the target positions. Furthermore, the use of spectral data (instead of spatial data) for transformation estimation makes our method robust to sensor-induced noise and partial occlusions. We validate our approach by performing experiments using point clouds acquired by a robot-mounted depth camera. Obtained results demonstrate the effectiveness of our visual servoing approach.

This paper presents an assisted telemanipulation framework for reaching and grasping desired objects from clutter. Specifically, the developed system allows an operator to select an object from a cluttered heap and effortlessly grasp it, with the system assisting in selecting the best grasp and guiding the operator to reach it. To this end, we propose an object pose estimation scheme, a dynamic grasp re-ranking strategy, and a reach-to-grasp hybrid force/position trajectory guidance controller. We integrate them, along with our previous Spect-G RASP grasp planner, into a classical bilateral teleoperation system that allows to control the robot using a haptic device while providing force feedback to the operator. For a user-selected object, our system first identifies the object in the heap and estimates its full six degrees of freedom (DoF) pose. Then, SpectGRASP generates a set of ordered, collision-free grasps for this object. Based on the current location of the robot gripper, the proposed grasp re-ranking strategy dynamically updates the best grasp. In assisted mode, the hybrid controller generates a zero force-torque path along the reach-to-grasp trajectory while automatically controlling the orientation of the robot. We conducted real-world experiments using a haptic device and a 7-DoF cobot with a 2-finger gripper to validate individual components of our telemanipulation system and its overall functionality. Obtained results demonstrate the effectiveness of our system in assisting humans to clear cluttered scenes.

Adjusting the level of autonomy in human-machine systems (e.g., human-robot systems) holds great potential for achieving high system performance while maintaining operator involvement. To support operators with the task of setting the proper level of autonomy, we present a novel approach to realise a Model Predictive Controller that determines the optimal LoA for each tessellation in the robot's path plan based on the estimated performance degradation due environmental adversities. We also report on an experimental evaluation of a mixed-initiative system where both the operator and the Model Predictive Controller are in charge of dynamically adjusting the level of autonomy cooperatively while performing a challenging navigational task with a mobile ground robot in a high-fidelity simulation. To this end, we conducted a user study with 15 participants comparing the performance and user experience of the model predictive system with a state-of-the-art system. The results show significant benefits of the model predictive system in terms of a reduction of conflicts for control and an improved user experience. Additionally, there are indications of benefits in terms of robot health and, consequently, performance for the model predictive system.

Existing fusion-based local community detection algorithms have achieved good results. However, when assigning a node to a community, similarity functions are sometimes used, which only use node information, while ignoring connection information within the community. These algorithms sometimes fail to find influential nodes, which eventually leads to the failure to find a complete local community. To address these problems, a new local community detection algorithm is proposed in this article. Two strategies, of strong fusion followed by weak fusion, are used alternately to fuse nodes. Compared with using two fusion strategies alone, the alternating loop method can improve the solution of the algorithm in each stage. In strong fusion, we propose a new membership function that considers both node information and connection information in the local community. This improves the quality of the fused node while preserving the structure of the current community. In weak fusion, we propose a parameter-based similarity measure, which can detect influential nodes for a local community. We also propose a local community evaluation metric, which does not require true division to determine the optimal local community under different parameters. Experiments, compared to six state-of-the-art algorithms, show that the proposed algorithm improves accuracy and stability, and also demonstrate the effectiveness of the new local community evaluation metrics in parameter selection.

Handling object deformations for robotic grasping is still a major problem to solve. In this paper, we propose an efficient learning-free solution for this problem where generated grasp hypotheses of a region of an object are adapted to its deformed configurations. To this end, we investigate the applicability of functional map (FM) correspondence, where the shape matching problem is treated as searching for correspondences between geometric functions in a reduced basis. For a user selected region of an object, a ranked list of grasp candidates is generated with local contact moment (LoCoMo) based grasp planner. The proposed FM-based methodology maps these candidates to an instance of the object that has suffered arbitrary level of deformation. The best grasp, by analysing its kinematic feasibility while respecting the original finger configuration as much as possible, is then executed on the object. We have compared the performance of our method with two different state-of-the-art correspondence mapping techniques in terms of grasp stability and region grasping accuracy for 4 different objects with 5 different deformations.

This paper reports on insights by robotics researchers that participated in a 5-day robot-assisted nuclear disaster response field exercise conducted by Kerntechnische Hilfdienst GmbH (KHG) in Karlsruhe, Germany. The German nuclear industry established KHG to provide a robot-assisted emergency response capability for nuclear accidents. We present a systematic description of the equipment used; the robot operators' training program; the field exercise and robot tasks; and the protocols followed during the exercise. Additionally, we provide insights and suggestions for advancing disaster response robotics based on these observations. Specifically, the main degradation in performance comes from the cognitive and attentional demands on the operator. Furthermore, robotic platforms and modules should aim to be robust and reliable in addition to their ease of use. Last, as emergency response stakeholders are often skeptical about using autonomous systems, we suggest adopting a variable autonomy paradigm to integrate autonomous robotic capabilities with the human-in-the-loop gradually. This middle ground between teleoperation and autonomy can increase end-user acceptance while directly alleviating some of the operator's robot control burden and maintaining the resilience of the human-in-the-loop.

This paper addresses the problem of transfer of control authority between a robot’s AI and a remote human operator, when controlling a Mixed-Initiative (MI) robotic system. We propose a negotiation-theoretic method that enables the robot’s AI and the human operator to cooperatively and dynamically determine (i. e. negotiate) the transfer of control authority between these two agents. An experimental study is presented in which a state-of-the-art Expert-guided Mixed-Initiative Control Switcher (EMICS) method is compared with our proposed Negotiation-Enabled Mixed-Initiative Control Switcher (NEMICS) algorithm. Results suggest that the NEMICS framework is able to successfully avoid conflicts for control, which is a fundamental challenge encountered with previous MI control methods. Comparing NEMICS with the EMICS, we provide evidence of improved navigational safety (i. e. fewer collisions). Additionally, our usability study suggests that human operators perceived their interactions with NEMICS as less intrusive than with EMICS.

In the information age of big data, and increasingly large and complex networks, there is a growing challenge of understanding how best to restrain the spread of harmful information, for example, a computer virus. Establishing models of propagation and node immunity are important parts of this problem. In this article, a dynamic node immune model, based on the community structure and threshold (NICT), is proposed. First, a network model is established, which regards nodes carrying harmful information as new nodes in the network. The method of establishing the edge between the new node and the original node can be changed according to the needs of different networks. The propagation probability between nodes is determined by using community structure information and a similarity function between nodes. Second, an improved immune gain, based on the propagation probability of the community structure and node similarity, is proposed. The improved immune gain value is calculated for neighbors of the infected node at each time step, and the node is immunized according to the hand-coded parameter: immune threshold. This can effectively prevent invalid or insufficient immunization at each time step. Finally, an evaluation index, considering both the number of immune nodes and the number of infected nodes at each time step, is proposed. The immune effect of nodes can be evaluated more effectively. The results of network immunization experiments, on eight real networks, suggest that the proposed method can deliver better network immunization than several other well-known methods from the literature.

This paper proposes a novel segmentation algorithm for synthetic aperture radar (SAR) images. The algorithm performs region-level segmentation based on edge feature and label assistance (REFLA). It demonstrates improved performance in terms of segmentation accuracy while better preserving image edges. Firstly, an edge detection scheme is implemented, which fuses information from two advanced edge detection methods, thereby obtaining a more precise edge strength map (ESM). Secondly, a Canny algorithm is performed to divide the SAR image into edge regions and homogeneous regions, and different smoothing templates are selected according to pixel positions. Therefore, an anisotropic smoothing on the SAR image can be achieved, aiming at suppressing the noise within targets while also accurately maintaining the target boundaries. Thirdly, K-means clustering is applied on the smoothed result, to generate an initial set of labels. Using ESM and the initial labels as inputs, a watershed transformation and a majority voting strategy are employed to realize an initial segmentation at the region level. Finally, a label-aided region merging (LaRM) strategy is used to correctly segment the wrongly labeled regions, to give the final segmentation result. The LaRM, with merging rules based on label rather than gray characteristics, can avoid the need for calculating a large number of complex formulae, thus accelerating the region merging. Results are presented of experiments, on both simulated and real SAR images, in which the proposed REFLA method is compared against six state-of-the-art algorithms from the literature. REFLA achieves higher accuracy, while better retaining the image edges.

Synthetic aperture radar (SAR) is widely used in the field of modern remote sensing due to its high resolution for a comparatively small antenna. However, there are still some difficulties in the processing of SAR images. In particular, accurate segmentation of small targets and image corners remains an important challenge, as these can easily be lost during conventional image smoothing and denoising methods. To address this, we propose an SAR image segmentation algorithm based on constrained smoothing and hierarchical label correction (CSHLC). First, a Canny algorithm is used to extract the edges of SAR images, and the Gaussian smoothing is performed on SAR images under edge constraints to achieve noise reduction so that the edges of small and big targets are well preserved. Second, a preliminary K-means clustering is conducted on the smoothing results, and then, a Markov random field (MRF) model is used on the clustering results (“original label” results), iteratively calculating a maximum likelihood set of pixel labels. Finally, through two label correction methods, pixel group counting comparison (PGCC) and gray similarity comparison (GSC), the labels of the MRF output are further checked and corrected to obtain final segmentation results. Compared with seven state-of-the-art algorithms, simulation results on both simulated SAR images and real SAR images show that the proposed CSHLC delivers higher accuracy while better retaining corners and small targets.

This paper presents a spectral correlation-based method (SpectGRASP) for robotic grasping of arbitrarily shaped, unknown objects. Given a point cloud of an object, SpectGRASP extracts contact points on the object’s surface matching the hand configuration. It neither requires offline training nor a-priori object models. We propose a novel Binary Extended Gaussian Image (BEGI), which represents the point cloud surface normals of both object and robot fingers as signals on a 2-sphere. Spherical harmonics are then used to estimate the correlation between fingers and object BEGIs. The resulting spectral correlation density function provides a similarity measure of gripper and object surface normals. This is highly efficient in that it is simultaneously evaluated at all possible finger rotations in SO(3). A set of contact points are then extracted for each finger using rotations with high correlation values. We then use our previous work, Local Contact Moment (LoCoMo) similarity metric, to sequentially rank the generated grasps such that the one with maximum likelihood is executed. We evaluate the performance of SpectGRASP by conducting experiments with a 7-axis robot fitted with a parallel-jaw gripper, in a physics simulation environment. Obtained results indicate that the method not only can grasp individual objects, but also can successfully clear randomly organized groups of objects. The SpectGRASP method also outperforms the closest state-of-the-art method in terms of grasp generation time and grasp-efficiency.

This paper investigates how trust, shared under-standing between a human operator and a robot, and the Locus of Control (LoC) personality trait, evolve and affect Human-Robot Interaction (HRI) in mixed-initiative robotic systems. As such systems become more advanced and able to instigate actions alongside human operators, there is a shift from robots being perceived as a tool to being a team-mate. Hence, the team-oriented human factors investigated in this paper (i.e. trust, shared understanding, and LoC) can play a crucial role in efficient HRI. Here, we present the results from an experiment inspired by a disaster response scenario in which operators remotely controlled a mobile robot in navigation tasks, with either human-initiative or mixed-initiative control, switching dynamically between two different levels of autonomy: teleoperation and autonomous navigation. Evidence suggests that operators trusted and developed an understanding of the robotic systems, especially in mixed-initiative control, where trust and understanding increased over time, as operators became more familiar with the system and more capable of performing the task. Lastly, evidence and insights are presented on how LoC affects HRI.

This paper addresses the problem of the human operator cognitive workload estimation while controlling a robot. Being capable of assessing, in real-time, the operator’s workload could help prevent calamitous events from occurring. This workload estimation could enable an AI to make informed decisions to assist or advise the operator, in an advanced human-robot interaction framework. We propose a method, named Fessonia, for real-time cognitive workload estimation from multiple parameters of an operator’s driving behaviour via the use of behavioural entropy. Fessonia is comprised of: a method to calculate the entropy (i.e. unpredictability) of the operator driving behaviour profile; the Driver Profile Update algorithm which adapts the entropy calculations to the evolving driving profile of individual operators; and a Warning And Indication System that uses workload estimations to issue advice to the operator. Fessonia is evaluated in a robot teleoperation scenario that incorporated cognitively demanding secondary tasks to induce varying degrees of workload. The results demonstrate the ability of Fessonia to estimate different levels of imposed workload. Additionally, it is demonstrated that our approach is able to detect and adapt to the evolving driving profile of the different operators. Lastly, based on data obtained, a decrease in entropy is observed when a warning indication is issued, suggesting a more attentive approach focused on the primary navigation task.

This paper addresses the problem of human operator intent recognition during teleoperated robot navigation. In this context, recognition of the operator’s intended navigational goal, could enable an artificial intelligence (AI) agent to assist the operator in an advanced human-robot interaction framework. We propose a Bayesian Operator Intent Recognition (BOIR) probabilistic method that utilizes: (i) an observation model that fuses information as a weighting combination of multiple observation sources providing geometric information; (ii) a transition model that indicates the evolution of the state; and (iii) an action model, the Active Intent Recognition Model (AIRM), that enables the operator to communicate their explicit intent asynchronously. The proposed method is evaluated in an experiment where operators controlling a remote mobile robot are tasked with navigation and exploration under various scenarios with different map and obstacle layouts. Results demonstrate that BOIR outperforms two related methods from literature in terms of accuracy and uncertainty of the intent recognition.

This article presents an Expert-guided Mixed-initiative Control Switcher (EMICS) for remotely operated mobile robots. The EMICS enables switching between different levels of autonomy during task execution initiated by either the human operator and/or the EMICS. The EMICS is evaluated in two disaster-response-inspired experiments, one with a simulated robot and test arena, and one with a real robot in a realistic environment. Analyses from the two experiments provide evidence that: (a) Human-Initiative (HI) systems outperform systems with single modes of operation, such as pure teleoperation, in navigation tasks; (b) in the context of the simulated robot experiment, Mixed-initiative (MI) systems provide improved performance in navigation tasks, improved operator performance in cognitive demanding secondary tasks, and improved operator workload compared to HI. Last, our experiment on a physical robot provides empirical evidence that identify two major challenges for MI control: (a) the design of context-aware MI control systems; and (b) the conflict for control between the robot’s MI control system and the operator. Insights regarding these challenges are discussed and ways to tackle them are proposed.

Raw polarimetric images are captured by a focal plane polarimeter which is covered by a micro-polarizer array (MPA). The design of the MPA plays a crucial role in polarimetric imaging. MPAs are predominantly designed according to expert engineering experience and rules of thumb. Typically, only one optimization criterion, maximizing bandwidth, is used to design the MPA. To select a design, an exhaustive search is usually performed on a very limited set of available polarizing patterns, which must be constrained in order to make the search tractable. In contrast, this paper proposes a fully automated and optimal MPA design method (AO-MPA) which generates significantly improved MPAs. Instead of the single criterion of bandwidth, we propose six design principles, and show how they can be utilized to mutually optimize the MPA design by formulating a tri-objective optimization problem with multiple constraints. A much larger set of possible MPA patterns is rapidly and automatically searched by applying advanced multi-objective optimization techniques. We have tested AO-MPA using two groups of experiments, in which AO-MPA is compared against several other leading MPA design methods, and the patterns generated by AO-MPA are compared against state-of-the-art patterns from the literature. The results, obtained using a public benchmark dataset, show that the AO-MPA method is very computationally efficient, and can find all optimal MPA patterns for all array sizes. Moreover, for each size, AO-MPA obtains all optimal layouts simultaneously. AO-MPA generates designs which require fewer polarization orientations, while also yielding better performance in estimating intensity measurements, Stokes vector and the degree of linear polarization. This results in MPAs which are easier to manufacture while also being more robust to noise.

Estimating the inertial properties of an object can make robotic manipulations more efficient, especially in extreme environments. This paper presents a novel method of estimating the 2D inertial parameters of an object, by having a robot applying a push on it. We draw inspiration from previous analyses on quasi-static pushing mechanics, and introduce a data-driven model that can accurately represent these mechanics and provide a prediction for the object's inertial parameters. We evaluate the model with two datasets. For the first dataset, we set up a V-REP simulation of seven robots pushing objects with large range of inertial parameters, acquiring 48000 pushes in total. For the second dataset, we use the object pushes from the MIT M-Cube lab pushing dataset. We extract features from force, moment and velocity measurements of the pushes, and train a Multi-Output Regression Random Forest. The experimental results show that we can accurately predict the 2D inertial parameters from a single push, and that our method retains this robust performance under various surface types.

This paper presents a path planner, which enables a nonholonomic mobile manipulator to move its end-effector on an observed surface with a constrained orientation, given start and destination points. A partial point cloud of the environment is captured using a vision-based sensor, but no prior knowledge of the surface shape is assumed. We consider the multi-objective optimisation problem of finding robot paths which account for the nonholonomic constraints of the base, maximise the robot's manipulability throughout the motion, while also minimising surface-distance travelled between the two points. This work has application in industrial problems of rough robotic cutting, e.g. demolition of legacy nuclear plants, where dismantling does not require a precise path. We show how our approach embeds the nonholonomic constraints of the mobile platform into an extended Jacobian, while additionally encoding the constraint that the end-effector must remain in contact with the cut surface throughout the motion. We use this constrained Jacobian to plan a time-series of robot configurations. Additionally, we show how our novel cost function is suitable for combining with a variety of well-known path planners, such as RRT*. We present several empirical experiments in different scenarios, where a simulated non-holonomic mobile manipulator follows a trajectory, which is generated on noisy point clouds derived from real depth-camera images of real objects. Our planner (RRT*-CRMM) enables successful task completion by optimising the path over the travelled distance, the manipulability of the arm, and the movements of the mobile base.

Semi-supervised non-negative matrix factorization (NMF) exploits the strengths of NMF in effectively learning local information contained in data and is also able to achieve effective learning when only a small fraction of data is labeled. NMF is particularly useful for dimensionality reduction of high-dimensional data. However, the mapping between the low-dimensional representation, learned by semi-supervised NMF, and the original high-dimensional data contains complex hierarchical and structural information, which is hard to extract by using only single-layer clustering methods. Therefore, in this article, we propose a new deep learning method, called semi-supervised graph regularized deep NMF with bi-orthogonal constraints (SGDNMF). SGDNMF learns a representation from the hidden layers of a deep network for clustering, which contains varied and unknown attributes. Bi-orthogonal constraints on two factor matrices are introduced into our SGDNMF model, which can make the solution unique and improve clustering performance. This improves the effect of dimensionality reduction because it only requires a small fraction of data to be labeled. In addition, SGDNMF incorporates dual-hypergraph Laplacian regularization, which can reinforce high-order relationships in both data and feature spaces and fully retain the intrinsic geometric structure of the original data. This article presents the details of the SGDNMF algorithm, including the objective function and the iterative updating rules. Empirical experiments on four different data sets demonstrate state-of-the-art performance of SGDNMF in comparison with six other prominent algorithms.

This paper presents an assisted telemanipulation approach with integrated grasp planning. It also studies how the human teleoperation performance benefits from the incorporated visual and haptic cues while manipulating objects in cluttered environments. The developed system combines the widely used master-slave teleoperation with our previous model-free and learning-free grasping algorithm by means of a dynamic grasp re-ranking strategy and a semi-autonomous reach-to-grasptrajectory guidance. The proposed re-ranking metric helps in dynamically updating the stable grasps based on the current state of the slave device. The trajectory guidance system assists in maintaining smooth trajectory by controlling the haptic forces. A virtual pose controller has been integrated with the guidance scheme to automatically correct the end-effector orientation while reaching towards the grasp. Various experiments are conducted evaluating the proposed method using a six degrees of freedom (dof) haptic master and a seven dof slave robot. Results obtained with these tests along with the results gathered from the performed human-factor trials demonstrate the efficiency of our method in terms of objective metrics of task completion, and also subjective metrics of user experience.

Conventional teleoperative interfaces, used for remote manipulation in the nuclear industry, utilise multiple stationary CCTV cameras for situational awareness. This is cognitively challenging for the human operator, who must mentally combine information from multiple 2D views, before attempting 3D reasoning about the remote environment. To enhance performance on telemanipulation tasks, this paper explores the merits of employing an orbital camera view. The orbital camera's gaze direction is automatically fixated towards the robot's moving end-effector. Meanwhile, the camera can be controlled by the operator to move on a spherical surface around the end-effector, in addition to zooming capability. The robot's Cartesian motion coordinates are also continuously transformed to be controllable with respect to the axes of the orbital camera view. Human test-subject experiments were conducted in a simulation environment using novice robot operators, and a variety of metrics were collected. Although the results do not show an objective difference between the camera modes, participant interviews suggest the orbital camera is the preferred method for visual feedback with its dynamic views helping to overcome positioning difficulties associated with stationary cameras. The experiment also revealed significant confounding factors that contributed to the results being inconclusive. These are discussed and recommendations are made for future empirical experiments to evaluate such systems.

This paper investigates learning effects and human operator training practices in variable autonomy robotic systems. These factors are known to affect performance of a human-robot system and are frequently overlooked. We present the results from an experiment inspired by a search and rescue scenario in which operators remotely controlled a mobile robot with either Human-Initiative (HI) or Mixed-Initiative (MI) control. Evidence suggests learning in terms of primary navigation task and secondary (distractor) task performance. Further evidence is provided that MI and HI performance in a pure navigation task is equal. Lastly, guidelines are proposed for experimental design and operator training practices.

This paper addresses the problem of grasping arbitrarily shaped objects, observed as partial point-clouds, without requiring: models of the objects, physics parameters, training data, or other a-priori knowledge. A grasp metric is proposed based on Local Contact Moment (LoCoMo). LoCoMo combines zero-moment shift features, of both hand and object surface patches, to determine local similarity. This metric is then used to search for a set of feasible grasp poses with associated grasp likelihoods. LoCoMo overcomes some limitations of both classical grasp planners and learning-based approaches. Unlike force-closure analysis, LoCoMo does not require knowledge of physical parameters such as friction coefficients, and avoids assumptions about fingertip contacts, instead enabling robust contacts of large areas of hand and object surface. Unlike more recent learning-based approaches, LoCoMo does not require training data, and does not need any prototype grasp configurations to be taught by kinesthetic demonstration. We present results of real-robot experiments grasping 21 different objects, observed by a wrist-mounted depth camera. All objects are grasped successfully when presented to the robot individually. The robot also successfully clears cluttered heaps of objects by sequentially grasping and lifting objects until none remain.

This paper introduces a fully deep learning approach to monocular SLAM, which can perform simultaneous localization using a neural network for learning visual odometry (L-VO) and dense 3D mapping. Dense 2D flow and a depth image are generated from monocular images by sub-networks, which are then used by a 3D flow associated layer in the L-VO network to generate dense 3D flow. Given this 3D flow, the dual-stream L-VO network can then predict the 6DOF relative pose and furthermore reconstruct the vehicle trajectory. In order to learn the correlation between motion directions, the Bivariate Gaussian modeling is employed in the loss function. The L-VO network achieves an overall performance of 2.68 % for average translational error and 0.0143°/m for average rotational error on the KITTI odometry benchmark. Moreover, the learned depth is leveraged to generate a dense 3D map. As a result, an entire visual SLAM system, that is, learning monocular odometry combined with dense 3D mapping, is achieved.

Feature selection is an important approach for reducing the dimension of high-dimensional data. In recent years, many feature selection algorithms have been proposed, but most of them only exploit information from the data space. They often neglect useful information contained in the feature space, and do not make full use of the characteristics of the data. To overcome this problem, this paper proposes a new unsupervised feature selection algorithm, called non-negative spectral learning and sparse regression-based dual-graph regularized feature selection (NSSRD). NSSRD is based on the feature selection framework of joint embedding learning and sparse regression, but extends this framework by introducing the feature graph. By using low dimensional embedding learning in both data space and feature space, NSSRD simultaneously exploits the geometric information of both spaces. Second, the algorithm uses non-negative constraints to constrain the low-dimensional embedding matrix of both feature space and data space, ensuring that the elements in the matrix are non-negative. Third, NSSRD unifies the embedding matrix of the feature space and the sparse transformation matrix. To ensure the sparsity of the feature array, the sparse transformation matrix is constrained using the -norm. Thus feature selection can obtain accurate discriminative information from these matrices. Finally, NSSRD uses an iterative and alternative updating rule to optimize the objective function, enabling it to select the representative features more quickly and efficiently. This paper explains the objective function, the iterative updating rules and a proof of convergence. Experimental results show that NSSRD is significantly more effective than several other feature selection algorithms from the literature, on a variety of test data.

This paper presents a novel robust method for single target tracking in RGB-D images, and also contributes a substantial new benchmark dataset for evaluating RGB-D trackers. While a target object's color distribution is reasonably motion-invariant, this is not true for the target's depth distribution, which continually varies as the target moves relative to the camera. It is therefore nontrivial to design target models which can fully exploit (potentially very rich) depth information for target tracking. For this reason, much of the previous RGB-D literature relies on color information for tracking, while exploiting depth information only for occlusion reasoning. In contrast, we propose an adaptive range-invariant target depth model, and show how both depth and color information can be fully and adaptively fused during the search for the target in each new RGB-D image. We introduce a new, hierarchical, two-layered target model (comprising local and global models) which uses spatio-temporal consistency constraints to achieve stable and robust on-the-fly target relearning. In the global layer, multiple features, derived from both color and depth data, are adaptively fused to find a candidate target region. In ambiguous frames, where one or more features disagree, this global candidate region is further decomposed into smaller local candidate regions for matching to local-layer models of small target parts. We also note that conventional use of depth data, for occlusion reasoning, can easily trigger false occlusion detections when the target moves rapidly toward the camera. To overcome this problem, we show how combining target information with contextual information enables the target's depth constraint to be relaxed. Our adaptively relaxed depth constraints can robustly accommodate large and rapid target motion in the depth direction, while still enabling the use of depth data for highly accurate reasoning about occlusions. For evaluation, we introduce a new RGB-D benchmark dataset with per-frame annotated attributes and extensive bias analysis. Our tracker is evaluated using two different state-of-the-art methodologies, VOT and object tracking benchmark, and in both cases it significantly outperforms four other state-of-the-art RGB-D trackers from the literature.

This paper proposes a novel method based on deformable dictionary learning for detecting the regions of change between multitemporal image pairs. We build on our previous work, which constructed a pair of dictionaries. The main shortcoming of this method was its dependence on a large amount of training data. In practice, there is often a shortage of ground-truthed training images, which limits the expression capability of the resulting dictionaries. This paper overcomes this challenge by incorporating the concept of deformation, wherein each atom of a dictionary is no longer a simple image patch, but instead is a flexible image deformation function. This enables the creation of more expressive dictionaries, capable of generalizing to a far greater variety of image patterns, while using a far smaller amount of ground-truthed images for supervised dictionary training. Deformation similarity is employed for patch matching to find the best set of atoms in the difference image (DI) dictionary for reconstructing image patches for a new input DI. Each such atom can be deformed to achieve a better match, thus extending generality while reducing the number of atoms needed in the dictionary. Multiple deformed atoms are weighted and combined to best reconstruct the input DI patch. Then, the same set of deformations and weights is projected to the corresponding atoms in the CD dictionary to obtain the output change-detection map. Experiments in six realistic synthetic aperture radar data sets demonstrate the robustness and efficiency of the proposed method in comparison with five other state-of-the-art methods from the literature.

This paper addresses the problem of mixed initiative, shared control for master-slave grasping and manipulation. We propose a novel system, in which an autonomous agent assists a human in teleoperating a remote slave arm/gripper, using a haptic master device. Our system is designed to exploit the human operator's expertise in selecting stable grasps (still an open research topic in autonomous robotics). Meanwhile, a-priori knowledge of: i) the slave robot kinematics, and ii) the desired post-grasp manipulative trajectory, are fed to an autonomous agent which transmits force cues to the human, to encourage maximally manipulable grasp pose selections. Specifically, the autonomous agent provides force cues to the human, during the reach-to-grasp phase, which encourage the human to select grasp poses which maximise manipulation capability during the post-grasp object manipulation phase. We introduce a task-oriented velocity manipulability cost function (TOV), which is used to identify the maximum kinematic capability of a manipulator during post-grasp motions, and feed this back as force cues to the human during the pre-grasp phase. We show that grasps which minimise TOV result in significantly reduced control effort of the manipulator, compared to other feasible grasps. We demonstrate the effectiveness of our approach by experiments with both real and simulated robots.

This paper addresses the problem of selecting from a choice of possible grasps, so that impact forces will be minimised if a collision occurs while the robot is moving the grasped object along a post-grasp trajectory. Such considerations are important for safety in human-robot interaction, where even a certified “human-safe” (e.g. compliant) arm may become hazardous once it grasps and begins moving an object, which may have significant mass, sharp edges or other dangers. Additionally, minimising collision forces is critical to preserving longevity of robots which operate in uncertain and hazardous environments, e.g. robots deployed for nuclear decommissioning, where removing a damaged robot from a contaminated zone for repairs may be extremely difficult and costly. Also, unwanted collisions between a robot and critical infrastructure (e.g. pipework) in such high-consequence environments can be disastrous. In this paper we investigate how the safety of the post-grasp motion can be considered during the pre-grasp approach phase, so that the selected grasp is optimal in terms of applying minimum impact forces if a collision occurs during the desired post-grasp manipulation. We build on the methods of augmented robot-object dynamic model and “effective mass” and propose a method for combining these concepts with modern grasp and trajectory planners, to enable the robot to achieve a grasp which maximises the safety of the post-grasp trajectory, by minimising potential collision forces. We demonstrate the effectiveness of our approach through several experiments with both simulated and real robots.

This paper proposes a single-shot approach for recognising clothing categories from 2.5D features. We propose two visual features, BSP (B-Spline Patch) and TSD (Topology Spatial Distances) for this task. The local BSP features are encoded by LLC (Locality-constrained Linear Coding) and fused with three different global features. Our visual feature is robust to deformable shapes and our approach is able to recognise the category of unknown clothing in unconstrained and random configurations. We integrated the category recognition pipeline with a stereo vision system, clothing instance detection, and dual-arm manipulators to achieve an autonomous sorting system. To verify the performance of our proposed method, we build a high-resolution RGBD clothing dataset of 50 clothing items of 5 categories sampled in random configurations (a total of 2,100 clothing samples). Experimental results show that our approach is able to reach 83.2% accuracy while classifying clothing items which were previously unseen during training. This advances beyond the previous state-of-the-art by 36.2%. Finally, we evaluate the proposed approach in an autonomous robot sorting system, in which the robot recognises a clothing item from an unconstrained pile, grasps it, and sorts it into a box according to its category. Our proposed sorting system achieves reasonable sorting success rates with single-shot perception.

Nonnegative matrix factorization (NMF) is well known to be an effective tool for dimensionality reduction in problems involving big data. For this reason, it frequently appears in many areas of scientific and engineering literature. This letter proposes a novel semisupervised NMF algorithm for overcoming a variety of problems associated with NMF algorithms, including poor use of prior information, negative impact on manifold structure of the sparse constraint, and inaccurate graph construction. Our proposed algorithm, nonnegative matrix factorization with rank regularization and hard constraint (NMFRC), incorporates label information into data representation as a hard constraint, which makes full use of prior information. NMFRC also measures pairwise similarity according to geodesic distance rather than Euclidean distance. This results in more accurate measurement of pairwise relationships, resulting in more effective manifold information. Furthermore, NMFRC adopts rank constraint instead of norm constraints for regularization to balance the sparseness and smoothness of data. In this way, the new data representation is more representative and has better interpretability. Experiments on real data sets suggest that NMFRC outperforms four other state-of-the-art algorithms in terms of clustering accuracy.

This paper presents a principled experimental analysis of a variable autonomy control approach to mobile robot navigation. A Human-Initiative (HI) variable autonomy system is investigated, in which a human operator is able to switch the Level of Autonomy (LOA) between teleoperation (joystick control) and autonomous control (robot navigates autonomously towards waypoints selected by the human) on-the-fly. Our hypothesis is that the HI system will enable superior navigation performance compared to either teleoperation or autonomy alone, especially in scenarios where the performance of both the human and the robot may at times become degraded. We evaluate our hypothesis through carefully controlled and repeatable experiments using a significant number of human test-subjects.

This paper addresses the problem of jointly planning both grasps and subsequent manipulative actions. Previously, these two problems have typically been studied in isolation, however joint reasoning is essential to enable robots to complete real manipulative tasks. In this paper, the two problems are addressed jointly and a solution that takes both into consideration is proposed. To do so, a manipulation capability index is defined, which is a function of both the task execution waypoints and the object grasping contact points. We build on recent state-of-the-art grasp-learning methods, to show how this index can be combined with a likelihood function computed by a probabilistic model of grasp selection, enabling the planning of grasps which have a high likelihood of being stable, but which also maximise the robot's capability to deliver a desired post-grasp task trajectory. We also show how this paradigm can be extended, from a single arm and hand, to enable efficient grasping and manipulation with a bi-manual robot. We demonstrate the effectiveness of the approach using experiments on a simulated as well as a real robot.

This paper presents a vision-based approach for estimating the configuration of, and providing control signals for, an under-sensored robot manipulator using a single monocular camera. Some remote manipulators, used for decommissioning tasks in the nuclear industry, lack proprioceptive sensors because electronics are vulnerable to radiation. Additionally, even if proprioceptive joint sensors could be retrofitted, such heavy-duty manipulators are often deployed on mobile vehicle platforms, which are significantly and erratically perturbed when powerful hydraulic drilling or cutting tools are deployed at the end-effector. In these scenarios, it would be beneficial to use external sensory information, e.g. vision, for estimating the robot configuration with respect to the scene or task. Conventional visual servoing methods typically rely on joint encoder values for controlling the robot. In contrast, our framework assumes that no joint encoders are available, and estimates the robot configuration by visually tracking several parts of the robot, and then enforcing equality between a set of transformation matrices which relate the frames of the camera, world and tracked robot parts. To accomplish this, we propose two alternative methods based on optimisation. We evaluate the performance of our developed framework by visually tracking the pose of a conventional robot arm, where the joint encoders are used to provide ground-truth for evaluating the precision of the vision system. Additionally, we evaluate the precision with which visual feedback can be used to control the robot's end-effector to follow a desired trajectory.

The capacitated arc routing problem (CARP) has attracted considerable attention from researchers due to its broad potential for social applications. This paper builds on, and develops beyond, the cooperative coevolutionary algorithm based on route distance grouping (RDG-MAENS), recently proposed by Mei et al. Although Mei's method has proved superior to previous algorithms, we discuss several remaining drawbacks and propose solutions to overcome them. First, although RDG is used in searching for potential better solutions, the solution generated from the decomposed problem at each generation is not the best one, and the best solution found so far is not used for solving the current generation. Second, to determine which sub-population the individual belongs to simply according to the distance can lead to an imbalance in the number of the individuals among different sub-populations and the allocation of resources. Third, the method of Mei et al. was only used to solve single-objective CARP. To overcome the above issues, this paper proposes improving RDG-MAENS by updating the solutions immediately and applying them to solve the current solution through areas shared, and then according to the magnitude of the vector of the route direction, and a fast and simple allocation scheme is proposed to determine which decomposed problem the route belongs to. Finally, we combine the improved algorithm with an improved decomposition-based memetic algorithm to solve the multiobjective large scale CARP (LSCARP). Experimental results suggest that the proposed improved algorithm can achieve better results on both single-objective LSCARP and multiobjective LSCARP.

This paper presents a method for single target tracking of arbitrary objects in challenging video sequences. Targets are modeled at three different levels of granularity (pixel level, parts-based level and bounding box level), which are cross-constrained to enable robust model relearning. The main contribution is an adaptive clustered decision tree method which dynamically selects the minimum combination of features necessary to sufficiently represent each target part at each frame, thereby providing robustness with computational efficiency. The adaptive clustered decision tree is implemented in two separate parts of the tracking algorithm: firstly to enable robust matching at the parts-based level between successive frames; and secondly to select the best superpixels for learning new parts of the target. We have tested the tracker using two different tracking benchmarks (VOT2013-2014 and CVPR2013 tracking challenges), based on two different test methodologies, and show it to be significantly more robust than the best state-of-the-art methods from both of those tracking challenges, while also offering competitive tracking precision.

This paper addresses the problem of constrained motion for a manipulator performing a task while in contact with the environment, and investigates two force control frameworks, one based on projected inverse dynamics, and one based on optimal control. Firstly, we propose a control method based on projected inverse dynamics, which directly exploits the contact constraints to minimise the instantaneous joint torques needed to perform a task. Secondly, we propose an optimal control strategy which provides a tool to minimise the joint torques over an interval of time. We show how contact constraints can be used as optimisation constraints in the definition of the problem, and how to formulate the optimal control problem directly using projected dynamics. Initially we explore a positional control problem, where the robot is required to follow a desired path, and show that both of the proposed methods can satisfy the positional task while significantly reducing the joint torques as compared to simple kinematic control and also classical inverse dynamics control. We also show that the proposed optimal control method outperforms the pure projected inverse dynamics method in terms of minimising the required joint torques. We then show how each method can be extended to follow a desired path while also exerting a desired contact force. Again, the method incorporating optimal control is shown to satisfy the task requirements with significantly smaller commanded torques than the pure projected inverse dynamics method. To confirm the analysis, and demonstrate proof of concept, we present the results of empirical experiments with a simulated 3-degree-of-freedom planar manipulator which is constrained to move while in contact with a rigid surface.

Safety critical and demanding tasks (e.g. Search and rescue or hazardous environments inspection), can benefit from robotic systems that offer a spectrum of control modes. These can range from direct teleoperation to full autonomy. This paper describes a pilot-study experiment in which a variable autonomy robot completes a navigation task. It explores the comparative performances of the human-robot system at different autonomy levels under different sets of conditions. This is done from a Mixed-Initiative system investigation perspective. Sensor noise was added to degrade robot performance, while a secondary task induced varying degrees of additional workload on the human operator. Carrying out these experiments and analyzing the initial results, has highlighted the profound complexities of designing tasks, conditions, and performance metrics which are: principled, eliminate confounding factors, and yield scientifically rigorous insights into the intricacies of a collaborative system that combines both human and robot intelligences. A key contribution of this paper is to describe the lessons learned from attempting these experiments, and to suggest a variety of guidelines for other researchers to consider when designing experiments in this context.

This letter presents a feature weighting method for polarimetric synthetic aperture radar (PolSAR) image classification. Appropriate feature weighting is essential for obtaining accurate classifications but so far has remained an open research problem. We propose in this letter a supervised three-component feature weighting method based on the Fisher linear discriminant. Fisher linear discriminant method is used to calculate a coefficient for each feature. Then, these coefficients are modified according to a three-component scattering power decomposition model, combining both physical and statistical scattering characteristics to adapt them for the particular scattering mechanisms inherent in PolSAR data and assigned to the coherency matrix to enhance the discriminating ability of the features. Freeman decomposition and Wishart classifier are used to classify the PolSAR image. The effectiveness of the proposed method is demonstrated by experiments NASA/JPL AIRSAR L-band and CSA Radarsat-2 C-band PolSAR images of the San Francisco area.

Generalising dexterous grasps to novel objects is an open problem. We show how to learn grasps for high DoF hands that generalise to novel objects, given as little as one demonstrated grasp. During grasp learning two types of probability density are learned that model the demonstrated grasp. The first density type (the contact model) models the relationship of an individual finger part to local surface features at its contact point. The second density type (the hand configuration model) models the whole hand configuration during the approach to grasp. When presented with a new object, many candidate grasps are generated, and a kinematically feasible grasp is selected that maximises the product of these densities. We demonstrate 31 successful grasps on novel objects (an 86% success rate), transferred from 16 training grasps. The method enables: transfer of dexterous grasps within object categories; across object categories; to and from objects where there is no complete model of the object available; and using two different dexterous hands.

This paper addresses the problem of finding sparse solutions to linear systems. Although this problem involves two competing cost function terms (measurement error and a sparsity-inducing term), previous approaches combine these into a single cost term and solve the problem using conventional numerical optimization methods. In contrast, the main contribution of this paper is to use a multiobjective approach. The paper begins by investigating the sparse reconstruction problem, and presents data to show that knee regions do exist on the Pareto front (PF) for this problem and that optimal solutions can be found in these knee regions. Another contribution of the paper, a new soft-thresholding evolutionary multiobjective algorithm (StEMO), is then presented, which uses a soft-thresholding technique to incorporate two additional heuristics: one with greater chance to increase speed of convergence toward the PF, and another with higher probability to improve the spread of solutions along the PF, enabling an optimal solution to be found in the knee region. Experiments are presented, which show that StEMO significantly outperforms five other well known techniques that are commonly used for sparse reconstruction. Practical applications are also demonstrated to fundamental problems of recovering signals and images from noisy data.

Dexterous grasping of objects with uncertain pose is a hard unsolved problem in robotics. This paper solves this problem using information gain re-planning. First we show how tactile information, acquired during a failed attempt to grasp an object can be used to refine the estimate of that object's pose. Second, we show how this information can be used to replan new reach to grasp trajectories for successive grasp attempts. Finally we show how reach-to-grasp trajectories can be modified, so that they maximise the expected tactile information gain, while simultaneously delivering the hand to the grasp configuration that is most likely to succeed. Our main novel outcome is thus to enable tactile information gain planning for Dexterous, high degree of freedom (DoFs) manipulators. We achieve this using a combination of information gain planning, hierarchical probabilistic roadmap planning, and belief updating from tactile sensors for objects with non-Gaussian pose uncertainty in 6 dimensions. The method is demonstrated in trials with simulated robots. Sequential replanning is shown to achieve a greater success rate than single grasp attempts, and trajectories that maximise information gain require fewer re-planning iterations than conventional planning methods before a grasp is achieved.

This paper presents an algorithm for planning sequences of pushes, by which a robotic arm equipped with a single rigid finger can move a manipulated object (or manipulandum) towards a desired goal pose. Pushing is perhaps the most basic kind of manipulation, however it presents difficult challenges for planning, because of the complex relationship between manipulative pushing actions and resulting manipulandum motions. The motion planning literature has well developed paradigms for solving e.g. the piano-mover's problem, where the search occurs directly in the configuration space of the manipulandum object being moved. In contrast, in pushing manipulation, a plan must be built in the action space of the robot, which is only indirectly linked to the motion space of the manipulandum through a complex interaction for which inverse models may not be known. In this paper, we present a two stage approach to planning pushing operations. A global RRT path planner is used to explore the space of possible manipulandum configurations, while a local push planner makes use of predictive models of pushing interactions, to plan sequences of pushes to move the manipulandum from one RRT node to the next. The effectiveness of the algorithm is demonstrated in simulation experiments in which a robot must move a rigid body through complex 3D transformations by applying only a sequence of simple single finger pushes.

The problem of model-based object tracking in three dimensions is addressed. Most previous work on tracking assumes simple motion models, and consequently tracking typically fails in a variety of situations. Our insight is that incorporating physics models of object behaviour improves tracking performance in these cases. In particular it allows us to handle tracking in the face of rigid body interactions where there is also occlusion and fast object motion. We show how to incorporate rigid body physics simulation into a particle filter. We present two methods for this based on pose and force noise. The improvements are tested on four videos of a robot pushing an object, and results indicate that our approach performs considerably better than a plain particle filter tracker, with the force noise method producing the best results over the range of test videos.

An important problem in robotic manipulation is the ability to predict how objects behave under manipulative actions. This ability is necessary to allow planning of object manipulations. Physics simulators can be used to do this, but they model many kinds of object interaction poorly. An alternative is to learn a motion model for objects by interacting with them. In this paper we address the problem of learning to predict the interactions of rigid bodies in a probabilistic framework, and demonstrate the results in the domain of robotic push manipulation. A robot arm applies random pushes to various objects and observes the resulting motion with a vision system. The relationship between push actions and object motions is learned, and enables the robot to predict the motions that will result from new pushes. The learning does not make explicit use of physics knowledge, or any pre-coded physical constraints, nor is it even restricted to domains which obey any particular rules of physics. We use regression to learn efficiently how to predict the gross motion of a particular object. We further show how different density functions can encode different kinds of information about the behaviour of interacting objects. By combining these as a product of densities, we show how learned predictors can cope with a degree of generalisation to previously unencountered object shapes, subjected to previously unencountered push directions. Performance is evaluated through a combination of virtual experiments in a physics simulator, and real experiments with a 5-axis arm equipped with a simple, rigid finger.

Visual tracking of an object can provide a powerful source of feedback information during complex robotic manipulation operations, especially those in which there may be uncertainty about which new object pose may result from a planned manipulative action. At the same time, robotic manipulation can provide a challenging environment for visual tracking, with occlusions of the object by other objects or by the robot itself, and sudden changes in object pose that may be accompanied by motion blur. Recursive filtering techniques use motion models for predictor-corrector tracking, but the simple models typically used often fail to adequately predict the complex motions of manipulated objects. We show how statistical machine learning techniques can be used to train sophisticated motion predictors, which incorporate additional information by being conditioned on the planned manipulative action being executed. We then show how these learned predictors can be used to propagate the particles of a particle filter from one predictor-corrector step to the next, enabling a visual tracking algorithm to maintain plausible hypotheses about the location of an object, even during severe occlusion and other difficult conditions. We demonstrate the approach in the context of robotic push manipulation, where a 5-axis robot arm equipped with a rigid finger applies a series of pushes to an object, while it is tracked by a vision algorithm using a single camera.

We consider the task of monocular visual motion estimation from video image sequences. We hypothesise that performance on the task can be improved by incorporating an understanding of physically likely and feasible object dynamics. We test this hypothesis by incorporating a physical simulator into a least-squares estimation procedure. We initialise a full trajectory estimate using RANSAC followed by gradient descent refinement. We present results for 2D image sequences consisting of single ambiguous, visible or occluded balls, as well as results for 3D computer-generated sequences of objects in free-flight with added noise. Results suggest that restricting the estimation to allow only motions that are feasible according to the physics simulator can produce marked improvement when the observed object motion is within the limits of the physics simulator and its world model. Conversely, merely penalising deviations from feasible physical dynamics produces a consistent but incremental improvement over more common dynamics models.

Visual tracking algorithms have important robotic applications such as mobile robot guidance and servoed wide area surveillance systems. These applications ideally require vision algorithms which are robust to camera motion and scene change but are cheap and fast enough to run on small, low power embedded systems. Unfortunately most robust visual tracking algorithms are either computationally expensive or are restricted to a stationary camera. This paper describes a new color based tracking algorithm, the Adaptive Background CAMSHIFT (ABCshift) tracker and an associated technique, mean shift servoing, for efficient pan-tilt servoing of a motorized camera platform. ABCshift achieves robustness against camera motion and other scene changes by continuously relearning its background model at every frame. This also enables robustness in difficult scenes where the tracked object moves past backgrounds with which it shares significant colors. Despite this continuous machine learning, ABCshift needs minimal training and is remarkably computationally cheap. We first demonstrate how ABCshift tracks robustly in situations where related algorithms fail, and then show how it can be used for real time tracking with pan-tilt servo control using only a small embedded microcontroller.

This paper addresses the problems of automatically planning autonomous underwater vehicle (AUV) paths which best exploit complex current data, from computational estuarine model forecasts, while also avoiding obstacles. In particular we examine the possibilities for a novel type of AUV mission deployment in fast flowing tidal river regions which experience bi-directional current flow. These environments are interesting in that, by choosing an appropriate path in space and time, an AUV may both bypass adverse currents which are too fast to be overcome by the vehicle's motors and also exploit favorable currents to achieve far greater speeds than the motors could otherwise provide, while substantially saving energy. The AUV can "ride" currents both up and down the river, enabling extended monitoring of otherwise energy-exhausting, fast flow environments. The paper discusses suitable path parameterizations, cost functions and optimization techniques which enable optimal AUV paths to be efficiently generated. These paths take maximum advantage of the river currents in order to minimize energy expenditure, journey time and other cost parameters. The resulting path planner can automatically suggest useful alternative mission start and end times and locations to those specified by the user. Examples are presented for navigation in a simple simulation of the fast flowing Hudson River waters around Manhattan.