Modern robotics has enabled the advancement in yield estimation for precision agriculture. However, when applied to the olive industry, the high variation of olive colors and their similarity to the background leaf canopy presents a challenge. Labeling several thousands of very dense olive grove images for segmentation is a labor-intensive task. This paper presents a novel approach to detecting olives without the need to manually label data. In this work, we present the world's first olive detection dataset comprised of synthetic and real olive tree images. This is accomplished by generating an auto-labeled photorealistic 3D model of an olive tree. Its geometry is then simplified for lightweight rendering purposes. In addition, experiments are conducted with a mix of synthetically generated and real images, yielding an improvement of up to 66% compared to when only using a small sample of real data. When access to real, human-labeled data is limited, a combination of mostly synthetic data and a small amount of real data can enhance olive detection.

The use of Reinforcement Learning (RL) approaches for dialogue policy optimization has been the new trend for dialogue management systems. Several methods have been proposed, which are trained on dialogue data to provide optimal system response. However, most of these approaches exhibit performance degradation in the presence of noise, poor scalability to other domains, as well as performance instabilities. To overcome these problems, we propose a novel approach based on the incremental, sample-efficient Least-Squares Policy Iteration (LSPI) algorithm, which is trained on compact, fixed-size dialogue state encodings, obtained from deep Variational Denoising Autoencoders (VDAE). The proposed scheme exhibits stable and noise-robust performance, which significantly outperforms the current state-of-the-art, even in mismatched noise environments.

This article presents a complete formulation of the challenging task of stable humanoid robot omnidirectional walk based on the Cart and Table model for approximating the robot dynamics. For the control task, we propose two novel approaches: preview control augmented with the inverse system for negotiating strong disturbances and uneven terrain and linear model-predictive control approximated by an orthonormal basis for computational efficiency coupled with constraints for improved stability. For the generation of smooth feet trajectories, we present a new approach based on rigid body interpolation, enhanced by adaptive step correction. Finally, we present a sensor fusion approach for sensor-based state estimation and an effective solution to sensors' noise, delay, and bias issues, as well as to errors induced by the simplified dynamics and actuation imperfections. Our formulation is applied on a real NAO humanoid robot, where it achieves real-time onboard execution and yields smooth and stable gaits.

Several recent learning approaches based on approximate policy iteration suggest the use of classifiers for representing policies compactly. The space of possible policies, even under such structured representations, is huge and must be searched carefully to avoid computationally expensive policy simulations (rollouts). In our recent work, we proposed a method for directed exploration of policy space using support vector classifiers, whereby rollouts are directed to states around the boundaries between different action choices indicated by the separating hyper planes in the represented policies. While effective, this method suffers from the growing number of support vectors in the underlying classifiers as the number of training examples increases. In this paper, we propose an alternative method for directed policy search based on relevance vector machines. Relevance vector machines are used both for classification (to represent a policy) and regression (to approximate the corresponding relative action advantage function). Exploiting the internal structure of the regress or, we guide the probing of the state space only to critical areas corresponding to changes of action dominance in the underlying policy. This directed focus on critical parts of the state space iteratively leads to refinement and improvement of the underlying policy and delivers excellent control policies in only a few iterations, while the small number of relevance vectors yields significant computational time savings. We demonstrate the proposed approach and compare it with our previous method on standard reinforcement learning domains.

The ability of learning is critical for agents designed to compete in a variety of two-player, turn-taking, tactical adversarial games, such as Backgammon, Othello/Reversi, Chess, Hex, etc. The mainstream approach to learning in such games consists of updating some state evaluation function usually in a Temporal Difference (TD) sense either under the MiniMax optimality criterion or under optimization against a specific opponent. However, this approach is limited by several factors: (a) updates to the evaluation function are incremental, (b) stored samples from past games cannot be utilized, and (c) the quality of each update depends on the current evaluation function due to bootstrapping. In this paper, we present a learning approach based on the Least-Squares Policy Iteration (LSPI) algorithm that overcomes these limitations by focusing on learning a state-action evaluation function. The key advantage of the proposed approach is that the agent can make batch updates to the evaluation function with any collection of samples, can utilize samples from past games, and can make updates that do not depend on the current evaluation function since there is no bootstrapping. We demonstrate the efficiency of the LSPI agent over the TD agent in the classical board game of Othello/Reversi.

The majority of learning algorithms available today focus on approximating the state (V ) or state-action (Q) value function and efficient action selection comes as an afterthought. On the other hand, real-world problems tend to have large action spaces, where evaluating every possible action becomes impractical. This mismatch presents a major obstacle in successfully applying reinforcement learning to real-world problems. In this paper we present an effective approach to learning and acting in domains with multidimensional and/or continuous control variables where efficient action selection is embedded in the learning process. Instead of learning and representing the state or state-action value function of the MDP, we learn a value function over an implied augmented MDP, where states represent collections of actions in the original MDP and transitions represent choices eliminating parts of the action space at each step. Action selection in the original MDP is reduced to a binary search by the agent in the transformed MDP, with computational complexity logarithmic in the number of actions, or equivalently linear in the number of action dimensions. Our method can be combined with any discrete-action reinforcement learning algorithm for learning multidimensional continuous-action policies using a state value approximator in the transformed MDP. Our preliminary results with two well-known reinforcement learning algorithms (Least-Squares Policy Iteration and Fitted Q-Iteration) on two continuous action domains (1-dimensional inverted pendulum regulator, 2-dimensional bicycle balancing) demonstrate the viability and the potential of the proposed approach.

Good policies in reinforcement learning problems typically exhibit significant structure. Several recent learning approaches based on the approximate policy iteration scheme suggest the use of classifiers for capturing this structure and representing policies compactly. Nevertheless, the space of possible policies, even under such structured representations, is huge and needs to be explored carefully to avoid computationally expensive simulations (rollouts) needed to probe the improved policy and obtain training samples at various points over the state space. Regarding rollouts as a scarce resource, we propose a method for directed exploration of policy space using support vector classifiers. We use a collection of binary support vector classifiers to represent policies, whereby each of these classifiers corresponds to a single action and captures the parts of the state space where this action dominates over the other actions. After an initial training phase with rollouts uniformly distributed over the entire state space, we use the support vectors of the classifiers to identify the critical parts of the state space with boundaries between different action choices in the represented policy. The policy is subsequently improved by probing the state space only at points around the support vectors that are distributed perpendicularly to the separating border. This directed focus on critical parts of the state space iteratively leads to the gradual refinement and improvement of the underlying policy and delivers excellent control policies in only a few iterations with a conservative use of rollouts. We demonstrate the proposed approach on three standard reinforcement learning domains: inverted pendulum, mountain car, and acrobot.

Reinforcement learning for control in stochastic processes has received significant attention in the last few years. Several data-efficient methods, even for continuous state spaces, have been proposed, however most of them assume a small and discrete action space. While continuous action spaces are quite common in real-world problems, the most common approach still employed in practice is coarse discretization of the action space. This paper presents a novel, computationally-efficient method, called adaptive action modification, for realizing continuous-action policies, using binary decisions corresponding to adaptive increment or decrement changes in the values of the continuous action variables. The proposed approach essentially approximates any continuous action space to arbitrary resolution and can be combined with any discrete-action reinforcement learning algorithm for learning continuous-action policies. Our approach is coupled with three well-known reinforcement learning algorithms (Q-learning, fitted Q-iteration, and least-squares policy iteration) and its use and properties are thoroughly investigated and demonstrated on the continuous state-action inverted pendulum and bicycle balancing and riding domains.

Reinforcement Learning methods for controlling stochastic processes typically assume a small and discrete action space. While continuous action spaces are quite common in real-world problems, the most common approach still employed in practice is coarse discretization of the action space. This paper presents a novel method, called Binary Action Search, for realizing continuousaction policies by searching efficiently the entire action range through increment and decrement modifications to the values of the action variables according to an internal binary policy defined over an augmented state space. The proposed approach essentially approximates any continuous action space to arbitrary resolution and can be combined with any discrete-action reinforcement learning algorithm for learning continuous-action policies. Binary Action Search eliminates the restrictive modification steps of Adaptive Action Modification and requires no temporal action locality in the domain. Our approach is coupled with two well-known reinforcement learning algorithms (Least-Squares Policy Iteration and Fitted Q-Iteration) and its use and properties are thoroughly investigated and demonstrated on the continuous state-action Inverted Pendulum, Double Integrator, and Car on the Hill domains.

Low-cost robots with a large number of degrees of freedom are becoming increasingly popular, nevertheless their programming is still a domain for experts. This paper introduces the Kouretes Motion Editor (KME), a freely-available interactive software tool for designing complex motion patterns on robots with many degrees of freedom using intuitive means. KME allows for a TCP/IP connection to a real or simulated robot, over which various robot poses can be communicated to or from the robot and manipulated locally using the KME graphical user interface. This portability and flexibility enables the user to work under different modes, with different robots, using different host machines. KME is originally designed for the Aldebaran Nao humanoid robot which features a total of 21 degrees of freedom, but can be easily customized for other robots. KME has been employed successfully by Kouretes, the RoboCup team of the Technical University of Crete, for designing various special actions at the RoboCup 2008 competition (Standard Platform League).

Motivated by recent proposals that view a reinforcement learning problem as a collection of classification problems, we investigate various aspects of policy representation using classifiers. In particular, we derive optimal policies for two standard reinforcement learning domains (inverted pendulum and mountain car) in both deterministic and stochastic versions and we examine their internal structure. We then proceed in an evaluation of the representational ability of a variety of classifiers for these policies, using both a multi-class and a binary formulation of the classification problem. Finally, we evaluate the actual performance of the policies learned by the classifiers in the original control problem as a function of the amount of training examples provided. Our results offer significant insight in making the reinforcement-learning-via-classification technology successfully applicable to hard learning problems.

Teams of mobile robots have been recently proposed as effective means of completing complex missions involving multiple tasks spatially distributed over a large area. A central problem in such domains is multi-robot routing, namely the problem of coordinating a team of robots in terms of the locations they should visit and the routes they should follow in order to accomplish their common mission. A typical assumption made in prior work on multi-robot routing is that robots are able to communicate uninterruptedly at all times independently of their locations. In this paper, we investigate the multi-robot routing problem under communication constraints, reflecting on the fact that real mobile robots have a limited range of communication and the requirement that connectivity must remain intact (even through relaying) during the entire mission. We propose four algorithms for this problem, all based on the same reactive framework, ranging from greedy to deliberative approaches. All algorithms are tested in various scenarios implemented using the Player-Stage robot simulation environment. Our results demonstrate that effective multi-robot routing can be achieved even under limited communication range with moderate loss compared to the case of infinite communication range.

Several researchers have recently investigated the connection between reinforcement learning and classification. We are motivated by proposals of approximate policy iteration schemes without value functions, which focus on policy representation using classifiers and address policy learning as a supervised learning problem. This paper proposes variants of an improved policy iteration scheme which addresses the core sampling problem in evaluating a policy through simulation as a multi-armed bandit machine. The resulting algorithm offers comparable performance to the previous algorithm achieved, however, with significantly less computational effort. An order of magnitude improvement is demonstrated experimentally in two standard reinforcement learning domains: inverted pendulum and mountain-car.

For a decade now, the RoboCup competition promotes research in robotics through soccer games between autonomous robot teams. The ability to coordinate the players within such a team of robots is the key to the success of the team. Team coordination in a human soccer game is achieved through various team formations, tactics, and strategies. Unfortunately, research in the four-legged RoboCup league has focused mostly on single player skills, demonstrating only limited results in coordinated team play. In our work, we adapt and transfer formations, tactics, and strategies used by human soccer teams, such as the popular 4-4-2 scheme, to our four-legged RoboCup team Kouretes. We define roles for each player in all the cases we consider and we implement these roles using Petri Net Plans (PNP). The assignment of appropriate roles to players is performed dynamically during the game depending on the current game state using a simple communication scheme and a finite state machine. Our approach is implemented and tested on our four-legged RoboCup team. The proposed coordination scheme can be generalized and used in various robot team applications beyond robotic soccer, such as planetary exploration and search-and-rescue missions.

This paper describes a motion controller which issues incremental velocity commands to a nonholonomic mobile robot guided by a holonomic path planner. The controller finds the best pair of rotational and translational velocities through discrete optimization of a multi-criteria objective function. Valid velocity choices are restricted within the dynamic velocity window as determined by the current velocities and the maximum acceleration and deceleration ability of the robot (smoothness criterion). In addition, for each choice the objective function examines the proximity of the resulting stopping configuration to the current goal set by the planner (accuracy criterion). Finally, it takes into account the potential of unforeseen, perhaps unavoidable, collisions due to dynamic obstacles or due to the holonomic nature of the path planner (safety criterion). The controller issues commands which balance these criteria as determined by the user. The proposed controller has been implemented and tested on a Nomad 200 mobile robot achieving smooth, accurate, and safe motion control both in simulated and in real environments

We consider the problem of allocating a number of exploration tasks to a team of mobile robots. Each task consists of a target location that needs to be visited by a robot. The objective of the allocation is to minimize the total cost, that is, the sum of the travel costs of all robots for visiting all targets. We show that finding an optimal allocation is an NP-hard problem, even in known environments. The main contribution of this paper is PRIM ALLOCATION, a simple and fast approximate algorithm for allocating targets to robots which provably computes allocations whose total cost is at most twice as large as the optimal total cost. We then cast PRIM ALLOCATION in terms of a multi-round single-item auction where robots bid on targets, which allow for a decentralized implementation. To the best of our knowledge, PRIM ALLOCATION is the first auction-based allocation algorithm that provides a guarantee on the quality of its allocations. Our experimental results in a multi-robot simulator demonstrate that PRIM ALLOCATION is fast and results in close-to-optimal allocations despite its simplicity and decentralized nature. In particular, it needs an order of magnitude fewer bids than a computationally intensive allocation algorithm based on combinatorial auctions, yet its allocations are at least as good.

We present a new method for learning good strategies in zero-sum Markov games in which each side is composed of multiple agents col- laborating against an opposing team of agents. Our method requires full observability and communication during learning, but the learned poli- cies can be executed in a distributed manner. The value function is rep- resented as a factored linear architecture and its structure determines the necessary computational resources and communication bandwidth. This approach permits a tradeoff between simple representations with little or no communication between agents and complex, computationally inten- sive representations with extensive coordination between agents. Thus, we provide a principled means of using approximation to combat the exponential blowup in the joint action space of the participants. The ap- proach is demonstrated with an example that shows the efficiency gains over naive enumeration.

We propose a new approach to reinforcement learning which combines least squares function approximation with policy iteration. Our method is model-free and completely off policy. We are motivated by the least squares temporal difference learning algorithm (LSTD), which is known for its efficient use of sample experiences compared to pure temporal difference algorithms. LSTD is ideal for prediction problems, however it heretofore has not had a straightforward application to control problems. Moreover, approximations learned by LSTD are strongly influenced by the visitation distribution over states. Our new algorithm, Least Squares Policy Iteration (LSPI) addresses these issues. The result is an off-policy method which can use (or reuse) data collected from any source. We have tested LSPI on several problems, including a bicycle simulator in which it learns to guide the bicycle to a goal efficiently by merely observing a relatively small number of completely random trials.

Neural maps have been recently proposed as an alternative method for mobile robot path planning. However, these proposals are mostly theoretical and primarily concerned with biological plausibility. This paper addresses the applicability of neural maps to mobile robot navigation with focus on efficient implementations It is suggested that neural maps offer a promising alternative compared to the traditional distance transform and harmonic function methods. Applications of neural maps are presented for both global and local navigation. Experimental results (both simulated and real-world on a Nomad 200 mobile robot) demonstrate the validity of the approach. Our work reveals that a key issue for success of the method is the organization of the map that needs to be optimized for the situation at hand.