We present ROBOVERINE, a neural dynamic robotic active vision process model of selective visual attention and scene grammar in naturalistic environments. The model addresses significant challenges for cognitive robotic models of visual attention: combined bottom-up salience and top-down feature guidance, combined overt and covert attention, coordinate transformations, two forms of inhibition of return, finding objects outside of the camera frame, integrated space-and object-based analysis, minimally supervised few-shot continuous online learning for recognition and guidance templates, and autonomous switching between exploration and visual search. Furthermore, it incorporates a neural process account of scene grammar — prior knowledge about the relation between objects in the scene — to reduce the search space and increase search efficiency. The model also showcases the strength of bridging two frameworks: Deep Neural Networks for feature extractions and Dynamic Field Theory for cognitive operations.

In Piaget's classical A-not-B-task, infants repeatedly make a sensorimotor decision to reach to one of two cued targets. Perseverative errors are induced by switching the cue from A to B, while spontaneous errors are unsolicited reaches to B when only A is cued. We argue that theoretical accounts of sensorimotor decision-making fail to address how motor decisions leave a memory trace that may impact future sensorimotor decisions. Instead, in extant neural models, perseveration is caused solely by the history of stimulation. We present a neural dynamic model of sensorimotor decision-making within the framework of Dynamic Field Theory, in which a dynamic instability amplifies fluctuations in neural activation into macroscopic, stable neural activation states that leave memory traces. The model predicts perseveration, but also a tendency to repeat spontaneous errors. To test the account, we pool data from several A-not-B experiments. A conditional probabilities analysis accounts quantitatively how motor decisions depend on the history of reaching. The results provide evidence for the interdependence among subsequent reaching decisions that is explained by the model, showing that by amplifying small differences in activation and affecting learning, decisions have consequences beyond the individual behavioural act.

A central task in the analysis of human movement behavior is to determine systematic patterns and differences across experimental conditions, participants and repetitions. This is possible because human movement is highly regular, being constrained by invariance principles. Movement timing and movement path, in particular, are linked through scaling laws. Separating variations of movement timing from the spatial variations of movements is a well-known challenge that is addressed in current approaches only through forms of preprocessing that bias analysis. Here we propose a novel nonlinear mixed-effects model for analyzing temporally continuous signals that contain systematic effects in both timing and path. Identifiability issues of path relative to timing are overcome by using maximum likelihood estimation in which the most likely separation of space and time is chosen given the variation found in data. The model is applied to analyze experimental data of human arm movements in which participants move a hand-held object to a target location while avoiding an obstacle. The model is used to classify movement data according to participant. Comparison to alternative approaches establishes nonlinear mixed-effects models as viable alternatives to conventional analysis frameworks. The model is then combined with a novel factor-analysis model that estimates the low-dimensional subspace within which movements vary when the task demands vary. Our framework enables us to visualize different dimensions of movement variation and to test hypotheses about the effect of obstacle placement and height on the movement path. We demonstrate that the approach can be used to uncover new properties of human movement.

We present a neural dynamics architecture for grasping that integrates perceptual processes of scene exploration, object selection and classification, and grasp pose estimation with motor processes such as planning and controlling reach and grasp movements. Inspired by theories of human embodied cognition, the entire architecture is essentially one big dynamical system from which discrete events such as initiating and terminating reaches and grasps emerge through dynamical instabilities. Using a Kinect sensor as input, we implement the architecture on a Kuka light weight arm with a Schunk Dextrous Hand and demonstrate grasping movements that are updated on-line when the object is shifted or rotated during movement planning or execution.

Autonomous lane keeping is a well studied problem with several good solutions that can be found in the literature. However, naturalistic lane keeping mechanisms, in the sense of imitating human car steering, are not so common. Based on existing knowledge of human driving, this paper analyses several controllers prone to generate human-like lane keeping behavior. Using systems identification, we compare how well the control models fit with real human steering data gathered in a simulator. Experimental results points towards a parsimonious control mechanism where angles relative to the direction of the road are directly used by the driver to steer the car and keep it on the lane. This result can be used to design human like autonomous lane keeping mechanisms or to improve the design of ADAS systems adapting them to individual drivers.

We propose a model that autonomously generates and flexibly organizes sequences of timed actions. The timing of the movements is controlled by non-linear oscillators. Their activation and deactivation is organized by a hierarchical neural-dynamic architecture. We demonstrate the features of our model in an exemplary robotic task where the manipulator arm keeps hitting a ball up an inclined plane. The autonomous generation of movement sequences is tightly coupled to visual sensory information about the ball motion and able to adapt, on-line, to perturbations introduced in the ball trajectory. The performance of the proposed model is evaluated and the reactions to different perturbations are discussed.

Robotic agents that interact with humans and perform complex, everyday tasks in natural environments will require a system to autonomously organize their behavior. Current systems for robotic behavioral organization typically abstract from the low-level sensory-motor embodiment of the robot, leading to a gap between the level at which a sequence of actions is planned and the levels of perception and motor control. This gap is a major bottleneck for the autonomy of systems in complex, dynamic environments. To address this issue, we present a neural-dynamic framework for behavioral organization, in which the action selection mechanism is tightly coupled to the agent's sensory-motor systems. The elementary behaviors (EBs) of the robot are dynamically organized into sequences based on task-specific behavioral constraints and online perceptual information. We demonstrate the viability of our approach by implementing a neural-dynamic architecture on the humanoid robot NAO. The system is capable of producing sequences of EBs that are directed at objects (e.g., grasping and pointing). The sequences are flexible in that the robot autonomously adapts the individual EBs and their sequential order in response to changes in the sensed environment. The architecture can accommodate different tasks and can be articulated for different robotic platforms. Its neural-dynamic substrate is particularly well-suited for learning and adaptation.

Spatial language is a privileged channel of human-robot interaction. Here, we extend a neural-dynamic architecture for grounded spatial language in three ways. First, we introduce autonomous selection between viewer-centered and intrinsic reference frames, using an estimation of the reference object orientation to determine its intrinsic axes. Second, we employ an orientation estimation dynamics to represent the configurations of reference objects for spatial terms such as “between”. Third, we enhance the autonomy of the system so that the required sequence of attentional shifts, coordinate transforms, and selection decisions emerges from the time-continuous neural dynamics. In a robotic implementation we demonstrate how spatial language may be grounded in simple feature information obtained from video cameras and applied flexibly to dynamical scenes.

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject's learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.

The movement of autonomous agents in natural environments is restricted by potentially large numbers of constraints. To generate behavior that fulfills all given constraints simultaneously, the attractor dynamics approach to movement generation represents each constraint by a dynamical system with attractors or repellors at desired or undesired values of a relevant variable. These dynamical systems are transformed into vector fields over the control variables of a robotic agent that force the state of the whole system in directions beneficial to the satisfaction of the behavioral constraint. The attractor dynamics approach was recently successfully applied to the generation of manipulator motion trajectories avoiding collision with obstacles [1] and constraints on gripper orientation during reaching and grasping movements [2]. Continuing that body of work, this paper proposes a system which generates movements satisfying both obstacle avoidance and gripper orientation constraints simultaneously. As an extension, the additional constraint of avoiding hardware limits for joint angles is included. Properties of the resulting system are demonstrated by a systematic study generating movements with a large number of constraints in different scene setups. Specific characteristics are highlighted by several showcase example movements.

We present a robotic vision system for object recognition, pose estimation and fast object learning. Our approach uses the Dynamic Neural Field Theory to combine bottom-up recognition of matching patterns and top-down estimation of pose parameters in a recurrent loop. Because Dynamic Neural Fields provide the system with stabilized percepts that still track changes in the incoming sensory stream, the system is able to do pose tracking even if objects are shortly occluded or distractor objects are moved into the scene.

For autonomous robots to manipulate objects in unknown environments, they must be able to move their arms without colliding with nearby objects, other agents or humans. The simultaneous avoidance of multiple obstacles in real time by all link segments of a manipulator is still a hard task both in practice and in theory. We present a systematic scheme for the generation of collision free movements for redundant manipulators in scenes with arbitrarily many obstacles. Based on the dynamical systems approach to robotics, constraints are formulated as contributions to a dynamical system that erect attractors for targets and repellors for obstacles. These contributions are formulated in terms of variables relevant to each constraint and then transformed into vector fields over the manipulator joint velocity vector as an embedding space in which all constraints are simultaneously observed. We demonstrate the feasibility of the approach by implementing it on a real anthropomorphic 8-degrees-of-freedom redundant manipulator. In addition, performance is characterized by detecting failures in a systematic simulation experiment in randomized scenes with varying numbers of obstacles.

For an autonomous robotic system, the ability to share the same workspace and interact with humans is the basis for cooperative behavior. In this work, we investigate human spatial language as the communicative channel between the robot and the human, facilitating their joint work on a tabletop. We specifically combine the theory of Dynamic Neural Fields that represent perceptual and cognitive states with motor control and linguistic input in a robotic demonstration. We show that such a neural dynamic framework can integrate across symbolic, perceptual, and motor processes to generate task-specific spatial communication in real time.

The ability to generate discrete movement with distinct and stable time courses is important for interaction scenarios both between different robots and with human partners, for catching and interception tasks, and for timed action sequences. In dynamic environments, where trajectories are evolving online, this is not a trivial task. The dynamical systems approach to robotics provides a framework for robust incorporation of fluctuating sensor information, but control of movement time is usually restricted to rhythmic motion and realized through stable limit cycles. The present work uses a Hopf oscillator to produce discrete motion and formulates an online adaptation rule to stabilize total movement time against a wide range of disturbances. This is integrated into a dynamical systems framework for the sequencing of movement phases and for directional navigation, using 2D-planar motion as an example. The approach is demonstrated on a Khepera mobile unit in order to show its reliability even when depending on low-level sensor information.

Learning to recognize objects from a small number of example views is a difficult problem of robot vision, of particular importance to assistance robots who are taught by human users. Here we present an approach that combines bottom-up recognition of matching patterns and top-down estimation of pose parameters in a recurrent loop that improves on previous efforts to reconcile invariance of recognition under view changes with discrimination among different objects. We demonstrate and evaluate the approach both in a service robotics implementation as well as on the COIL database. The robotic implementation highlights features of our approach that enable real-time pose tracking as well as recognition from views where figure ground segmentation is difficult.

Outside the laboratory, human movement typically involves redundant effector systems. How the nervous system selects among the task-equivalent solutions may provide insights into how movement is controlled. We propose a process model of movement generation that accounts for the kinematics of goal-directed pointing movements performed with a redundant arm. The key element is a neuronal dynamics that generates a virtual joint trajectory. This dynamics receives input from a neuronal timer that paces end-effector motion along its path. Within this dynamics, virtual joint velocity vectors that move the end effector are dynamically decoupled from velocity vectors that do not. Moreover, the sensed real joint configuration is coupled back into this neuronal dynamics, updating the virtual trajectory so that it yields to task-equivalent deviations from the dynamic movement plan. Experimental data from participants who perform in the same task setting as the model are compared in detail to the model predictions. We discover that joint velocities contain a substantial amount of self-motion that does not move the end effector. This is caused by the low impedance of muscle joint systems and by coupling among muscle joint systems due to multiarticulatory muscles. Back-coupling amplifies the induced control errors. We establish a link between the amount of self-motion and how curved the end-effector path is. We show that models in which an inverse dynamics cancels interaction torques predict too little self-motion and too straight end-effector paths.

We extend the attractor dynamics approach to generate goal-directed movement of a redundant, anthropomorphic arm while avoiding dynamic obstacles and respecting joint limits. To make the robot's movements human-like, we generate approximately straight-line trajectories by using two heading direction angles of the tool-point quite analogously to how movement is represented in the primate central nervous system. Two additional angles control the tool's spatial orientation so that it follows the tool-point's collision-free path. A fifth equation governs the redundancy angle, which controls the elevation of the elbow so as to avoid obstacles and respect joint limits. These variables make it possible to generate movement while sitting in an attractor (or, in the language of the potential field approach, in a minimum). We demonstrate the approach on an assistant robot, which interacts with human users in a shared workspace

To enable a robotic assistant to autonomously reach for and transport objects while avoiding obstacles we have generalized the attractor dynamics approach established for vehicles to trajectory formation in robot arms. This approach is able to deal with the time-varying environments that occur when a human operator moves in a shared workspace. Stable fixed points (attractors) for the heading direction of the end-effector shift during movement and are being tracked by the system. This enables the attractor dynamics approach to avoid the spurious states that hamper potential field methods. Separating planning and control computationally, the approach is also simpler to implement. The stability properties of the movement plan make it possible to deal with fluctuating and imprecise sensory information. We implement this approach on a seven degree of freedom anthropomorphic arm reaching for objects on a working surface. We use an exact solution of the inverse kinematics, which enables us to steer the spatial position of the elbow clear of obstacles. The straight-line trajectories of the end-effector that emerge as long as the arm is far from obstacles make the movement goals of the robotic assistant predictable for the human operator, improving man-machine interaction.

A new method is proposed for combining outputs of several classifiers. The method is based on the theory of dynamical systems. In our formulation each classifier output represents a forcelet in a phase space of the dynamical system. Depending on interactions (superposition) of forcelets the system can perform in two different regimes: non linear averaging among several classifiers outputs and selection among them. We show that the attractor dynamics method outperforms both the winner takes all algorithm and the single best classifier.

CORA is a robotic assistant whose task is to collaborate with a human operator on simple manipulation or handling tasks. Its sensory channels comprising vision, audition, haptics, and force sensing are used to extract perceptual information about speech, gestures and gaze of the operator, and object recognition. The anthropomorphic robot arm makes goal-directed movements to pick up and hand over objects. The human operator may mechanically interact with the arm by pushing it away (haptics) or by taking an object out of the robot's gripper (force sensing). The design objective has been to exploit the human operator's intuition by modeling the mechanical structure, the senses, and the behaviors of the assistant on human anatomy, human perception, and human motor behavior.

The WAD project is aimed to provide limited autonomy to electrical wheelchairs. The primary focus is to provide a secure obstacle avoidance behavior based on infrared distance sensors which generate contributions to the heading direction dynamics that steers the wheelchair away from obstructions. Moreover, the attractor dynamics approach is used to integrate the obstacle avoidance behavior to a user defined target acquisition behavior, in which the direction and the distance to the target are indicated by the user at different points in time.

The question of how behavioral patterns are flexibly adjusted to perceived environmental influences during learning or by intention is addressed from a theoretical perspective in which behavioral patterns are governed by collective variables and their dynamics. Due to the central concept of temporal stability, dynamic theories of behavior patterns give rise to a number of predictions and can therefore be subjected to experimental test. The concept of behavioral information provides a succinct encoding of behavioral requirements in terms of collective variables. A general prediction is that behavioral requirements act on the dynamic properties of behavioral patterns and may lead to loss of stability and discontinuous pattern change as the requirement is changed gradually. The influence of intrinsic dynamics can be observed as the systematic deviation from required patterns in the direction of intrinsically stable patterns. The author discusses how the concept of behavioral information can be used to express the cooperation or competition of different behavioral requirements