The advent of tactile sensors in robotics has sparked many ideas on how robots can leverage direct contact measurements of their environment interactions to improve manipulation tasks. An important line of research in this regard is grasp force control, which aims to manipulate objects safely by limiting the amount of force exerted on the object. While prior works have either hand-modeled their force controllers, employed model-based approaches, or not shown sim-to-real transfer, we propose a model-free deep reinforcement learning approach trained in simulation and then transferred to the robot without further fine-tuning. We, therefore, present a simulation environment that produces realistic normal forces, which we use to train continuous force control policies. A detailed evaluation shows that the learned policy performs similarly or better than a hand-crafted baseline. Ablation studies prove that the proposed inductive bias and domain randomization facilitate sim-to-real transfer. Code, models, and supplementary videos are available on https://sites.google.com/view/rl-force-ctrl

Our goal with this survey is to provide an overview of the state of the art deep learning methods for face generation and editing using StyleGAN. The survey covers the evolution of StyleGAN, from PGGAN to StyleGAN3, and explores relevant topics such as suitable metrics for training, different latent representations, GAN inversion to latent spaces of StyleGAN, face image editing, cross-domain face stylization, face restoration, and even Deepfake applications. We aim to provide an entry point into the field for readers that have basic knowledge about the field of deep learning and are looking for an accessible introduction and overview.

This work deals with a practical everyday problem: stable object placement on flat surfaces starting from unknown initial poses. Common object-placing approaches require either complete scene specifications or extrinsic sensor measurements, e.g., cameras, that occasionally suffer from occlusions. We propose a novel approach for stable object placing that combines tactile feedback and proprioceptive sensing. We devise a neural architecture called PlaceNet that estimates a rotation matrix, resulting in a corrective gripper movement that aligns the object with the placing surface for the subsequent object manipulation. We compare models with different sensing modalities, such as force-torque, an external motion capture system, and two classical baseline models in real-world object placing tasks with different objects. The experimental evaluation of our placing policies with a set of unseen everyday objects reveals significant generalization of our proposed pipeline, suggesting that tactile sensing plays a vital role in the intrinsic understanding of robotic dexterous object manipulation. Code, models, and supplementary videos are available on https://sites.google.com/view/placing-by-touching.

We present a holistic grasping controller, combining free-space position control and in-contact force-control for reliable grasping given uncertain object pose estimates. Employing tactile fingertip sensors, undesired object displacement during grasping is minimized by pausing the finger closing motion for individual joints on first contact until force-closure is established. While holding an object, the controller is compliant with external forces to avoid high internal object forces and prevent object damage. Gravity as an external force is explicitly considered and compensated for, thus preventing gravity-induced object drift. We evaluate the controller in two experiments on the TIAGo robot and its parallel-jaw gripper proving the effectiveness of the approach for robust grasping and minimizing object displacement. In a series of ablation studies, we demonstrate the utility of the individual controller components.

There is a growing trend to teach playing an instrument such as a piano at home using an automated system. A key component of such systems is the ability to rate performance of the learner in order to provide feedback and select appropriate exercises. In this study, we expand on previous works that have developed automatic evaluation systems for an overall grade by also providing predictions for specific aspects of performance: pitch, rhythm, tempo, and articulation & dynamics, as well as scheduling what is an appropriate next task. We describe how a set of salient features is extracted by comparing MIDI performance data of three piano players to an ideal performance, how the features used for evaluation are selected, and evaluate using linear regression how well the selected features are able to predict the mean scores given by a group of domain experts (piano teachers). Relatively good R2 scores (0.54 to 0.68) are achieved using a small number of features (2 - 4). Such automatic evaluation of different aspects of performance can be used as a part of an automatic learning system, and to help provide learners with detailed feedback on their performance.

Locomotion in animals provides a model for adaptive behavior as it is able to deal with various kinds of perturbations. Work in insects suggests that this evolved flexibility results from a modular architecture, which can be characterized by a recurrent neural network allowing for various emerging attractor states. Whereas a lower control-level coordinates joint movements on a short timescale, a higher-level handles action selection on longer timescales. Implementation of such a control system on a walking hexapod robot was able to deal with various walking patterns including disturbances such as uneven terrain or loss of a leg. Here, we propose a cognitive expansion to the adaptive control system that allows dealing with novel challenging situations. This approach makes use of an internal simulation-based planner that is triggered when the model-free controller fails to recover from an unstable pose. Using a grounded internal body model, the planner then tries, in internal simulation, different solutions out of context, and thus, proposes a new plan to be executed on the real robot. We demonstrate the feasibility of this control approach for walking over terrain with uncertain footholds in three scenarios.

We present an approach based on conditional generative adversarial networks (GANs) to generate grasps directly and in a feed-forward manner from a raw depth image input. Building on the recently introduced StyleGAN architecture we extend results from an earlier proof-of-concept paper [1] and demonstrate successful sim2real transfer of grasp outputs for a robot arm with a Shadow Dexterous Hand. We find that the GAN model, which was only trained on a limited set of primitive objects, was able to generalize to a range of everyday real-world objects that differed significantly from the primitive objects used in simulation training. In contrast to discriminative models, the approach learns a latent representation in the set of feasible grasps that can be used for navigation in grasp space and thus allows smooth integration with other motion planning tools.

We propose an autonomous grasping pipeline that relies on geometric information extracted from segmented point cloud data. This is in contrast to many recent approaches leveraging deep learning and thus relying on a rather large amount of training samples. We argue that the proposed geometric approach facilitates task-level planning as the shape, size, and symmetry of objects can be directly taken into account during the planning process that utilizes the new MoveIt! Task Constructor (MTC) framework to define and plan action sequences composed of several inter-related sub-tasks. The efficiency of the proposed grasping pipeline is illustrated in pick-and-place scenarios, including a long-distance pick-and-place requiring a hand-over between two hands.

Decentralization is a central characteristic of biological motor control that allows for fast responses relying on local sensory information. In contrast, the current trend of Deep Reinforcement Learning (DRL) based approaches to motor control follows a centralized paradigm using a single, holistic controller that has to untangle the whole input information space. This motivates to ask whether decentralization as seen in biological control architectures might also be beneficial for embodied sensori-motor control systems when using DRL. To answer this question, we provide an analysis and comparison of eight control architectures for adaptive locomotion that were derived for a four-legged agent, but with their degree of decentralization varying systematically between the extremes of fully centralized and fully decentralized. Our comparison shows that learning speed is significantly enhanced in distributed architectures-while still reaching the same high performance level of centralized architectures-due to smaller search spaces and local costs providing more focused information for learning. Second, we find an increased robustness of the learning process in the decentralized cases-it is less demanding to hyperparameter selection and less prone to becoming trapped in poor local minima. Finally, when examining generalization to uneven terrains-not used during training-we find best performance for an intermediate architecture that is decentralized, but integrates only local information from both neighboring legs. Together, these findings demonstrate beneficial effects of distributing control into decentralized units and relying on local information. This appears as a promising approach towards more robust DRL and better generalization towards adaptive behavior.

We present our second generation tactile sensor for the Shadow Dexterous Hand's palm. We were able to significantly improve the tactile sensor characteristics by utilizing our latest barometer-based tactile sensing technology with linear (R2≥ 0.9996) sensor output and no noticeable hysteresis. The sensitivity threshold of the tactile cells and the spatial density were both dramatically increased. We demonstrate the benefits of the new sensor by re-running an experiment to estimate the stiffness of different objects that we originally used to test our first generation palm sensor. The results underline a considerable performance boost in estimation accuracy, just due to the improved tactile skin. We also propose a revised neural network architecture that even further improves the average classification accuracy to 96% in a 5-fold cross-validation.

Tactile sensing is a key sensor modality for robots interacting with their surroundings. These sensors provide a rich and diverse set of data signals that contain detailed information collected from contacts between the robot and its environment. The data are however not limited to individual contacts and can be used to extract a wide range of information about the objects in the environment as well as the actions of the robot during the interactions. In this article, we provide an overview of tactile information and its applications in robotics. We present a hierarchy consisting of raw, contact, object, and action levels to structure the tactile information, with higher-level information often building upon lower-level information. We discuss different types of information that can be extracted at each level of the hierarchy. The article also includes an overview of different types of robot applications and the types of tactile information that they employ. Finally we end the article with a discussion for future tactile applications which are still beyond the current capabilities of robots.

A lot of motion planning research in robotics focuses on efficient means to find trajectories between individual start and goal regions, but it remains challenging to specify and plan robotic manipulation actions which consist of multiple interdependent subtasks. The Task Constructor framework we present in this work provides a flexible and transparent way to define and plan such actions, enhancing the capabilities of the popular robotic manipulation framework MoveIt!.11The Task Constructor framework is publicly available at https://github.com/ros-planning/moveit_task_constructor Subproblems are solved in isolation in black-box planning stages and a common interface is used to pass solution hypotheses between stages. The framework enables the hierarchical organization of basic stages using containers, allowing for sequential as well as parallel compositions. The flexibility of the framework is illustrated in multiple scenarios performed on various robot platforms, including bimanual ones.

Recent developments in machine-learning algorithms have led to impressive performance increases in many traditional application scenarios of artificial intelligence research. In the area of deep reinforcement learning, deep learning functional architectures are combined with incremental learning schemes for sequential tasks that include interaction-based, but often delayed feedback. Despite their impressive successes, modern machine-learning approaches, including deep reinforcement learning, still perform weakly when compared to flexibly adaptive biological systems in certain naturally occurring scenarios. Such scenarios include transfers to environments different than the ones in which the training took place or environments that dynamically change, both of which are often mastered by biological systems through a capability that we here term “fluid adaptivity” to contrast it from the much slower adaptivity (“crystallized adaptivity”) of the prior learning from which the behavior emerged. In this article, we derive and discuss research strategies, based on analyzes of fluid adaptivity in biological systems and its neuronal modeling, that might aid in equipping future artificially intelligent systems with capabilities of fluid adaptivity more similar to those seen in some biologically intelligent systems. A key component of this research strategy is the dynamization of the problem space itself and the implementation of this dynamization by suitably designed flexibly interacting modules.

Our fingernails help us to accomplish a variety of manual tasks, but surprisingly only a few robotic hands are equipped with nails. In this paper, we present a sensorized fingernail for mechatronic hands that can capture static and dynamic interaction forces with the nail. Over the course of several iterations, we have developed a very compact working prototype that fits together with our previously developed multi-cell tactile fingertip sensor into the cavity of the distal phalange of a human-sized robotic hand. We present the construction details, list the key performance characteristics and demonstrate an example application of finding the end of an adhesive tape roll using the signals captured by the sensors integrated in the nail. We conclude with a discussion about improvement ideas for future versions.

The usage of articulated tools for autonomous robots is still a challenging task. One of the difficulties is to automatically estimate the tool's kinematics model. This model cannot be obtained from a single passive observation, because some information, such as a rotation axis (hinge), can only be detected when the tool is being used. Inspired by a baby using its hands while playing with an articulated toy, we employ a dual arm robotic setup and propose an interactive manipulation strategy based on visual-tactile servoing to estimate the tool's kinematics model. In our proposed method, one hand is holding the tool's handle stably, and the other arm equipped with tactile finger flips the movable part of the articulated tool. An innovative visuo-tactile servoing controller is introduced to implement the flipping task by integrating the vision and tactile feedback in a compact control loop. In order to deal with the temporary invisibility of the movable part in camera, a data fusion method which integrates the visual measurement of the movable part and the fingertip's motion trajectory is used to optimally estimate the orientation of the tool's movable part. The important tool's kinematic parameters are estimated by geometric calculations while the movable part is flipped by the finger. We evaluate our method by flipping a pivoting cleaning head (flap) of a wiper and estimating the wiper's kinematic parameters. We demonstrate that the flap of the wiper is flipped robustly, even the flap is shortly invisible. The orientation of the flap is tracked well compared to the ground truth data. The kinematic parameters of the wiper are estimated correctly.

Abstract We present a method for automatically generating reduced marker layouts for marker‐based optical motion capture of human hands. The employed motion reconstruction method is based on subspace‐constrained inverse kinematics, which allows for the recovery of realistic hand movements even from sparse input data. We additionally present a user‐specific hand model calibration procedure that fits an articulated hand model to point cloud data of the user's hand. Our marker layout optimization is sensitive to the kinematic structure and the subspace representations of hand articulations utilized in the reconstruction method, in order to generate sparse marker configurations that are optimal for solving the constrained inverse kinematics problem. We propose specific quality criteria for reduced marker sets that combine numerical stability with geometric feasibility of the resulting layout. These criteria are combined in an objective function that is minimized using a specialized surface‐constrained particle swarm optimization scheme, which generates marker layouts bound to the surface of an animated hand model. Our method provides a principled way for determining reduced marker layouts based on subspace representations of hand articulations. We demonstrate the effectiveness of our motion reconstruction and model calibration methods in a thorough evaluation.

We present a novel force-limitation algorithm to protect fragile robot actuation and transmission systems (like tendon-driven systems) from early wear-out. Inspired by human muscle fatigue, we model artificial actuator fatigue by integrating applied forces over time, gradually limiting applicable forces when fatigue increases. The algorithm is applied to protect from long-term tendon wear-out on our Shadow Dexterous Hands as well as to restrict grasping forces for compliant grasping in unknown environments. In grasping experiments the efficiency of various grasp-force limitation approaches are compared to each other, including one exploiting tactile-based slip detection.

The advent of advanced tactile sensing technology triggered the development of methods to employ them for grasp evaluation, online slip detection, and tactile servoing. In contrast to recent approaches to slip detection, distinguishing slip from non-slip conditions, we consider the more difficult task of distinguishing different types of slippage. Particularly we consider an object pushing task, where forces can only be applied from the top. In that case, the robot needs to notice when the object successfully moves vs. when the object gets stuck while the finger slips over its surface. As an example, consider the task of pushing around a piece of paper. We propose and evaluate three different convolutional network architectures and proof the applicability of the method for online classification in a robot pushing task.

The dynamics of deformable objects, especially that of highly flexible articles of clothing, is difficult to model. This is due to their vast number of degrees of freedom in addition to the noisy and incomplete measurements robots have to cope with. Therefore, we suggest focusing on the structures and object parts which are relevant to the task at hand. The openings (e.g., at the waist, leg or sleeve ends) characterize garments surprisingly well, not only from a topological perspective, but also in terms of their inherent function, namely dressing. We model openings as closed, oriented chains of movable points which we refer to as Active Boundary Component Models (ABCMs). Compared with the hardly predictable motions of an overall piece of clothing, relatively strict assumptions regarding the dynamics of these contour models can be made. We express these assumptions through position-based constraints which drastically restrict the degrees of freedom. In the present paper, we show how ABCMs can be initialized exploiting geometric prior knowledge of garments, and how they can be tracked visually using 3D point cloud data. Additionally, we consider the task of sliding a rod through a pant leg as a first step toward robotic dressing assistance for physically handicapped persons.

Identifying the relevant functional degrees of freedom is a key prerequisite for the proper handling of everyday objects. Recognizing and exploiting these degrees of freedom in the context of non-rigid objects poses challenges that are significantly different from the rigid case. As a major generic subtask, we consider the identification and exploitation of boundary components during clothes manipulation, combining RGBD vision with uni- and bi-manual handling through a robot. Specifically, we present a novel graph-based approach to detecting boundary components by extracting closed contours from depth images. Based on that, we suggest a planner minimizing a heuristic energy function for an optimal grasp pose of a robot hand around the boundary of a garment. We demonstrate the effectiveness of the approach in interactive perception and regrasping experiments with a dual arm and two attached anthropomorphic hands. Furthermore, we show how to make use of these capabilities to implement a basic skill for a coat-check robot: hanging up a knit cap on a hat-stand.

We present a novel, soft, tactile skin composed of a fabric-based, stretchable sensor technology based on the piezoresistive effect. Softness is achieved by a combination of a soft silicone padding covered by a skin of more durable, tearproof silicone with an imprinted surface pattern mimicking human glabrous skin, found e.g. in fingertips. Its very thin layer structure (starting from 2.5 mm) facilitates integration on existing robot surfaces, particularly on small and highly curved links. For example, we augmented our Shadow Dexterous Hand with 12 palm sensors, and 2 resp. 3 sensors in the middle resp. proximal phalanges of each finger. To demonstrate the usefulness and efficiency of the proposed sensor skin, we performed a challenging classification task distinguishing squeezed objects based on their varying stiffness.

A necessary skill when using liquids in the preparation of food is to be able to estimate viscosity, e.g. in order to control the pouring velocity or to determine the thickness of a sauce. We introduce a method to allow a robotic kitchen assistant discriminate between different but visually similar liquids. Using a Kinect depth camera, surface changes, induced by a simple pushing motion, are recorded and used as input to nearest neighbour and polynomial regression classification models. Results reveal that even when the classifier is trained on a relatively small dataset it generalises well to unknown containers and liquid fill rates. Furthermore, the regression model allows us to determine the approximate viscosity of unknown liquids.

A cascade of revolutions has transformed robotics into a new science that begins to link physical concepts of control and interaction with qualities and concepts analyzed so far mainly in disciplines such as psychology, biology, linguistics or the social sciences.

We present a method for real-time bare hand tracking that utilizes natural hand synergies to reduce the complexity and improve the plausibility of the hand posture estimation. The hand pose and posture are estimated by fitting a virtual hand model to the 3D point cloud obtained from a Kinect camera using an inverse kinematics approach. We use real human hand movements captured with a Vicon motion tracking system as the ground truth for deriving natural hand synergies based on principal component analysis. These synergies are integrated in the tracking scheme by optimizing the posture in a reduced parameter space. Tracking in this reduced space combined with joint limit avoidance constrains the posture estimation to natural hand articulations. The information loss associated with dimension reduction can be dealt with by employing a hierarchical optimization scheme. We show that our synergistic hand tracking approach improves runtime performance and increases the quality of the posture estimation.

Increasingly widespread available haptic sensors mounted on articulated hands offer new sensory channels that can complement shape extraction from vision to enable a more robust handling of objects in cases when vision is restricted or even unavailable. However, to estimate object shape from haptic interaction data is a difficult challenge due to the complexity of the contact interaction between the movable object and sensor surfaces, leading to a coupled estimation problem of shape and object pose. While for vision efficient solutions to the underlying SLAM problem are known, the available information is much sparser in the tactile case, posing great difficulties for a straightforward adoption of standard SLAM algorithms. In the present paper, we thus explore whether a biologically inspired model based on dynamic neural fields can offer a route towards a practical algorithm for tactile SLAM. Our study is focused on a restricted scenario where a two-fingered robot hand manipulates an n-gon with a fixed rotational axis. We demonstrate that our model can accumulate shape information from reasonably short interaction sequences and autonomously build a representation despite significant ambiguity of the tactile data due to the rotational periodicity of the object. We conclude that the presented framework may be a suitable basis to solve the tactile SLAM problem also in more general settings which will be the focus of subsequent work.

Brain-machine interfaces open a direct channel between a brain and a robot. This channel is commonly used to provide direct and active input to the robot, resulting in a tele-operation system. We argue in favor of a more passive brain-machine interface as a means for human-robot interaction. There, the brain signals of the human interaction partner are constantly monitored and decoded to detect particular states that correlate with events in the robot's behavior. Such a state can be surprise due to a strange or erroneous robot action. We review three studies that we conducted with our own EEG-based brain-robot interface framework. The interface is active, that is, we directly control humanoid robots in different application scenarios in a semi-autonomous manner. Our results show that automated and unconscious components in the EEG are the most robust and acceptable for the user. These are exactly the components that are useful for a passive interface. Finally, we present a pilot study where we extract correlates of human surprise from an interaction with a real humanoid robot. We show that, currently in offline analysis, we are able to extract similar components used in the structured, stimulus based active interfaces. We pinpoint the issues that need to be solved, such as a more reliable real-time decoding of brain signals from real-world interaction situations.

We propose a feedback-based solution for the accurate manipulation of an unknown object in hand. This method does not explicitly models friction and surface geometry details, but employs a fast feedback loop based on visual and tactile feedback to perform robust manipulation even in the presence of unexpected slippage or rolling. At every control step, fingertip motions are computed to realize the intended object relocation, employing a composite position/force controller. Subsequently inverse hand kinematics is employed to retrieve joint-level motions, which are implemented on the robot with a position servo loop. We evaluate our method on a setup of two KUKA robot arms, each equipped with a tactile sensor array as end-effectors to perform the object manipulation task. The experimental results show the feasibility of our proposed method, even in presence of slippage or external disturbances.

Eye movements play an essential role in planning and executing manual actions. Eye-hand coordination is a natural human skill. We exploit this skill for an intuitive remote manipulation system that allows even non-expert users to operate a robot safely without prior experience. Specifically, we propose a visio-haptic approach to controlling a 7-DOF robotic arm. Our system is fully mobile, allowing for unconstraint operation in any environment. An eyetracker captures the operator's gaze. The end effector or particular joints are selected by simply fixating the to-be-controlled segment. A sensor-equipped tangible object provides a haptic interface between the operator's hand and the focused part of the robotic arm. The system features two operation modes, direct joint rotation and 3d end effector control in a global cartesian frame. We evaluated the system in a proof-of-concept study with untrained users. The participants safely operated the robot and accomplished an obstacle avoidance task. For this purpose, they used both operation modes.

We present a real-time algorithm that segments unstructured and highly cluttered scenes. The algorithm robustly separates objects of unknown shape in congested scenes of stacked and occluded objects. The model-free approach finds smooth surface patches, using a depth image from a Kinect camera, which are subsequently combined to form highly probable object hypotheses. Coplanarity and curvature matching is used to recombine surfaces separated by occlusion. The real-time capabilities are proven and the quality of the algorithm is evaluated on a benchmark database. Advantages compared to existing approaches as well as weaknesses are discussed.

We implemented a system, called the VICON-EyeTracking Visualizer, that combines mobile eye tracking data with motion capture data to calculate and visualize the 3D gaze vector within the motion capture co-ordinate system. To ensure that both devices were temporally synchronized we used previously developed software by us. By placing reflective markers on objects in the scene, their positions are known and by spatially synchronizing both the eye tracker and the motion capture system allows us to automatically compute how many times and where fixations occur, thus overcoming the time consuming and error-prone disadvantages of the traditional manual annotation process. We evaluated our approach by comparing its outcome for a simple looking task and a more complex grasping task against the average results produced by the manual annotation process. Preliminary data reveals that the program only differed from the average manual annotation results by approximately 3 percent in the looking task with regard to the number of fixations and cumulative fixation duration on each point in the scene. In case of the more complex grasping task the results depend on the object size: for larger objects there was good agreement (less than 16 percent (or 950ms)), but this degraded for smaller objects, where there are more saccades towards object boundaries. The advantages of our approach are easy user calibration, the ability to have unrestricted body movements (due to the mobile eye-tracking system), and that it can be used with any wearable eye tracker and marker based motion tracking system. Extending existing approaches, our system is also able to monitor fixations on moving objects. The automatic analysis of gaze and movement data in complex 3D scenes can be applied to a variety of research domains, i. e., Human Computer Interaction, Virtual Reality or grasping and gesture research.

We present an algorithm to segment an unstructured table top scene. Operating on the depth image of a Kinect camera, the algorithm robustly separates objects of previously unknown shape in cluttered scenes of stacked and partially occluded objects. The model-free algorithm finds smooth surface patches which are subsequently combined to form object hypotheses. We evaluate the algorithm regarding its robustness and real-time capabilities and discuss its advantages compared to existing approaches as well as its weak spots to be addressed in future work. We also report on an autonomous grasping experiment with the Shadow Robot Hand which employs the estimated shape and pose of segmented objects.

In this paper we showcase an integrative approach for our actuated Tangible Active Objects (TAOs), that demonstrates distributed collaboration support to become a versatile and comprehensive dynamic user interface with multi-modal feedback. We incorporated physical actuation, visual projection in 2D and 3D, and vibro-tactile feedback. We demonstrate this approach in a furniture placing scenario where the users can interactively change the furniture model represented by each TAO using a dial-based tangible actuated menu. We demonstrate virtual constraints between our TAOs to automatically maintain spatial relations.

Tactile sensing is an important field of research in the domains of human-computer and human-robot interaction. To provide appropriate tactile sensing capabilities, this work presents the development of a new modular tactile sensor system focusing especially on high frame rates (up to 1.9 kHz) and good spatial resolution (5 mm). Larger sensor areas are composed from identical sensor modules providing a 16×16 matrix of tactels. We compare different tactel layouts and different force-sensitive materials to achieve optimal sensitivity especially to low forces in order to facilitate detection of first touch. An example application demonstrates the capability of the developed sensor to detect tiny variations in applied force.

In this paper we propose an approach for unsupervised segmentation of continuous object manipulation sequences into semantically differing subsequences. The proposed method estimates segment borders based on an integrated consideration of three modalities (tactile feedback, hand posture, audio) yielding robust and accurate results in a single pass. To this end, a Bayesian approach originally applied by Fearnhead to segment one-dimensional time series data -- is extended to allow an integrated segmentation of multi-modal sequences. We propose a joint product model which combines modality-specific likelihoods to model segments. Weight parameters control the influence of each modality within the joint model. We discuss the relevance of all modalities based on an evaluation of the temporal and structural correctness of segmentation results obtained from various weight combinations.

We propose the concept of a brain-computer interface interaction system that allows patients to virtually use non-verbal interaction affordances, in particular gestures and facial expressions, by means of a humanoid robot. Here, we present a pilot study on controlling such a robot via a hybrid BCI. The results indicate that users can intuitively address interaction partners by looking in their direction and employ gestures and facial expressions in every-day interaction situations.

The ability to manipulate deformable objects, such as textiles or paper, is a major prerequisite to bringing the capabilities of articulated robot hands closer to the level of manual intelligence exhibited by humans. We concentrate on the manipulation of paper, which affords us a rich interaction domain and that has not yet been solved for anthropomorphic robot hands. A key ability needed for this is the robust tracking and modelling of paper under conditions of occlusion and strong deformation. We present a marker based framework that realizes these properties robustly and in real-time. We compare a purely mathematical representation of the paper manifold with a soft-body-physics model and demonstrate the use of our visual tracking method to facilitate the coordination of two anthropomorphic 20 DOF Shadow Dexterous Hands while they grasp a flat-lying piece of paper, using a combination of visually guided bulging and pinching.

To enable the creation of manual interaction databases, aiding the replication of dexterous capabilities with anthropomorphic robot hands by utilizing information about how humans perform complex manipulation tasks, requires the capability to record and analyze large amounts of manual interaction sequences. For this goal we have studied and compared three mappings from captured human hand motion data to a simulated model, which allow for robust and accurate real-time hand posture tracking. We evaluate the effectiveness of these mappings and discuss their pros and cons in various real-world scenarios. The first method is based on data glove data and aims for direct gaging of hand joints. The other two methods utilize a VICON motion tracking system which monitors markers placed on all finger segments. Here we compare two approaches: a direct computation of hand postures from angles between adjacent markers and an iterative inverse kinematics approach to optimally reproduce fingertip positions. For a quantitative evaluation, we employ a “calibration objects” technique to obtain a reliable ground truth of task-relevant hand posture data.

In this paper, we present a probabilistic spatial approach to build compact 3-D representations of unknown objects probed by tactile sensors. Our approach exploits the high frame rates provided by modern tactile sensors and utilizes Kalman filters to build a probabilistic model of the contact point cloud that is efficiently stored in a kd-tree. The quality of generated shape representations is compared with a naive averaging approach, and we show that our method provides superior accuracy. We also evaluate the feasibility of object classification combining the generated object representations, together with the iterative closest point algorithm.

Head-mounted displays (HMDs) that use a see-through display method allow for superimposing computer-generated images upon a real-world view. Such devices, however, normally restrict the user's field of view. Furthermore, low display resolution and display curvature are suspected to make foveal as well as peripheral vision more difficult and may thus affect visual processing. In order to evaluate this assumption, we compared performance and eye-movement patterns in a visual search paradigm under different viewing conditions: participants either wore an HMD, had their field of view restricted by blinders or could avail themselves of an unrestricted field of view (normal viewing). From the head and eye-movement recordings we calculated the contribution of eye rotation to lateral shifts of attention. Results show that wearing an HMD leads to less eye rotation and requires more head movements than under blinders conditions and during normal viewing.

We have developed a novel extension of the NEAT neuroevolution method, termed NEATfields, to solve problems with large input and output spaces. NEATfields networks are layered into two-dimensional fields of identical or similar subnetworks with an arbitrary topology. The subnetworks are evolved with genetic operations similar to those used in the NEAT neuroevolution method. We show that information processing within the neural fields can be organized by providing suitable building blocks to evolution. NEATfields can solve a number of visual discrimination tasks and a newly introduced multiple pole balancing task.

The recognition and prediction of intersection situations and an accompanying threat assessment are an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2-4 s ahead. Possible situations involving two vehicles are generated by a pairwise combination of these individual motion trajectories. An interaction model based on the mutual visibility of the vehicles and the assumption that a driver will attempt to avoid a collision is used to rate possible situations. The correspondingly favoured situations are classified with a probabilistic framework. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of about 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1.5-3 s before the last vehicle has passed its minimum distance to the centre of the intersection.

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2-4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1-2 seconds before the distance to the intersection centre becomes minimal.

We introduce a greedy algorithm that works from coarse to fine by iteratively applying localized principal component analysis (PCA). The decision where and when to split or add new components is based on two antagonistic criteria. Firstly, the well known quadratic reconstruction error and secondly a measure for the homogeneity of the distribution. For the latter criterion, which we call “generation error”, we compared two different possible methods to assess if the data samples are distributed homogeneously. The proposed algorithm does not involve a costly multi-objective optimization to find a partition of the inputs. Further, the final number of local PCA units, as well as their individual dimensionality need not to be predefined. We demonstrate that the method can flexibly react to different intrinsic dimensionalities of the data.

Transferring human skills to dextrous robots in an easy, fast and robust way is one of the key challenges that still have to be tackled in order to bring robots to our every-day life. However, many problems remain unsolved. In particular, researchers are seeking new paradigms along with efficient and robust task representations that facilitate adaptation to new contexts and provide a means to appropriately react to unforeseen situations. In this paper, we present a new method for robot behaviour synthesis, where intrinsic characteristics of `Structured UKR manifolds' [13] are used to derive a closed-loop controller based on motion data obtained by the `Robot Skill Synthesis via Human Learning' paradigm [10]. We apply the method to the task of swapping Chinese health balls with a real 16 DOF robotic hand. Our results indicate that the marriage of `Structured UKR manifolds' with the `Robot Skill Synthesis via Human Learning' paradigm yields an efficient way of realising a dexterous manipulation capability on real robots.

Task learning from observations of non-expert human users will be a core feature of future cognitive robots. However, the problem of task segmentation has only received minor attention. In this paper, we present a new approach to classifying and segmenting series of observations into a set of candidate motions. As basis for these candidates, we use structured UKR manifolds, a modified version of unsupervised kernel regression which has been introduced in order to easily reproduce and synthesise represented dextrous manipulation tasks. Together with the presented mechanism, it then realises a system that is able both to reproduce and recognise the represented motions.

The ability to assess the relevance of topics and related sources in information-rich environments is a key to success when scanning business environments. This paper introduces a hybrid system to support managerial information gathering. The system is made up of three components: 1) a hierarchical hyperbolic SOM for structuring the information environment and visualizing the intensity of news activity with respect to identified topics, 2) a spreading activation network for the selection of the most relevant information sources with respect to an already existing knowledge infrastructure, and 3) measures of interestingness for association rules as well as statistical testing facilitates the monitoring of already identified topics. Embedding the system by a framework describing three modes of human information seeking behavior endorses an active organization, exploration and selection of information that matches the needs of decision makers in all stages of the information gathering process. By applying our system in the domain of the hotel industry we demonstrate how typical information gathering tasks are supported. Moreover, we present an empirical study investigating the effectiveness and efficiency of the visualization framework of our system.

Service robots which directly interact with humans in highly unstructured, unpredictable and dynamic environments must be able to flexibly adapt their motion in reaction to unforeseen events or obstacles and they must provide a new feasible trajectory in real-time. Hence, algorithms come into focus which replan the motion path and its time evolution from arbitrary initial conditions within milliseconds. We present a real-time algorithm to generate synchronised and time-optimal third-order manipulator trajectories complying maximal motion limits on velocity, acceleration and jerk. Experimental results carried out on a Mitsubishi PA10-7C arm are presented.

In dextrous manipulation, the implementation of manipulation movements still is a complex and intricate undertaking. Often, a lot of object physics and modelling effort has to be incorporated into a controller working only for a very restricted task specification and performing quite artificially looking movements. In this paper, we present the first steps towards a representation of manipulation movements recorded from human demonstration which facilitates later application and promotes natural motion. We use manifolds of hand postures embedded in the finger joint angle space which are constructed such that manipulation parameters including the advance in time are represented by distinct manifold dimensions. This allows for purposive navigation within such manifolds. We present the manifold construction using the Unsupervised Kernel Regression (UKR) and the way of applying it for manipulation in the example of turning a bottle cap in a physics-based simulation.

We present a database of 2D pressure profile time series as a testbed for tactile object and surface recognition. The tactile database captures the surfaces of household and toy objects by moving a 2D pressure sensor mounted to an industrial robot arm around the objects using real-time trajectory calculation. Thus, it represents different "views" of the objects in a similar way as the well known Columbia Object Image Library (COIL) captures different views of an object by a camera. As a first application, objects in the database are classified using a neural network architecture.

We present a strategy for grasping of real world objects with two anthropomorphic hands, the three-fingered 9- DOF hydraulic TUM and the very dextrous 20-DOF pneumatic Bielefeld Shadow Hand. Our approach to grasping is based on a reach-pre-grasp-grasp scheme loosely motivated by human grasping. We comparatively describe the two robot setups, the control schemes, and the grasp type determination. We show that the grasp strategy can robustly cope with inaccurate control and object variation. We demonstrate that it can be ported among platforms with minor modifications. Grasping success is evaluated by comparative experiments performing a benchmark test on 21 everyday objects.

Algorithms for dextrous robot grasping always have to cope with the challenge of achieving high object specialisation for a wide range of grasping contexts. In this paper, we present a tactile-driven approach that dynamically uses the robot's grasping experience to address this issue. During the grasp movement, the current contact information is used to dynamically adapt the grasping control by targeting the best matching posture from the experience base. Thus, the robot recalls and actuates a grasp it already successfully performed in a similar tactile context. To efficiently represent the experience, we introduce the grasp manifold assuming that grasp postures form a smooth manifold in hand posture space. We present a simple way of providing approximations of grasp manifolds using self-organising maps (SOMs). The algorithm is evaluated on three different geometry primitives - box, cylinder and sphere - in a physics-based computer simulation.

We present a biologically motivated architecture for object recognition that is capable of online learning of several objects based on interaction with a human teacher. The system combines biological principles such as appearance-based representation in topographical feature detection hierarchies and context-driven transfer between different levels of object memory. Training can be performed in an unconstrained environment by presenting objects in front of a stereo camera system and labeling them by speech input. The learning is fully online and thus avoids an artificial separation of the interaction into training and test phases. We demonstrate the performance on a challenging ensemble of 50 objects.

We present an on-line, robust, and efficient path planner for the redundant Mitsubishi PA-10 arm with 7 degrees of freedom (DOF) in non-stationary environments. Because of the specific kinematic model of the arm, path planning can be first reduced to a redundant 6-DOF problem in a 5D configuration space, which can be further decomposed into two problems: (i) 3D position planning in Cartesian space and (ii) planning in a 3D space composed of two orientation angles and an explicit parameterization of the arms redundancy. Position and orientation planning are interweaving and performed "on-the-fly" without explicit global knowledge of the environment using two instances of the dynamic wave expansion neural network (DWENN), an effective method for path generation in arbitrarily changing environments. The dynamic and explorative nature of the DWENN algorithm allows to treat stationary and dynamic obstacles in a unified manner. Through a number of simulative tests, we show that the planner is capable of reaching both a satisfactory robustness level and real-time performance, as required by many practical applications

We present a hybrid learning method bridging the fields of recurrent neural networks, unsupervised Hebbian learning, vector quantization, and supervised learning to implement a sophisticated image and feature segmentation architecture. This architecture is based on the competitive layer model (CLM), a dynamic feature binding model, which is applicable on a wide range of perceptual grouping and segmentation problems. A predefined target segmentation can be achieved as attractor states of this linear threshold recurrent network, if the lateral weights are chosen by Hebbian learning. The weight matrix is given by the correlation matrix of special pattern vectors with a structure dependent on the target labeling. Generalization is achieved by applying vector quantization on pair-wise feature relations, like proximity and similarity, defined by external knowledge. We show the successful application of the method to a number of artifical test examples and a medical image segmentation problem of fluorescence microscope cell images.

We apply a hierarchical feed-forward neural architecture to the problem of face recognition. The network is similar to the neocognitron-approach and a two-layer variation of this architecture, which has previously been successfully applied to patch classification tasks. We extend this architecture to a three-layer one, which allows not only identification of image patches, but also detection in larger images. In the research area of face recognition, a lot of expertise has been developed for the problem of either identification or detection, but approaches which deal with both problems simultaneously are rarely to be found. In this work, we apply the hierarchical approach to this problem and evaluate the performance on artificial datasets.

Our research focuses on the development of a mobile augmented reality system which is capable of acquiring image data in an unrestricted environment and which provides a comfortable facility to label this data. To structure the image data modified MPEG-7 features are computed and by means of self organizing maps (SOM), the imagery can be labeled stepwise. First, the complete data set is projected onto the SOM using a combination of color and edge features. In a second step, selected parts of the imagery are retrained weighting the feature blocks depending on characteristics of the acquired image data. Within few steps, the partitioning leads to SOM nodes on which the projected imagery can be labeled as objects or rejected.

We present an on-line decentralized approach to collision-free path planning for two robot arms. During the real-time planning, each arm represents a dynamic obstacle for another one, which allows to treat the motion of the latter, as well as the motion of other objects in the workspace in a unified fashion. The motion for each arm is planned independently, and the only information which is "shared" is the intended configuration of each robot. The approach relies therefore exclusively on the dynamic, explorative path generation, which is performed using the dynamic wave expansion neural network. Our simulative experiments for the case of two robot arms with 3-DOFs in 3D reveal that the proposed approach, without any complicated heuristics, any priority assignment, and any global optimization of an objective cost function, is capable of producing feasible paths "on-the-fly". The robustness and efficiency of the method are demonstrated statistically through a number of random tests.

We propose a nonparametric approach to learning of principal surfaces based on an unsupervised formulation of the Nadaraya-Watson kernel regression estimator. As compared with previous approaches to principal curves and surfaces, the new method offers several advantages: First, it provides a practical solution to the model selection problem because all parameters can be estimated by leave-one-out cross-validation without additional computational cost. In addition, our approach allows for a convenient incorporation of nonlinear spectral methods for parameter initialization, beyond classical initializations based on linear PCA. Furthermore, it shows a simple way to fit principal surfaces in general feature spaces, beyond the usual data space setup. The experimental results illustrate these convenient features on simulated and real data.

This paper introduces Crystallization Sonification , a sonification model for exploratory analysis of high-dimensional datasets. The model is designed to provide information about the intrinsic data dimensionality (which is a local feature) and the global data dimensionality, as well as the transitions between a local and global view on a dataset. Furthermore the sound allows to display the clustering in high-dimensional datasets. The model defines a crystal growth process in the high-dimensional data-space which starts at a user selected “condensation nucleus” and incrementally includes neighboring data according to some growth criterion. The sound summarizes the temporal evolution of this crystal growth process. For introducing the model, a simple growth law is used. Other growth laws which are used in the context of hierarchical clustering are also suited and their application in crystallization sonification offers new ways to inspect the results of data clustering as an alternative to dendrogram plots. In this paper, the sonification model is described and example sonifications are presented for some synthetic high-dimensional datasets.

We discuss the framework of Model-Based Sonification (MBS) and its contribution to a principled design of mediators between high-dimensional data spaces and perceptual spaces, particularly sound spaces. Data Crystallization Sonification, discussed in the reprinted paper, exemplifies the design of sonification models according to this framework. Finally, promising lines of development in this area are pointed out, concerning generalizations, applications, and open research directions.

In This work we present an approach using incrementally constructed neural gas networks to 'grow' an intuitive interface for interactive exploratory sonification of high-dimensional data. The sonifications portray information about the intrinsic data dimensionality and its variation within the data space. The interface follows the paradigm of model-based sonification and consists of a graph of nodes that can be acoustically 'excited' with simple mouse actions. The sound generation process is defined in terms of the node parameters and the graph topology, following a physically motivated model of energy flow through the graph structure. The resulting sonification model is tied to the given data set by constructing both graph topology and node parameters by an adaptive, fully data-driven learning process, using a growing neural gas network. We report several examples of applying this method to static data sets and point out a generalization to the task of process analysis.

Data-driven fMRI analysis techniques include independent component analysis (ICA) and different types of clustering in the temporal domain. Since each of these methods has its particular strengths, it is natural to look for an approach that unifies Kohonen's self-organizing map and ICA. This is given by the topographic independent component analysis. While achieved by a slight modification of the ICA model, it can be at the same time used to define a topographic order (clusters) between the components, and thus has the usual computational advantages associated with topographic maps. In this contribution, we can show that when applied to fMRI analysis it outperforms FastICA.

Auditory data display is an interdisciplinary field linking auditory perception research, sound engineering, data mining, and human-computer interaction in order to make semantic contents of data perceptually accessible in the form of (nonverbal) audible sound. For this goal it is important to understand the different ways in which sound can encode meaning. We discuss this issue from the perspectives of language, music, functionality, listening modes, and physics, and point out some limitations of current techniques for auditory data display, in particular when targeting high-dimensional data sets. As a promising, potentially very widely applicable approach, we discuss the method of model-based sonification (MBS) introduced recently by the authors and point out how its natural semantic grounding in the physics of a sound generation process supports the design of sonifications that are accessible even to untrained, everyday listening. We then proceed to show that MBS also facilitates the design of an intuitive, active navigation through "acoustic aspects", somewhat analogous to the use of successive two-dimensional views in three-dimensional visualization. Finally, we illustrate the concept with a first prototype of a "tangible" sonification interface which allows us to "perceptually map" sonification responses into active exploratory hand motions of a user, and give an outlook on some planned extensions.

We propose the combination of two recently introduced methods for the interactive visual data mining of large collections of data. Both hyperbolic multidimensional scaling (HMDS) and hyperbolic self-organizing maps (HSOM) employ the extraordinary advantages of the hyperbolic plane (H2): (i) the underlying space grows exponentially with its radius around each point deal for embedding high-dimensional (or hierarchical) data; (ii) the Poincare model of the IH/sup 2/ exhibits a fish-eye perspective with a focus area and a context preserving surrounding; (in) the mouse binding of focus-transfer allows intuitive interactive navigation. The HMDS approach extends multidimensional scaling and generates a spatial embedding of the data representing their dissimilarity structure as faithfully as possible. It is very suitable for interactive browsing of data object collections, but calls for batch precomputation for larger collection sizes. The HSOM is an extension of Kohonen's self-organizing map and generates a partitioning of the data collection assigned to an IH/sup 2/ tessellating grid. While the algorithm's complexity is linear in the collection size, the data browsing is rigidly bound to the underlying grid. By integrating the two approaches, we gain the synergetic effect of adding advantages of both. And the hybrid architecture uses consistently the IH/sup 2/ visualization and navigation concept. We present the successfully application to a text mining example involving the Reuters-21578 text corpus.

In this paper, we present a neural network model that realizes a dynamic version of the distance transform algorithm (used for path planning in a stationary domain). The novel version is capable of performing path generation for highly dynamic environments. The neural network has discrete-time dynamics, is locally connected, and, hence, computationally efficient. No preliminary information about the world status is required for the planning process. Path generation is performed via the neural-activity landscape, which forms a dynamically-updating potential field over a distributed representation of the configuration space of a robot. The network dynamics guarantees local adaptations and includes a set of strict rules for determining the next step in the path for a robot. According to these rules, planned paths tend to be optimal in a L/sub 1/ metric. Simulation results in a series of experiments for various dynamical situations prove the effectiveness of the proposed model.

This paper presents two contributions: (i) a new type of neural network the dynamic wave expansion neural network, for path generation in a dynamic environment for both mobile robots and robotic manipulators, and (ii) the simulative comparisons to known discrete-time neural network models - the classical resistive grid model, and the Hopfield-type neural network, proposed by Glasius et al. The network has discrete-time dynamics, is locally connected, highly parallel, and hence, computationally efficient. The model does not require any a-priory information about the environment. The path is generated according to a neural-activity landscape, which forms a dynamically updating scalar potential field over a distributed representation of the configuration space of a robot. The simulations reveal that the proposed model yields dominantly shorter paths, especially in highly-dynamic environments.

We present a content-based image retrieval system, INDI (techniques for Intelligent Navigation in Digital Image databases), that combines the use of low-level pattern recognition techniques, machine learning and an intuitive human-computer interface in order to support intelligent and user-friendly semantic navigation in large image databases. To keep independence from specific image domains and to encompass different search tasks, the system is highly modular and contains a hierarchical mechanism for the adaptive reweighting of similarity measures implemented by dynamically reloadable modules at different semantic levels.

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.

We propose a novel projection based visualization method for high-dimensional datasets by combining concepts from MDS and the geometry of the hyperbolic spaces. Our approach Hyperbolic Multi-Dimensional Scaling (H-MDS) extends earlier work [7] using hyperbolic spaces for visualization of tree structures data ( "hyperbolic tree browser" ).By borrowing concepts from multi-dimensional scaling we map proximity data directly into the 2-dimensional hyperbolic space (H2). This removes the restriction to "quasihierarchical", graph-based data -- limiting previous work. Since a suitable distance function can convert all kinds of data to proximity (or distance-based) data this type of data can be considered the most general.We used the circular Poincaré model of the H2 which allows effective human-computer interaction: by moving the "focus" via mouse the user can navigate in the data without loosing the "context". In H2 the "fish-eye" behavior originates not simply by a non-linear view transformation but rather by extraordinary, non-Euclidean properties of the H2. Especially, the exponential growth of length and area of the underlying space makes the H2 a prime target for mapping hierarchical and (now also) high-dimensional data.We present several high-dimensional mapping examples including synthetic and real world data and a successful application for unstructured text. By analyzing and integrating multiple film critiques from news:rec.art.movies.reviews and the internet movie database, each movie becomes placed within the H2. Here the idea is, that related films share more words in their reviews than unrelated. Their semantic proximity leads to a closer arrangement. The result is a kind of high-level content structured display allowing the user to explore the "space of movies".

In this paper we present results of a study on brain computer interfacing. We adopted an approach of Farwell & Donchin [4], which we tried to improve in several aspects. The main objective was to improve the trans- fer rates based on offline analysis of EEG-data but within a more realistic setup closer to an online realization than in the original studies. The ob- jective was achieved along two different tracks: on the one hand we used state-of-the-art machine learning techniques for signal classification and on the other hand we augmented the data space by using more electrodes for the interface. For the classification task we utilized SVMs and, as mo- tivated by recent findings on the learning of discriminative densities, we accumulated the values of the classification function in order to combine several classifications, which finally lead to significantly improved rates as compared with techniques applied in the original work. In combina- tion with the data space augmentation, we achieved competitive transfer rates at an average of 50.5 bits/min and with a maximum of 84.7 bits/min.

We propose a framework for classifier design based on discriminative densities for representation of the differences of the class-conditional dis- tributions in a way that is optimal for classification. The densities are selected from a parametrized set by constrained maximization of some objective function which measures the average (bounded) difference, i.e. the contrast between discriminative densities. We show that maximiza- tion of the contrast is equivalent to minimization of an approximation of the Bayes risk. Therefore using suitable classes of probability den- sity functions, the resulting maximum contrast classifiers (MCCs) can approximate the Bayes rule for the general multiclass case. In particular for a certain parametrization of the density functions we obtain MCCs which have the same functional form as the well-known Support Vec- tor Machines (SVMs). We show that MCC-training in general requires some nonlinear optimization but under certain conditions the problem is concave and can be tackled by a single linear program. We indicate the close relation between SVM- and MCC-training and in particular we show that Linear Programming Machines can be viewed as an approxi- mate realization of MCCs. In the experiments on benchmark data sets, the MCC shows a competitive classification performance.

This paper describes GREFIT (Gesture REcognition based on FInger Tips), a neural network-based system which recognizes continuous hand postures from gray-level video images (posture capturing). Our approach yields a full identification of all finger joint angles (making, however, some assumptions about joint couplings to simplify computations). This allows a full reconstruction of the three-dimensional (3-D) hand shape, using an articulated hand model with 16 segments and 20 joint angles. GREFIT uses a two-stage approach to solve this task. In the first stage, a hierarchical system of artificial neural networks (ANNs) combined with a priori knowledge locates the two-dimensional (2-D) positions of the finger tips in the image. In the second stage, the 2-D position information is transformed by an ANN into an estimate of the 3-D configuration of an articulated hand model, which is also used for visualization. This model is designed according to the dimensions and movement possibilities of a natural human hand. The virtual hand imitates the user's hand to an remarkable accuracy and can follow postures from gray scale images at a frame rate of 10 Hz.

A distributed robot control system is proposed based on a temporal self-organizing neural network, called competitive and temporal Hebbian (CTH) network. The CTH network can learn and recall complex trajectories by means of two sets of synaptic weights, namely, competitive feedforward weights that encode the individual states of the trajectory and Hebbian lateral weights that encode the temporal order of trajectory states. Complex trajectories contain repeated or shared states which are responsible for ambiguities that occur during trajectory reproduction. Temporal context information are used to resolve such uncertainties. Furthermore, the CTH network saves memory space by maintaining only a single copy of each repeated/shared state of a trajectory and a redundancy mechanism improves the robustness of the network against noise and faults. The distributed control scheme is evaluated in point-to-point trajectory control tasks using a PUMA 560 robot. The performance of the control system is discussed and compared with other unsupervised and supervised neural network approaches. We also discuss the issues of stability and convergence of feedforward and lateral learning schemes.

A major goal for the realization of a new generation of intelligent robots is the capability of instructing work tasks by interactive demonstration. To make such a process efficient and convenient for the human user requires that both the robot and the user can establish and maintain a common focus of attention. We describe a hybrid architecture that combines neural networks and finite stale machines into a flexible framework for controlling the behaviour of a vision based robot called GRAVIS-robot (Gestural Recognition Active Vision System robot). It consists of a binocular camera head, a 6 DOF robot arm and a 9 DOF multifingered hand. We focus primarily on nonverbal communication based on gestural commands of a human instructor which will at a later stage be complemented by spoken instructions.

We suggest a nonparametric framework for unsupervised learning of projection models in terms of density estimation on quantized sample spaces. The objective is not to optimally reconstruct the data but in- stead the quantizer is chosen to optimally reconstruct the density of the data. For the resulting quantizing density estimator (QDE) we present a general method for parameter estimation and model selection. We show how projection sets which correspond to traditional unsupervised meth- ods like vector quantization or PCA appear in the new framework. For a principal component quantizer we present results on synthetic and real- world data, which show that the QDE can improve the generalization of the kernel density estimator although its estimate is based on significantly lower-dimensional projection indices of the data.

We introduce a new type of Self-Organizing Map (SOM) to navigate in the Semantic Space of large text collections. We propose a “hyper- bolic SOM” (HSOM) based on a regular tesselation of the hyperbolic plane, which is a non-euclidean space characterized by constant negative gaussian curvature. The exponentially increasing size of a neighborhood around a point in hyperbolic space provides more freedom to map the complex information space arising from language into spatial relations. We describe experiments, showing that the HSOM can successfully be applied to text categorization tasks and yields results comparable to other state-of-the-art methods.

A distributed robot control system is proposed based on a temporal self-organizing neural network, called a competitive temporal Hebbian (CTH) network. The CTH network can learn and recall complex trajectories using two sets of synaptic weights, namely competitive feedforward weights that encode the individual states of the trajectory and Hebbian lateral weights that encode the temporal order of the trajectory states. Ambiguities that occur during trajectory reproduction are resolved using temporal context information. Also, the CTH network saves memory space by maintaining only a single copy of each repeated/shared state of a complex trajectory. A distributed processing scheme is proposed to evaluate the CTH network in the point-to-point real-time trajectory control of a Puma 560 robot. The performance of the control system is discussed and compared with other neural network approaches.

In the domain of unsupervised learning, mixtures of gaussians have become a popular tool for statistical modeling. For this class of generative models, we present a complexity control scheme, which provides an effective means for avoiding the problem of overfitting usually encountered with unconstrained (mixtures of) gaussians in high dimensions. According to some prespecified level of resolution as implied by a fixed variance noise model, the scheme provides an automatic selection of the dimensionalities of some local signal subspaces by maximum likelihood estimation. Together with a resolution-based control scheme for adjusting the number of mixture components, we arrive at an incremental model refinement procedure within a common deterministic annealing framework, which enables an efficient exploration of the model space. The advantages of the resolution-based framework are illustrated by experimental results on synthetic and high-dimensional real-world data.

We establish two conditions that ensure the nondivergence of additive recurrent networks with unsaturating piecewise linear transfer functions, also called linear threshold or semilinear transfer functions. As Hahnloser, Sarpeshkar, Mahowald, Douglas, and Seung (2000) showed, networks of this type can be efficiently built in silicon and exhibit the coexistence of digital selection and analog amplification in a single circuit. To obtain this behavior, the network must be multistable and nondivergent, and our conditions allow determining the regimes where this can be achieved with maximal recurrent amplification. The first condition can be applied to nonsymmetric networks and has a simple interpretation of requiring that the strength of local inhibition match the sum over excitatory weights converging onto a neuron. The second condition is restricted to symmetric networks, but can also take into account the stabilizing effect of nonlocal inhibitory interactions. We demonstrate the application of the conditions on a simple example and the orientation-selectivity model of Ben-Yishai, Lev Bar-Or, and Sompolinsky (1995). We show that the conditions can be used to identify in their model regions of maximal orientation-selective amplification and symmetry breaking.

We present a recurrent neural network for feature binding and sensory segmentation: the competitive-layer model (CLM). The CLM uses topographically structured competitive and cooperative interactions in a layered network to partition a set of input features into salient groups. The dynamics is formulated within a standard additive recurrent network with linear threshold neurons. Contextual relations among features are coded by pairwise compatibilities, which define an energy function to be minimized by the neural dynamics. Due to the usage of dynamical winner-take-all circuits, the model gains more flexible response properties than spin models of segmentation by exploiting amplitude information in the grouping process. We prove analytic results on the convergence and stable attractors of the CLM, which generalize earlier results on winner-take-all networks, and incorporate deterministic annealing for robustness against local minima. The piecewise linear dynamics of the CLM allows a linear eigensubspace analysis, which we use to analyze the dynamics of binding in conjunction with annealing. For the example of contour detection, we show how the CLM can integrate figure-ground segmentation and grouping into a unified model.

A neural cell detection system (NCDS) for the automatic quantitation of fluorescent lymphocytes in tissue sections is presented in this paper. The system acquires visual knowledge from a set of training cell-image patches selected by a user. The trained system evaluates an image in 2 min calculating: the number, the positions, and the phenotypes of the fluorescent cells. For validation, the NCDS learning performance was tested by cross validation on digitized images of tissue sections obtained from inherently different types of tissue: diagnostic tissue sections across the human tonsil and across an inflammatory lymphocyte infiltrate of the human skeletal muscle. The NCDS detection results were compared with detection results from biomedical experts and were visually evaluated by our most experienced biomedical expert. Although the micrographs were noisy and the fluorescent cells varied in shape and size, the NCDS detected a minimum of 95% of the cells. In contrast, the cellular counts based on visual cell recognition of the experts were inconsistent and largely unreproducible for approximately 80% of the lymphocytes present in a visual field. The data indicate that the NCDS is rapid and delivers highly reproducible results and, therefore, enables high-throughput topological screening of lymphocytes in many types of tissue, e.g., as obtained by routine diagnostic biopsy procedures. High-throughput screening with the NCDS provides the platform for the quantitative analysis of the interrelationship between tissue environment, cellular phenotype, and cellular topology.

A three stage recognition architecture that can be trained to different recognition or segmentation tasks is presented. It consists of an adaptive feature extraction based on vector quantization and local PCA. The features are classified by neural expert networks. It is shown that the system can be applied to object classification, segmentation of partially occluded objects and classification of object parts without modifications in the architecture.

An algorithm for supervised learning the segmentation of partially occluded objects is presented. It is based on the classification of object windows which are small compared to the object size but large enough to evaluate structural object features as well as colour. From the input windows, features are extracted by local principal component analysis and subsequently classified by a neural network of the local linear map type. The performance is checked on images of objects with partial occlusion which were artificially generated from the Columbia Object Image Library.

We present the application of a recurrent neural network feature binding model to the segmentation of fluorescence micrographs, images showing fluorescent cells in tonsil tissue. Image primitives, referred to as features, consisting of position and local gradient information, build the input to the model. The competitive layer model is used to provide a binding of features to convex groups, corresponding to fluorescent cell bodies. Although the images contain noise, and the cells' shapes show considerable variation, the fluorescent cell contours are extracted with sufficient accuracy, according to a biomedical expert. The method achieves at the same time grouping and figure-ground segmentation, and does not require us to manually fix the number of groups.

This paper describes the use of neural network for gesture recognition based on finger tips, a system that recognizes continuous hand postures from video images. Our approach yields a full identification of all finger joint angles. This allows a full reconstruction of the 3D hand shape, using an artificial hand model with 16 segments and 20 joint angles. The focus of the present paper is how to employ a parametrised SOM neural network for the inverse kinematics task to compute the angles of a hand model out of 3D positions of the fingertips. We show that this type of neural net does not only achieve excellent results from very few training examples, but also can be applied to uncommon data structures.

Topology-representing networks, such as the SOM and the growing neural gas (GNG) are powerful tools for the adaptive formation of maps of feature and state spaces for a broad range of applications. However, these algorithms suffer severe difficulties when their training inputs are strongly correlated. This makes them unsuitable for the online formation of maps of state spaces whose exploration occurs most naturally along trajectories, which is typical in many applications in the fields of robotics and process control. Based on investigations of the SOM and the GNG for these cases, we devise a new network model, the "instantaneous topological map" (ITM) that is able to overcome these difficulties and form maps from strongly correlated stimulus sequences in a fast and robust manner. This makes the ITM highly suitable for mapping of state spaces in control tasks in general and especially in robotics, where workspace limitations are complex and probably more easily explored than analyzed and coded by hand.

Genetic programming (GP) is emerging as a promising technique for an impressive variety of computational tasks. An important prerequisite for exploring new applications of GP is a versatile and distributed simulation environment for evolving genetic programs with flexibly specifiable processing primitives and types. We consider the field of computer vision as a particularly interesting and challenging domain and analyze some requirements for the application of GP to the example-based synthesis of computer vision algorithms. We then present an outline of the GENCODER2 (GENetic COde DEvelopeR) distributed simulation environment, which was designed with particular attention to these demands. As an application example, we report results for the genetic construction of algorithms for the detection and classification of cell bodies in microscopic images. The generated segmentation operators allow accurate viability determination for suspension cultures, which plays a fundamental role in cell culture technology.

Humans easily recognize where another person is looking and often use this information for interspeaker coordination. We present a method based on three neural networks of the local linear map type which enables a computer to identify the head orientation of a user by learning from examples. One network is used for color segmentation, a second for localization of the face, and the third for the final recognition of the head orientation. The system works at a frame rate of one image per second on a common workstation, We analyze the accuracy achieved at different processing steps and discuss the usability of the approach in the context of a visual human-machine interface.

Tactile sensor systems are an essential prerequisite for the implementation of complex manipulation and exploration tasks using robots. The desire to perform real-time control and pattern recognition with tactile sensors led us to the design of a cost-effective artificial fingertip. At our laboratory two distinct types of sensors are in use: force/position sensors and slippage detectors. We report on first experimental results with a fingertip prototype performing rolling and sliding movements over a flat surface. To facilitate experimentation with tactile sensors, we designed and developed a data acquisition and transportation system that fulfils our demands on bandwidth, flexibility, and cost. This system consists of two hardware components, a configurable multichannel analog signal sampler to acquire sensor data, and an intelligent dual-ported random-access buffer to avoid data transportation bottlenecks.

Correlation-based learning (CBL) has been suggested as the mechanism that underlies the development of simple-cell receptive fields in the primary visual cortex of cats, including orientation preference (OR) and ocular dominance (OD) (Linsker, 1986; Miller, Keller, & Stryker, 1989). CBL has been applied successfully to the development of OR and OD individually (Miller, Keller, & Stryker, 1989; Miller, 1994; Miyashita & Tanaka, 1991; Erwin, Obermayer, & Schulten, 1995), but the conditions for their joint development have not been studied (but see Erwin & Miller, 1995, for independent work on the same question) in contrast to competitive Hebbian models (Obermayer, Blasdel, & Schulten, 1992). In this article, we provide insight into why this has been the case: OR and OD decouple in symmetric CBL models, and a joint development of OR and OD is possible only in a parameter regime that depends on nonlinear mechanisms.

With the availability of more powerful computers it is nowadays possible to perform pixel based operations on real camera images even in the full color space. New adaptive classification tools like neural networks make it possible to develop special-purpose object detectors that can segment arbitrary objects in real images with a complex distribution in the feature space after training with one or several previously labeled image(s). The paper focuses on a detailed comparison of a neural approach based on local linear maps (LLMs) to a classifier based on normal distributions. The proposed adaptive segmentation method uses local color information to estimate the membership probability in the object, respectively, background class. The method is applied to the recognition and localization of human hands in color camera images of complex laboratory scenes.

We present an object recognition architecture based on feature extraction by Gabor filter kernels and feature classification by an artificial neural network. The parameters of the Gabor filters are optimized to the specific problem by minimizing an energy function. Such Gabor filters extract features that can be more easily classified by the neural network. Moreover, the feature space is low-dimensional so feature extraction does not require much computational effort. The object recognition system is implemented on a Datacube and works in real-time.

Incrementally constructed cascade architectures are a promising alternative to networks of predefined size. This paper compares the direct cascade architecture (DCA) proposed in Littmann and Ritter (1992) to the cascade-correlation approach of Fahlman and Lebiere (1990) and to related approaches and discusses the properties on the basis of various benchmark results. One important virtue of DCA is that it allows the cascading of entire subnetworks, even if these admit no error-backpropagation. Exploiting this flexibility and using LLM networks as cascaded elements, we show that the performance of the resulting network cascades can be greatly enhanced compared to the performance of a single network. Our results for the Mackey-Glass time series prediction task indicate that such deeply cascaded network architectures achieve good generalization even on small data sets, when shallow, broad architectures of comparable size suffer from overfitting. We conclude that the DCA approach offers a powerful and flexible alternative to existing schemes such as, e.g., the mixtures of experts approach, for the construction of modular systems from a wide range of subnetwork types.

Introduces the method of "template-based image generation" (TBIG) which is based on self-organizing neural networks and aims at producing images and animations from still images. The main characteristics of the method are (a) storage of image data in self-organizing maps (SOMs) and (b) a dual image representation which requires the reliable automatic identification of interest points; the latter task is solved by a hierarchical analysis/synthesis system based on neural nets. Current work focuses on images of the human face. The new possibilities of storing, encoding and manipulating images are integrated in the "FaceCoder" system which is intended to work with image data in the same way as vocoders do with speech data. Sequences of expressions or articulatory movements are "transferred" from an image sequence to a still image. Applications are compact image encoding, production of phantom images for forensic use, and special effects for entertainment purposes.>

It is shown that for a class of vector quantization processes, related to neural modeling, that the asymptotic density Q(x ) of the quantization levels in one dimension in terms of the input signal distribution P(x) is a power law Q(x)=C-P(x)(alpha ), where the exponent alpha depends on the number n of neighbors on each side of a unit and is given by alpha=2/3-1/(3n (2)+3[n+1](2)). The asymptotic level density is calculated, and Monte Carlo simulations are presented.

A discussion is presented of the properties of the model in the high-dimensional limit. The authors present results from a Monte Carlo simulation indicating a facilitation of the map-ordering process if the dimensionality of the input space is increased, and they provide a mathematical analysis of the development of localized receptive fields via the input-selection mechanism. The analytical results are compared with data from large-scale simulations using a Connection Machine CM-2, and very good agreement is found

The problem of predicting highly nonlinear time sequence data, where the usual approach using adaptive linear regressive models encounters difficulty, is considered. For this case, the use of an adaptive covering of the state space of the process with a set of linear regressive models, each of which is only locally used, is suggested. It is shown that such an adaptive covering, together with learning of the appropriate prediction coefficients, can be realized using Kohonen's algorithm of self-organizing maps. To illustrate the method, simulation results for a set of benchmarking problems are given

An extension of T. Kohonen's (1982) self-organizing mapping algorithm together with an error-correction scheme based on the Widrow-Hoff learning rule is applied to develop a learning algorithm for the visuomotor coordination of a simulated robot arm. Learning occurs by a sequence of trial movements without the need for an external teacher. Using input signals from a pair of cameras, the closed robot arm system is able to reduce its positioning error to about 0.3% of the linear dimensions of its work space. This is achieved by choosing the connectivity of a three-dimensional lattice consisting of the units of the neural net.