Hierarchical world models can significantly improve model-based reinforcement learning (MBRL) and planning by enabling reasoning across multiple time scales. Nonetheless, the majority of state-of-the-art MBRL methods employ flat, non-hierarchical models. We propose Temporal Hierarchies from Invariant Context Kernels (THICK), an algorithm that learns a world model hierarchy via discrete latent dynamics. The lower level of THICK updates parts of its latent state sparsely in time, forming invariant contexts. The higher level exclusively predicts situations involving context changes. Our experiments demonstrate that THICK learns categorical, interpretable, temporal abstractions on the high level, while maintaining precise low-level predictions. Furthermore, we show that the emergent hierarchical predictive model seamlessly enhances the abilities of MBRL or planning methods. We believe that THICK contributes to the further development of hierarchical agents capable of more sophisticated planning and reasoning abilities.

We introduce a compositional physics-aware FInite volume Neural Network (FINN) for learning spatiotemporal advection-diffusion processes. FINN implements a new way of combining the learning abilities of artificial neural networks with physical and structural knowledge from numerical simulation by modeling the constituents of partial differential equations (PDEs) in a compositional manner. Results on both one- and two-dimensional PDEs (Burgers’, diffusion-sorption, diffusion-reaction, Allen{–}Cahn) demonstrate FINN’s superior modeling accuracy and excellent out-of-distribution generalization ability beyond initial and boundary conditions. With only one tenth of the number of parameters on average, FINN outperforms pure machine learning and other state-of-the-art physics-aware models in all cases{—}often even by multiple orders of magnitude. Moreover, FINN outperforms a calibrated physical model when approximating sparse real-world data in a diffusion-sorption scenario, confirming its generalization abilities and showing explanatory potential by revealing the unknown retardation factor of the observed process.

The ability to flexibly bind features into coherent wholes from different perspectives is a hallmark of cognition and intelligence. Importantly, the binding problem is not only relevant for vision but also for general intelligence, sensorimotor integration, event processing, and language. Various artificial neural network models have tackled this problem with dynamic neural fields and related approaches. Here we focus on a generative encoder-decoder architecture that adapts its perspective and binds features by means of retrospective inference. We first train a model to learn sufficiently accurate generative models of dynamic biological motion or other harmonic motion patterns, such as a pendulum. We then scramble the input to a certain extent, possibly vary the perspective onto it, and propagate the prediction error back onto a binding matrix, that is, hidden neural states that determine feature binding. Moreover, we propagate the error further back onto perspective taking neurons, which rotate and translate the input features onto a known frame of reference. Evaluations show that the resulting gradient-based inference process solves the perspective taking and binding problem for known biological motion patterns, essentially yielding a Gestalt perception mechanism. In addition, redundant feature properties and population encodings are shown to be highly useful. While we evaluate the algorithm on biological motion patterns, the principled approach should be applicable to binding and Gestalt perception problems in other domains.

A common approach to prediction and planning in partially observable domains is to use recurrent neural networks (RNNs), which ideally develop and maintain a latent memory about hidden, task-relevant factors. We hypothesize that many of these hidden factors in the physical world are constant over time, changing only sparsely. To study this hypothesis, we propose Gated $L_0$ Regularized Dynamics (GateL0RD), a novel recurrent architecture that incorporates the inductive bias to maintain stable, sparsely changing latent states. The bias is implemented by means of a novel internal gating function and a penalty on the $L_0$ norm of latent state changes. We demonstrate that GateL0RD can compete with or outperform state-of-the-art RNNs in a variety of partially observable prediction and control tasks. GateL0RD tends to encode the underlying generative factors of the environment, ignores spurious temporal dependencies, and generalizes better, improving sampling efficiency and overall performance in model-based planning and reinforcement learning tasks. Moreover, we show that the developing latent states can be easily interpreted, which is a step towards better explainability in RNNs.

Virtual reality technologies, along with motion based input devices allow for the design of innovative interfaces between learners and digital knowledge resources. These interfaces might facilitate knowledge work in educational and scientific contexts. Compared to 2D interfaces, immersive 3D environments provide greater flexibility regarding the interface design, however, so far no general, theory-driven and validated design principles are available. Seeing that complex learning environments can foster the development of various cognitive abilities, like multiperspective reasoning skills (MPRS), such design principles are highly desirable. Using multiperspective hypermedia environments (MHEs) as a testbed, the presented project aims to identify and evaluate design principles, derived from cognitive science. We will create and study interactive, immersive 3D-interface to MHEs using virtual reality technology. To evaluate the developed system, we will contrast the acquisition of MPRS in 2D and 3D learning environments. We expect that the developed design principles will be directly applicable for enhancing the accessibility of other knowledge environments.

One of the central tasks for a household robot is searching for specific objects. It does not only require localizing the target object but also identifying promising search locations in the scene if the target is not immediately visible. As computation time and hardware resources are usually limited in robotics, it is desirable to avoid expensive visual processing steps that are exhaustively applied over the entire image. The human visual system can quickly select those image locations that have to be processed in detail for a given task. This allows us to cope with huge amounts of information and to efficiently deploy the limited capacities of our visual system. In this paper, we therefore propose to use human fixation data to train a top-down saliency model that predicts relevant image locations when searching for specific objects. We show that the learned model can successfully prune bounding box proposals without rejecting the ground truth object locations. In this aspect, the proposed model outperforms a model that is trained only on the ground truth segmentations of the target object instead of fixation data.

It has been shown previously that the control of a robot arm can be efficiently learned using the XCSF learning classifier system, which is a nonlinear regression system based on evolutionary computation. So far, however, the predictive knowledge about how actual motor activity changes the state of the arm system has not been exploited. In this paper, we utilize the forward velocity kinematics knowledge of XCSF to alleviate the negative effect of noisy sensors for successful learning and control. We incorporate Kalman filtering for estimating successive arm positions, iteratively combining sensory readings with XCSF-based predictions of hand position changes over time. The filtered arm position is used to improve both trajectory planning and further learning of the forward velocity kinematics. We test the approach on a simulated kinematic robot arm model. The results show that the combination can improve learning and control performance significantly. However, it also shows that variance estimates of XCSF prediction may be underestimated, in which case self-delusional spiraling effects can hinder effective learning. Thus, we introduce a heuristic parameter, which can be motivated by theory, and which limits the influence of XCSF's predictions on its own further learning input. As a result, we obtain drastic improvements in noise tolerance, allowing the system to cope with more than 10 times higher noise levels.

It was previously shown that the control of a robot arm can be efficiently learned using the XCSF classifier system. So far, however, the predictive knowledge about how actual motor activity changes the state of the arm system has not been exploited. In this paper, we exploit the forward velocity kinematics knowledge of XCSF to alleviate the negative effect of noisy sensors for successful learning and control. We incorporate Kalman filtering for estimating successive arm positions iteratively combining sensory readings with XCSF-based predictions of hand position changes over time. The filtered arm position is used to improve both trajectory planning and further learning of the forward velocity kinematics. We test the approach on a simulated, kinematic robot arm model. The results show that the combination can improve learning and control performance significantly. However, it also shows that variance estimates of XCSF predictions maybe underestimated, in which case self-delusional spiraling effects hinder effective learning. Thus, we introduce a heuristic parameter, which limits the influence of XCSF's predictions on its own further learning input. As a result, we obtain drastic improvements in noise tolerance coping with more than ten times higher noise levels.

High-dimensional problems are challenging for iterative, online (Michigan-style) Learning Classifier Systems, especially because of the large size of the evolutionary search space. The present work proposes to guide mutation toward most suitable classifier condition structures. Focusing on XCSF, we introduce a guided mutation operator: Each classifier stores a finite set of matched samples. Mutation uses those samples to optimize classifier condition shapes by means of an accuracy-weighted covariance matrix. While this approach does not necessarily produce an optimal shape, it is sufficient to discriminate relevant from irrelevant dimensions. Regular evolutionary operators handle the fine tuning. We show that guided XCSF does not only drastically speed up learning, but it also solves higher-dimensional problems than regular XCSF. Experiments illustrate that guided XCSF quickly detects the intrinsic structure of non-linear, oblique, fully sampled, ten-dimensional approximation tasks. The results show that guided XCSF strongly outperforms regular XCSF as well as a statistics-based machine learning approach, the so called Locally Weighted Projection Regression algorithm.

When soft robots act in new or unforeseen situations, uncertainty becomes a major challenge. Sensors might fail and the forward kinematics might become faulty if limbs are deformed. To meet this challenge, we propose a model that uses additional sensors in different modalities and provides a highly modular body model which can be easily adapted. Longterm sensor-blackout is covered by usage of multiple sensors in different modalities, while short term sensor-blackout is covered by the maintenance of state representations over time and the inclusion of a full forward and inverse model suitable for prediction and simulation. Even more, as the state is represented redundantly in multiple modalities, the system compares the estimates to detect changes in sensor noise or even sensor offsets and discards failing sensors. Finally, the highly modularized architecture of our representation simplifies the adaptation of the body model.

Many automotive systems use linear approaches to track and predict other traffic participants. While this may be appropriate on highways, linear predictions do not work properly on curved roads or lane crossings. This contribution introduces a generic way for including environmental knowledge - such as the lane trajectory ahead - to anticipate yaw rate and acceleration of other traffic participants. The anticipatory knowledge is used to improve prediction in filtering tasks. It is embedded in a Bayesian framework by introducing attractors, which modify the probabilistic propagation of state estimations. The attractors model how traffic participants typically behave, given environmental knowledge such as lane information, traffic lights, or indicator lights. We demonstrate the potential of this approach by modeling the fact that vehicles usually stay in their lane. We show that given correct context information and nonlinear traffic situations, the tracking error is considerably lower compared to conventional tracking methods. In addition, we also show that the intentions of other traffic participants may be inferred by comparing actual sensory data with anticipated probability distributions, which were generated dependent on alternative attractors.

XCSF approximates function surfaces by evolving a suitable clustering of the input space, so that a simple -- typically linear -- predictor yields sufficient accuracy in each cluster. With an increasing number of distinct output dimensions, however, the accuracy of local predictions typically decreases. We analyze the performance of a single XCSF instance and compare it to the performance of a multiple-instance XCSF, where each instance predicts one dimension of the output. We show that dependent on the problem at hand, the multiple-instance XCSF approach is highly advantageous. In particular, we show that the more local linearity structures differ, the more a modularized approximation by multiple XCSF instances pays off. In fact, if modularization is not applied, the problem complexity may increase exponentially in the number of approximately orthogonally-structured output dimensions. To relate these results also to current XCSF application options, we show that the multiple-instance XCSF approach can also be applied to the problem of learning a compact model of the Jacobian of the forward-kinematics of a seven degree of freedom anthropomorphic robot arm for inverse robot arm control in simulation.

When dealing with light-weight robots with nonrigid limbs and joints and uncertain sensory readings, configuration state representations inevitably are approximate. Due to various types of sensory readings, which are usually body-grounded in different frames of reference, these configuration states may naturally be represented modularly distributed. From a different perspective, computational models of human motor planning suggest that the brain represents current body postures, such as the state of an arm, modularly within various frames of reference. Moreover, the different information sources, such as proprioceptive, anticipatory, and visual information, are probabilistically integrated. As a basis for such a representation, we propose a modular system that maintains a distributed self-representation in interactive frames of reference. We show that the resulting representation is highly noise robust and may be used to reach goals within various frames of reference while also considering other task constraints.

Neuroscientific research suggests that the human brain encodes spatial information in a Bayesian-optimal way by means of distributed, neural population codes. In this paper we apply this concept to Advanced Driver Assistance Systems, introducing a grid-based population code for tracking and predicting the behavior of individual vehicles. The representation encodes a spatially distributed hidden Markov model of current and future vehicle locations and velocities. Predictive information and additional sensory information are integrated over time by means of Bayesian filters. Performance of the system is compared with a Kalman Filter in an overtaking maneuver in a simulated environment. It is shown that the grid-based approach excels Kalman-Filtering performance in several situations, where the Gaussian distribution and linear system assumptions of the Kalman filter are strongly violated. Moreover, the grid-based approach allows the flexible incorporation of additional behavioral assumptions. When the approach assumes that the tracked vehicle will stay in its lane, the probability distribution can be even more favorably focused and unexpected lane changes can be detected.

The TORCS-based Simulated Car Racing Championship (SCRC) poses a demanding challenge for designing an effective racing car controller. Controllers do not receive any global track information, but only perceive simulated, car-centered sensory information about the current, local track properties and about surrounding opponents. Our racing controller, termed COgnitive BOdySpaces: TORCS-based Adaptive Racing (COBOSTAR), uses the sensors that give the most anticipatory information to learn a sensory-to-motor policy, which was optimized by means of the covariance matrix adaptation evolution strategy (CMA-ES). The basic policy was extended by additional modules to prevent detrimental skidding, to safely land after jumps, to implement effective opponent avoidance, and to recover and learn online from accidents. Owing to this approach, COBOSTAR won two out of the three competition legs of the 2009 SCRC and, without hardly any further modifications, also was among the first in the 2010 SCRC despite the addition of noise to the utilized sensors. This paper describes the COBOSTAR controller as it was submitted to the last of the three competition legs in 2009. Evaluations of distinct controller modules are provided where possible. A future outlook summarizes the lessons learned during the design of the racer and proposes the utilization of the framework also in broader research and application contexts.

In this paper, the advantages of introducing an additional amount of tests when evolving parameters for specific purposes is discussed. A set of optimal PID-controller parameters are sought for an exemplary system, which simulates a human-like robotic arm. When evolving the controller parameters, the number of different movements included in the optimization process is varied. By including extra movements to the optimization process, the time it takes to evolve the parameters does increase, but the uncertainty due to noise is correspondingly lowered. Additionally, it is shown that the added movements, which improve robustness of the system, do not significantly lower the overall performance of the resulting system, when utilizing the evolved parameters

In this paper, we overview the 2009 Simulated Car Racing Championship-an event comprising three competitions held in association with the 2009 IEEE Congress on Evolutionary Computation (CEC), the 2009 ACM Genetic and Evolutionary Computation Conference (GECCO), and the 2009 IEEE Symposium on Computational Intelligence and Games (CIG). First, we describe the competition regulations and the software framework. Then, the five best teams describe the methods of computational intelligence they used to develop their drivers and the lessons they learned from the participation in the championship. The organizers provide short summaries of the other competitors. Finally, we summarize the championship results, followed by a discussion about what the organizers learned about 1) the development of high-performing car racing controllers and 2) the organization of scientific competitions.

XCS has been shown to be an effective genetics-based classification, datamining, and reinforcement learning tool. The systems learns suitable, compact, maximally general problem solutions online. In the robotics and cognitive systems domains, however, applications of XCSF are very sparse and mostly restricted to small, symbolic problems. Recently, a sensorimotor XCSF system was applied to cognitive arm control. In this paper, we show how this XCSF-based armcontrol mechanisms can be extended (1) to efficiently exploit redundant behavioral alternatives and (2) to guide the control of dynamic arm plants. The XCSF system encodes redundant alternatives in its inverse control representations and resolves the encoded redundancies dependent on current constraints--such as arm posture preferences - on the fly. An adaptive PD controller translates the XCSF-based direction and distance commands into actual motor commands for dynamic arm control. We apply the complete system to the control of a simulated, physical arm with three degrees of freedom in a two-dimensional environment and to a simulation of the industrial KR16 Kuka arm with ODE-based physics engine.

This paper presents a class of NK landscapes with nearest-neighbor interactions and tunable overlap. The considered class of NK landscapes is solvable in polynomial time using dynamic programming; this allows us to generate a large number of random problem instances with known optima. Several genetic and evolutionary algorithms are then applied to the generated problem instances. The results are analyzed and related to scalability theory for genetic algorithms and estimation of distribution algorithms.

Many successful applications have proven the potential of Learning Classifier Systems and the XCS classifier system in particular in datamining, reinforcement learning, and function approximation tasks. Recent research has shown that XCS is a highly flexible system, which can be adapted to the task at hand by adjusting its condition structures, learning operators, and prediction mechanisms. However, fundamental theory concerning the scalability of XCS dependent on these enhancements and problem difficulty is still rather sparse and mainly restricted to boolean function problems. In this article we developed a learning scalability theory for XCSF---the XCS system applied to real-valued function approximation problems. We determine crucial dependencies on functional properties and on the developed solution representation and derive a theoretical scalability model out of these constraints. The theoretical model is verified with empirical evidence. That is, we show that given a particular problem difficulty and particular representational constraints XCSF scales optimally. In consequence, we discuss the importance of appropriate prediction and condition structures regarding a given problem and show that scalability properties can be improved by polynomial orders, given an appropriate, problem-suitable representation.

In this paper, we analyze the behavior of a simulated mobile robot, which interacts with an initially unknown maze-environment. The robot is controlled by an interactive system that is based on a model building Time Growing Neural Gas (TGNG) algorithm and a homeostatic motivational system, which activates movement preferences and goals within the emergent model structure for behavioral control. We propose to differentiate two types of drives (if not more), which we call location- and characteristics-based drives. We exemplary implement the two types of drives by ldquohungerrdquo and ldquofearrdquo, respectively. Several possible methods of combination of the two drives are investigated through simulation, identifying the combination that lead to the most suitable emergent behavior, such as emergent ldquowall-followingrdquo and ldquohidingrdquo. Moreover, we investigate performance in an ALife-like scenario, in which the robot interacts with several food-dispensers. It is shown that additional behavioral concepts, such as ldquocuriosityrdquo and ldquoinhibition of returnrdquo, can maximize the survival chances of the organism, who maintains maximal safety and keeps its belly full. In conclusion, we propose that the concept of motivation needs to be further differentiated to realize autonomous, life-like robots that are able to optimally satisfy multiple, competing types of motivations by emergent, innovative behavioral patterns.

While John Holland has always envisioned learning classifier systems (LCSs) as cognitive systems, most work on LCSs has focused on classification, datamining, and function approximation. In this paper, we show that the XCSF classifier system can be very suitably modified to control a robot system with redundant degrees of freedom, such as a robot arm. Inspired by recent research insights that suggest that sensorimotor codes are nearly ubiquitous in the brain and an essential ingredient for cognition in general, the XCSF system is modified to learn classifiers that encode piecewise linear sensorimotor structures, which are conditioned on prediction-relevant contextual input. In the investigated robot arm problem, we show that XCSF partitions the (contextual) posture space of the arm in such a way that accurate hand movements can be predicted given particular motor commands. Furthermore, we show that the inversion of the sensorimotor predictive structures enables accurate goal-directed closed-loop control of arm reaching movements. Besides the robot arm application, we also investigate performance of the modified XCSF system on a set of artificial functions. All results point out that XCSF is a useful tool to evolve problem space partitions that are maximally effective for the encoding of sensorimotor dependencies. A final discussion elaborates on the relation of the taken approach to actual brain structures and cognitive psychology theories of learning and behavior.

We investigate rule matching in learning classifier systems for problems involving binary and real inputs. We consider three rule encodings: the widely used character-based encoding, a specificity-based encoding, and a binary encoding used in Alecsys. We compare the performance of the three algorithms both on matching alone and on typical test problems. The results on matching alone show that the population generality influences the performance of the matching algorithms based on string representations in different ways. Character-based encoding becomes slower and slower as generality increases, specificity-based encoding becomes faster and faster as generality increases. The results on typical test problems show that the specificity-based representation can halve the time required for matching but also that binary encoding is about ten times faster on the most difficult problems. Moreover, we extend specificity-based encoding to real-inputs and propose an algorithm that can halve the time require for matching real inputs using an interval-based representation.

Recent advances in XCS technology have shown that self-adaptive mutation can be highly useful to speed-up the evolutionary progress in XCS. Moreover, recent publications have shown that XCS can also be successfully applied to challenging real-valued domains including datamining, function approximation, and clustering. In this paper, we combine these two advances and investigate self-adaptive mutation in the XCS system for function approximation with hyperellipsoidal condition structures, referred to as XCSF in this paper. It has been shown that XCSF solves function approximation problems with an accuracy, noise robustness, and generalization capability comparable to other statistical machine learning techniques and that XCSF outperforms simple clustering techniques to which linear approximations are added. This paper shows that the right type of self-adaptive mutation can further improve XCSF's performance solving problems more parameter independent and more reliably. We analyze various types of self-adaptive mutation and show that XCSF with self-adaptive mutation ranges,differentiated for the separate classifier condition values, yields most robust performance results. Future work may further investigate the properties of the self-adaptive values and may integrate advanced self-adaptation techniques.

An important strength of learning classifier systems (LCSs) lies in the combination of genetic optimization techniques with gradient-based approximation techniques. The chosen approximation technique develops locally optimal approximations, such as accurate classification estimates, Q-value predictions, or linear function approximations. The genetic optimization technique is designed to distribute these local approximations efficiently over the problem space. Together, the two components develop a distributed, locally optimized problem solution in the form of a population of expert rules, often called classifiers. In function approximation problems, the XCSF classifier system develops a problem solution in the form of overlapping, piecewise linear approximations. This paper shows that XCSF performance on function approximation problems additively benefits from: 1) improved representations; 2) improved genetic operators; and 3) improved approximation techniques. Additionally, this paper introduces a novel closest classifier matching mechanism for the efficient compaction of XCS's final problem solution. The resulting compaction mechanism can boil the population size down by 90% on average, while decreasing prediction accuracy only marginally. Performance evaluations show that the additional mechanisms enable XCSF to reliably, accurately, and compactly approximate even seven dimensional functions. Performance comparisons with other, heuristic function approximation techniques show that XCSF yields competitive or even superior noise-robust performance.

We analyze generalization and learning in XCS with gradient descent. At first, we show that the addition of gradient in XCS may slow down learning because it indirectly decreases the learning rate. However, in contrast to what was suggested elsewhere, gradient descent has no effect on the achieved generalization. We also show that when gradient descent is combined with roulette wheel selection, which is known to be sensitive to small values of the learning rate, the learning speed can slow down dramatically. Previous results reported no difference in the performance of XCS with gradient descent when roulette wheel selection or tournament selection were used. In contrast, we suggest that gradient descent should always be combined with tournament selection, which is not sensitive to the value of the learning rate. When gradient descent is used in combination with tournament selection, the results show that (i) the slowdown in learning is limited and (ii) the generalization capabilities of XCS are not affected.

Learning classifier systems (LCSs), introduced by John H. Holland in the 1970s, are rule-based evolutionary online learning systems that combine gradient-based rule evaluation with evolutionary-based rule structuring techniques. Since the introduction of the accuracy-based XCS classifier system by Stewart W. Wilson in 1995, LCSs showed to be flexible, online learning methods that are applicable to datamining, reinforcement learning, and function approximation problems. Comparisons showed that performance is competitive with state-of-the art machine learning algorithms, but the learning algorithms applied are usually more flexible and highly adaptive. Moreover, problem knowledge can be extracted easily. This tutorial provides a gentle introduction to LCSs and their general functioning. It then gives further details on the XCS classifier system and highlights various successful applications. In conclusion, promising future directions of LCS research and applications are discussed.

Psychological studies have shown immense behavioral flexibility in arm reaching tasks. Intermanual learning transfer (ILT) tasks have shown that both reaching movements adapt to distorted spaces rather rapidly and the adaptation generalizes to the behavior of other limbs. In this paper, we present an ILT experiment and replicate it with feedforward neural network (NN) architectures. We show that the NN architecture is the key to successfully replicating the experiments. Moreover, we show that dependent on the architecture and the initial training schedule applied, an internal space representation emerges that enables ILT. The results confirm that internal body spaces, identified in neuroscience and cognitive psychological research, can emerge solely due to an interdependence between different limb movements and the right neural architecture. We hypothesize that, in order to develop internal spatial representations observed in animals and humans, it might be sufficient to enforce the integration of multiple correlated sensory and motor information into one compact internal representation.

Many neural network models of (human) motor learning focus on the acquisition of direct goal-to-action mappings, which results in rather inflexible motor control programs. We propose a neural network architecture (SURE_REACH) that acquires complete body models through unsupervised learning. It encodes redundancy on the kinematic and on the motor command level in order to exert highly flexible, task-dependent optimal control. This paper shows that our approach accounts for two forms of effective human behavior based on exploiting kinematic redundancy. First, depending on the starting posture, hand targets are pursued in different ways optimizing movement efficiency. Second, the arm posture at the end of a movement can be aligned anticipatorily to facilitate a subsequent movement. A discussion of computational implications and relations to behavioral and neurophysiological findings concludes the paper.

The learning classifier system XCS is an iterative rule-learning system that evolves rule structures based on gradient-based prediction and rule quality estimates. Besides classification and reinforcement learning tasks, XCS was applied as an effective function approximator. Hereby, XCS learns space partitions to enable a maximally accurate and general function approximation. Recently, the function approximation approach was improved by replacing (1) hyperrectangular conditions with hyper-ellipsoids and (2) iterative linear approximation with the recursive least squares method. This paper combines the two approaches assessing the usefulness of each. The evolutionary process is further improved by changing the mutation operator implementing an angular mutation that rotates ellipsoidal structures explicitly. Both enhancements improve XCS performance in various non-linear functions. We also analyze the evolving ellipsoidal structures confirming that XCS stretches and rotates the evolving ellipsoids according to the shape of the underlying function. The results confirm that improvements in both the evolutionary approach and the gradient approach can result in significantly better performance.

Recently, studies with the XCS classifier system on Boolean functions have shown that in certain types of functions simple crossover operators can lead to disruption and, consequently, a more effective recombination mechanism is required. Simple crossover operators were replaced by recombination based on estimation of distribution algorithms (EDAs). The combination showed that XCS with such a statistics-based crossover operator can solve challenging hierarchical functions more efficiently. This study elaborates the gained competence further investigating the coding scheme for the EDA component (BOA in our case) of XCS as well as performance in randomly generated Boolean function problems. Results in hierarchical Boolean functions show that the originally used 2-bit coding scheme induces a certain learning bias that stresses additional diversity in the evolving XCS population. A 1-bit coding scheme as well as a restricted 2-bit coding scheme confirm the suspected bias. The alternative encodings decrease the unnecessary bias towards specificity and increase performance robustness. The paper concludes with a discussion on the challenges ahead for XCS in Boolean function problems as well as on the implications of the obtained results for real-valued and multiple-valued classification problems, multi-step problems, and function approximation problems.

... This paper describes a class of random additively decomposable problems with and without interactions between the subproblems and tests hBOA on a large number of random instances of the proposed class of problems. The performance of hBOA is compared to that of the simple genetic algorithm with standard crossover and mutation operators, the univariate marginal distribution algorithm, and the hill climbing with bitflip mutation. The results confirm that hBOA achieves quadratic or subquadratic performance on the proposed class of random decomposable problems and that it significantly outperforms all other methods included in the comparison. The results also show that low-order polynomial scalability is retained even when only a small percentage of hardest problems are considered and that hBOA is a robust algorithm because its performance does not change much across the entire spectrum of random problem instances of the same structure. The proposed class of decomposable problems can be used to test other optimization algorithms that address nearly decomposable problems

Learning Classifier Systems (LCSs), such as the accuracy-based XCS, evolve distributed problem solutions represented by a population of rules. During evolution, features are specialized, propagated, and recombined to provide increasingly accurate subsolutions. Recently, it was shown that, as in conventional genetic algorithms (GAs), some problems require efficient processing of subsets of features to find problem solutions efficiently. In such problems, standard variation operators of genetic and evolutionary algorithms used in LCSs suffer from potential disruption of groups of interacting features, resulting in poor performance. This paper introduces efficient crossover operators to XCS by incorporating techniques derived from competent GAs: the extended compact GA (ECGA) and the Bayesian optimization algorithm (BOA). Instead of simple crossover operators such as uniform crossover or one-point crossover, ECGA or BOA-derived mechanisms are used to build a probabilistic model of the global population and to generate offspring classifiers locally using the model. Several offspring generation variations are introduced and evaluated. The results show that it is possible to achieve performance similar to runs with an informed crossover operator that is specifically designed to yield ideal problem-dependent exploration, exploiting provided problem structure information. Thus, we create the first competent LCSs, XCS/ECGA and XCS/BOA, that detect dependency structures online and propagate corresponding lower-level dependency structures effectively without any information about these structures given in advance.

Many learning classifier system (LCS) implementations are restricted to the binary problem realm. Recently, the XCS classifier system was enhanced to be able to handle real-valued inputs among others. In the real-valued enhancement, XCSF applies as a function approximation system that partitions the input space in hyperrectangular subspaces specified in the classifiers. This paper changes the classifier conditions to hyperspheres and hyperellipsoids and investigates the consequent performance impact. It is shown that the modifications yield improved performance in continuous functions. Even in discontinuous functions with parallel boundaries, XCS's performance does not degrade. Thus, for the real-valued problem domain, ellipsoidal condition structures can improve XCS's performance. From a more general perspective, this paper shows that XCS is readily applicable in diverse problem domains. To apply the system even more successfully, suitable kernel-based bases need to be found and used as classifier conditions. XCS distributes the available structures over the problem space evolving more specialized structures in more complex problem subspaces.

Michigan-style learning classifier systems (LCSs), such as the accuracy-based XCS system, evolve distributed problem solutions represented by a population of rules. Recently, it was shown that decomposable problems may require effective processing of subsets of problem attributes, which cannot be generally assured with standard crossover operators. A number of competent crossover operators capable of effective identification and processing of arbitrary subsets of variables or string positions were proposed for genetic and evolutionary algorithms. This paper effectively introduces two competent crossover operators to XCS by incorporating techniques from competent genetic algorithms (GAs): the extended compact GA (ECGA) and the Bayesian optimization algorithm (BOA). Instead of applying standard crossover operators, here a probabilistic model of the global population is built and sampled to generate offspring classifiers locally. Various offspring generation methods are introduced and evaluated. Results indicate that the performance of the proposed learning classifier systems XCS/ECGA and XCS/BOA is similar to that of XCS with informed crossover operators that is given all information about problem structure on input and exploits this knowledge using problem-specific crossover operators.

In this article, two learning classifier systems based on evolutionary techniques are described to classify remote sensing images. Usually, these images contain voluminous, complex, and sometimes erroneous and noisy data. The first approach implements ICU, an evolutionary rule discovery system, generating simple and robust rules. The second approach applies the real-valued accuracy-based classification system XCSR. The two algorithms are detailed and validated on hyperspectral data.

The accuracy-based XCS classifier system has been shown to solve typical data mining problems in a machine-learning competitive way. However, successful applications in multistep problems, modeled by a Markov decision process, were restricted to very small problems. Until now, the temporal difference learning technique in XCS was based on deterministic updates. However, since a prediction is actually generated by a set of rules in XCS and Learning Classifier Systems in general, gradient-based update methods are applicable. The extension of XCS to gradient-based update methods results in a classifier system that is more robust and more parameter independent, solving large and difficult maze problems reliably. Additionally, the extension to gradient methods highlights the relation of XCS to other function approximation methods in reinforcement learning.

Takes initial steps toward a theory of generalization and learning in the learning classifier system XCS. We start from Wilson's generalization hypothesis, which states that XCS has an intrinsic tendency to evolve accurate, maximally general classifiers. We analyze the different evolutionary pressures in XCS and derive a simple equation that supports the hypothesis theoretically. The equation is tested with a number of experiments that confirm the model of generalization pressure that we provide. Then, we focus on the conditions, termed "challenges," that must be satisfied for the existence of effective fitness or accuracy pressure in XCS. We derive two equations that suggest how to set the population size and the covering probability so as to ensure the development of fitness pressure. We argue that when the challenges are met, XCS is able to evolve problem solutions reliably. When the challenges are not met, a problem may provide intrinsic fitness guidance or the reward may be biased in such a way that the problem will still be solved. The equations and the influence of intrinsic fitness guidance and biased reward are tested on large Boolean multiplexer problems. The paper is a contribution to understanding how XCS functions and lays the foundation for research on XCS's learning complexity.

The recent detection of mirror neurons in monkeys suggests that brains encode parts of an observed action in a similar way they encode own actions. This paper models such a mirror system by means of adaptive resonance theory (ART) artificial neural networks coupling them in a manner similar to ARTMAP systems. Particularly, a bidirectional ARTMAP system (BiARTMAP) is created. The system associates executed actions with consequent action-effects. The associative structure gives the system mirror capabilities: on the one hand, perceived environmental changes cause an action association. On the other hand, activated action patterns cause the expectation of resulting environmental change. We also show that many other proposed cognitive processes relate to the BiARTMAP architecture. Future work includes the incorporation of situational dependencies, the combination of BiARTMAP with vector associative maps (VAMs), and the integration of BiARTMAP in a behavioral module enabling anticipatory behavior. Application wise, BiARTMAP can be applied as a general classifier and/or associative network.

The evolutionary learning mechanism in XCS strongly depends on its accuracy-based fitness approach. The approach is meant to result in an evolutionary drive from classifiers of low accuracy to those of high accuracy. Since, given inaccuracy, lower specificity often corresponds to lower accuracy, fitness pressure most often also results in a pressure towards higher specificity. Moreover, fitness pressure should cause the evolutionary process to be innovative in that it combines low-order building blocks of lower accurate classifiers, to higher-order building blocks with higher accuracy. This paper investigates how, when, and where accuracy-based fitness results in successful rule evolution in XCS. Along the way, a weakness in the current proportionate selection method in XCS is identified. Several problem bounds are derived that need to be obeyed to enable proper evolutionary pressure. Moreover, a fitness dilemma is identified that causes accuracy-based fitness to be misleading. Improvements are introduced to XCS to make fitness pressure more robust and overcome the fitness dilemma. Specifically, (1) tournament selection results in a much better fitness-bias exploitation, and (2) bilateral accuracy prevents the fitness dilemma. While the improvements stand for themselves, we believe they also contribute to the ultimate goal of an evolutionary learning system that is able to solve decomposable machine-learning problems quickly, accurately,and reliably. The paper also contributes to the further understanding of XCS in general and the fitness approach in XCS in particular.