We present the agent we developed for the 2024 DareFightingICE competition that integrates advanced audio encoders, deep reinforcement learning, and hard-coded rules to improve the agent’s behavior for one specific, rarely used attack. The integration of hard-coded rules with our 1D-CNN encoder let our agent outperform the 2023 DareFightingICE AI winners (Pythunder and CAS). Our agent achieved second place in the competition, being the only top-ranking submission using machine learning, while the other entries implemented scripted strategies. Despite its strengths, our entry can be defeated by analyzing its favorite moves. We plan to enhance the audio encoders, refine long-term planning strategies, and optimize training methods to narrow the gap between our entry and the other top entries.

Radiotherapy (RT) is a key component in the treatment of various cancers, including Acute Lymphocytic Leukemia (ALL) and Acute Myelogenous Leukemia (AML). Precise delineation of organs at risk (OARs) and target areas is essential for effective treatment planning. Intensity Modulated Radiotherapy (IMRT) techniques, such as Total Marrow Irradiation (TMI) and Total Marrow and Lymph node Irradiation (TMLI), provide more precise radiation delivery compared to Total Body Irradiation (TBI). However, these techniques require time-consuming manual segmentation of structures in Computerized Tomography (CT) scans by the Radiation Oncologist (RO). In this paper, we present a deep learning-based auto-contouring method for segmenting Planning Target Volume (PTV) for TMLI treatment using the U-Net architecture. We trained and compared two segmentation models with two different loss functions on a dataset of 100 patients treated with TMLI at the Humanitas Research Hospital between 2011 and 2021. Despite challenges in lymph node areas, the best model achieved an average Dice score of 0.816 for PTV segmentation. Our findings are a preliminary but significant step towards developing a segmentation model that has the potential to save radiation oncologists a considerable amount of time. This could allow for the treatment of more patients, resulting in improved clinical practice efficiency and more reproducible contours.

Organ at Risk (OAR) segmentation from CT scans is a key component of the radiotherapy treatment workflow. In recent years, deep learning techniques have shown remarkable potential in automating this process. In this paper, we investigate the performance of Generative Adversarial Networks (GANs) compared to supervised learning approaches for segmenting OARs from CT images. We propose three GAN-based models with identical generator architectures but different discriminator networks. These models are compared with well-established CNN models, such as SE-ResUnet and DeepLabV3, using the StructSeg dataset, which consists of 50 annotated CT scans containing contours of six OARs. Our work aims to provide insight into the advantages and disadvantages of adversarial training in the context of OAR segmentation. The results are very promising and show that the proposed GAN-based approaches are similar or superior to their CNN-based counterparts, particularly when segmenting more challenging target organs.

In the medical images field, semantic segmentation is one of the most important, yet difficult and time-consuming tasks to be performed by physicians. Thanks to the recent advancement in the Deep Learning models regarding Computer Vision, the promise to automate this kind of task is getting more and more realistic. However, many problems are still to be solved, like the scarce availability of data and the difficulty to extend the efficiency of highly specialised models to general scenarios. Organs at risk segmentation for radiotherapy treatment planning falls in this category, as the limited data available negatively affects the possibility to develop general-purpose models; in this work, we focus on the possibility to solve this problem by presenting three types of ensembles of single-organ models able to produce multi-organ masks exploiting the different specialisations of their components. The results obtained are promising and prove that this is a possible solution to finding efficient multi-organ segmentation methods.

Large language models (LLMs) have taken the scientific world by storm, changing the landscape of natural language processing and human-computer interaction. These powerful tools can answer complex questions and, surprisingly, perform challenging creative tasks (e.g., generate code and applications to solve problems, write stories, pieces of music, etc.). In this paper, we present a collaborative game design framework that combines interactive evolution and large language models to simulate the typical human design process. We use the former to exploit users' feedback for selecting the most promising ideas and large language models for a very complex creative task---the recombination and variation of ideas. In our framework, the process starts with a brief and a set of candidate designs, either generated using a language model or proposed by the users. Next, users collaborate on the design process by providing feedback to an interactive genetic algorithm that selects, recombines, and mutates the most promising designs. We evaluated our framework on three game design tasks with human designers who collaborated remotely.

We present an educational application of virtual reality that we created to help students gain an in-depth understanding of the internal structure of crystals and related key concepts. Teachers can use it to give lectures to small groups (10-15) of students in a shared virtual environment, both remotely (with teacher and students in different locations) and locally (while sharing the same physical space). Lectures can be recorded, stored in an online repository, and shared with students who can either review a recorded lecture in the same virtual environment or can use the application for self-studying by exploring a large collection of available crystal structures. We validated our application with human subjects receiving positive feedback. Supplementary Information: The online version contains supplementary material available at 10.1007/s11042-022-13410-0https://doi.org/10.1007/s11042-022-13410-0.

In the last decade, Deep Learning has revolutionized Computer Vision thanks to Convolutional Neural Networks (CNN), that achieved state-of-the-art results in many tasks. In the medical field, imaging techniques, like MRI and CT, are widely used to acquire 3D images of regions that need to be analyzed to identify targets or regions of interest (ROIs). In particular, semantic segmentation is a common image processing task involved in several clinical procedures. When using Deep Learning to solve this task it is possible to either apply a 2D CNN to each slice of the acquired 3D image or apply a 3D CNN to the entire volume acquired. Despite both this approaches have been investigated in the literature, there is neither yet a clear understanding of which one is better (if this is the case) nor a fair comparison of their performances on the same datasets. In this work we aim at making a first step toward to providing an empirical guidance on choosing between 2D and 3D CNNs for medical imaging segmentation. To this purpose we compared a 2D CNN and a 3D CNN based on deep residual U-Net (ResUnet) architecture on different datasets. Our results suggest that the potential benefits of using a 3D CNN are difficult to exploit due to the very limited amount of data that is typically available in medical datasets.

Chest X-ray (CXR) is perhaps the most frequently-performed radiological investigation globally. In this work, we present and study several machine learning approaches to develop automated CXR diagnostic models. In particular, we trained several Convolutional Neural Networks (CNN) on the CheXpert dataset, a large collection of more than 200k CXR labeled images. Then, we used the trained CNNs to compute embeddings of the CXR images, in order to train two sets of tree-based classifiers from them. Finally, wed escribed and compared three ensembling strategies to combine together the classifiers trained. Rather than expecting some performance-wise benefits, o ur goal i n this work iss howing that t he above two methodologies, i.e., the extraction of image embeddings and models ensembling, can be effective and viable to solve tasks that require medical imaging understanding. Our results in that perspective are encouraging and worthy of further investigation.

Ten years ago we started a course on video game design and development. It was the first course on video games in our university and possibly in our country. We were immediately daunted by two main decisions: (i) the selection of the projects to be developed during the course and (ii) the evaluation of students’ projects. We wanted to give students the maximum freedom and no limit to their creativity. We wanted them to focus on the creation of a game that people would love to play without worrying about some score objectives to maximize and without caring about their instructors’ game design preferences. Accordingly, we decided to ask students to do all the job, starting from the submission and the selection of the game concepts to develop during the course, up to the final evaluation of the projects, the evaluation of their teammates, and thus basically the grading. In this paper, we discuss our experience over the last ten years with our course organization and grading model that, we believe, gives students complete freedom to express themselves and leaves them most of, if not all, the agency.

Deep Learning is a promising approach to either automate or simplify several tasks in the healthcare domain. In this work, we introduce SegAN-CAT, an end-to-end approach to brain tumor segmentation in Magnetic Resonance Images (MRI), based on Adversarial Networks. In particular, we extend SegAN, successfully applied to the same task in a previous work, in two respects: (i) we used a different model input and (ii) we employed a modified loss function to train the model. We tested our approach on two large datasets, made available by the Brain Tumor Image Segmentation Benchmark (BraTS). First, we trained and tested some segmentation models assuming the availability of all the major MRI contrast modalities, i.e., T1-weighted, T1 weighted contrast enhanced, T2-weighted, and T2-FLAIR. However, as these four modalities are not always all available for each patient, we also trained and tested four segmentation models that take as input MRIs acquired with a single contrast modality. Finally, we proposed to apply transfer learning across different contrast modalities to improve the performance of these single-modality models. Our results are promising and show that not only SegAN-CAT is able to outperform SegAN when all the four modalities are available, but also that transfer learning can actually lead to better performances when only a single modality is available.

Level design in first person shooters is a critical and complex process with many facets. Among them, the placement of the spawn points, i.e., the positions of the level where players starts the game after being killed, has a huge impact on the game experience. In this work, we propose a novel approach to this problem that combines a set of design rules with a graph-based analysis of the level. As a results, we design a set of heuristics that, based on the topological features of the graphs extracted from the levels, can be used for the placement of spawn point. Finally, we test our approach with a small user study on three different levels, comparing our heuristic placement with an uniform strategy. Although preliminary, our results are promising and suggest that the level design principles are effectively captured by our heuristics.

In this work, following the same approach successfully applied to evolve Super Mario levels, we applied the CMA-ES to search the latent space of a GAN previously trained to generate DOOM levels. Combining a search algorithm with a model trained in a supervised setting, allows to take advantage from both these paradigms. From one hand, the GAN is able to generate contents exploiting the design patterns learned from all the examples it was trained from. On the other hand, the CMA-ES can effectively search this design space for specific contents that meet some given design objectives. In particular, we tested our approach evolving three very different type of levels: an arena level (i.e., few large areas), a labyrinth level (i.e., many corridors and small areas), and a complex level (i.e., a balanced mix of large and small areas). Our results show that the latent space of a GAN can be effectively searched by the CMA-ES to find DOOM levels that fit accurately the objectives but, at the same time, are also novel.

Map design is a major challenge in the development of a successful multiplayer first-person shooter. In fact, it has a large impact on the game dynamics and deeply affects the player experience. In this paper, we present a search-based procedural content generation approach to the map design problem in Cube 2: Sauerbraten, an open-source first-person shooter. Extending previous works introduced in the literature, we propose several design objectives to evaluate the maps as well as a novel methodology to compute them. To test our approach, we designed two different design problems that require to deal with conflicting objectives (i.e., balancing, pacing, and achievement of long kill streaks) and with players using different playing styles. The results are promising as they show that our approach, exploiting multiobjective evolution, is able to explore effectively the map design space and to provide maps that feature interesting tradeoffs between the conflicting objectives.

We present a framework for the procedural generation of tracks for a high-end car racing game (TORCS) using interactive evolution. The framework maintains multiple populations and allow users to work both on their own population (in single-user mode) or to collaborate with other users on a shared population. Our architecture comprises a web frontend and an evolutionary backend. The former manages the interaction with users (e.g., logs registered and anonymous users, collects evaluations, provides access to all the evolved populations) and maintains the database server that stores all the present/past populations. The latter runs all the tasks related to evolution (selection, recombination and mutation) and all the tasks related to the target racing game (e.g., the track generation). We performed two sets of experiments involving five human subjects to evolve racing tracks alone (in a single-user mode) or cooperatively. Our preliminary results on five human subjects show that, in all the experiments, there is an increase of users' satisfaction as the evolution proceeds. Users stated that they perceived improvements in the quality of the individuals between subsequent populations and that, at the end, the process produced interesting tracks.

In this paper, we investigate the application of evolutionary computation to the automatic generation of tracks for high-end racing games. The idea underlying our approach is that diversity is a major source of challenge/interest for racing tracks and, eventually, might play a key role in contributing to the player's fun. In particular, we focus on the diversity of a track in terms of its shape (i.e., the number and the assortment of turns and straights it contains), and in terms of driving experience it provides (i.e., the range of speeds achievable while driving on the track). We define two fitness functions that capture our idea of diversity as the entropy of the track's curvature and speed profiles. We apply both a single-objective and a multiobjective real-coded genetic algorithm (GA) to evolve tracks involving both a wide variety of turns and straights and also a large range of driving speeds. The results we report show that both single-objective and multiobjective approaches can successfully evolve tracks with a high degree of diversity both in terms of shape and achievable speeds.

The TORCS Endurance World Championship is an international competition in which programmers develop and tune their drivers to race against each other using TORCS, a state-of-the-art car racing simulator. In this work, we applied evolutionary computation to develop a driver for the 2009 edition of this competition. In particular, we focused on the optimization of the car setup of an existing driver (the winner of the 2008 edition) and applied cooperative coevolution to evolve the best car setup for the qualifying rounds of each leg of the championship. After 10 legs involving 12 teams, our driver was able to reach the 4th position in the final standings. We believe that this is a very promising result especially if we consider that we only focused on the car setup and the other teams participated also to most of the previous four editions (gaining much domain knowledge). Overall, our results show that cooperative coevolution can be very effective in this complex optimization task producing setups that can be competitive with respect to the ones tuned by human experts. Therefore, our results also suggest that simple evolutionary computation might represent a helpful tool to human designers for improving the performance of already heavily tuned drivers.

In modern racing games programming non-player characters with believable and sophisticated behaviors is getting increasingly challenging. Recently, several works in the literature suggested that computational intelligence can provide effective solutions to support the development process of NPCs. In this paper, we applied the Behavior Analysis and Training (BAT) methodology to define a behavior-based architecture for the NPCs in The Open Racing Car Simulator (TORCS), a well-known open source racing game. Then, we focused on two major overtaking behaviors: (i) the overtaking of a fast opponent either on a straight stretch or on a large bend; (ii) the overtaking on a tight bend, which typically requires a rather advanced braking policy. We applied simple reinforcement learning, namely Q-learning, to learn both these overtaking behaviors. We tested our approach in several overtaking situations and compared the learned behaviors against one of the best NPC provided with TORCS. Our results suggest that, exploiting the proposed behavior-based architecture, Q-learning can effectively learn sophisticated behaviors and outperform programmed NPCs. In addition, we also show that the same approach can be successfully applied to adapt a previously learned behavior to a dynamically changing game situation.

In this paper, we applied online neuroevolution to evolve nonplayer characters for The Open Racing Car Simulator (TORCS). While previous approaches allowed online learning with performance improvements during each generation, our approach enables a finer grained online learning with performance improvements within each lap. We tested our approach on three tracks using two methods of online neuroevolution (NEAT and rtNEAT) combined with four evaluation strategies ( -greedy, -greedy-improved, softmax, and interval-based) taken from the literature. We compared the eight resulting configurations on several driving tasks involving the learning of a driving behavior for a specific track, its adaptation to a new track, and the generalization capability to unknown tracks. The results we present show that, notwithstanding the several challenges that online learning poses, our approach 1) can successfully evolve drivers from scratch, 2) can also be used to transfer evolved knowledge to other tracks, and 3) can generalize effectively producing controllers that can drive on difficult unseen tracks. Our results also suggest that the approach performs better when coupled with online NEAT and also indicate that -greedy-improved and softmax are generally better than the other evaluation strategies. A comparison with typical offline neuroevolution suggests that online neuroevolution can be competitive and even outperform traditional offline approaches on more difficult tracks while providing all the interesting features of online learning. Overall, we believe that this study may represent an initial step toward the application of online neuroevolution in games.

In this paper, we overview the 2009 Simulated Car Racing Championship-an event comprising three competitions held in association with the 2009 IEEE Congress on Evolutionary Computation (CEC), the 2009 ACM Genetic and Evolutionary Computation Conference (GECCO), and the 2009 IEEE Symposium on Computational Intelligence and Games (CIG). First, we describe the competition regulations and the software framework. Then, the five best teams describe the methods of computational intelligence they used to develop their drivers and the lessons they learned from the participation in the championship. The organizers provide short summaries of the other competitors. Finally, we summarize the championship results, followed by a discussion about what the organizers learned about 1) the development of high-performing car racing controllers and 2) the organization of scientific competitions.

The application of on-line learning techniques to modern computer games is a promising research direction. In fact, they can be used to improve the game experience and to achieve a true adaptive game AI. So far, several works proved that neuroevolution techniques can be successfully applied to modern computer games but they are usually restricted to offline learning scenarios. In on-line learning problems the main challenge is to find a good trade-off between the exploration, i.e., the search for better solutions, and the exploitation of the best solution discovered so far. In this paper we propose an on-line neuroevolution approach to evolve non-player characters in The Open Car Racing Simulator (TORCS), a state-of-the-art open source car racing simulator. We tested our approach on two on-line learning problems: (i) on-line evolution of a fast controller from scratch and (ii) optimization of an existing controller for a new track. Our results show that on-line neuroevolution can effectively improve the performance achieved during the learning process.

Modern computer games are at the same time an attractive application domain and an interesting testbed for the evolutionary computation techniques. In this paper we apply NeuroEvolution of Augmenting Topologies (NEAT), a well known neuroevolution approach, to evolve competitive non-player characters for a racing game. In particular, we focused on The Open Car Racing Simulator (TORCS), an open source car racing simulator, already used as a platform for several scientific competitions dedicated to games. We suggest that a competitive controller should have two basic skills: it should be able to drive fast and reliably on a wide range of tracks and it should be able to effectively overtake the opponents avoiding the collisions. In this paper we apply NEAT to evolve separately these skills and then we combined them together in a single controller. Our results show that the resulting controller outperforms the best available controllers on a challenging racing task. In addition, the experimental analysis also confirms that both the skills are necessary to develop a competitive controller.

In XCS with computed prediction, namely XCSF, the classifier prediction parameter is replaced by a parametrized prediction function. So far, the works on the computed prediction in XCSF has been limited to evolve a single type of prediction function at once. Recently, several works studied and extended the computed prediction in XCSF. However, it is still not clear how the most adequate prediction function should be chosen for a given problem. In this paper we introduce XCSF with voting predictors that extends XCSF to let it select best prediction function to use in each problem subspace. We compared XCSFV to XCSF on several problems. Our results suggest that XCSFV performs as well as XCSF with the best prediction function in all the tested problems. In addition, XCSFV finds the most accurate prediction function in each problem subspace.

Computed prediction was originally devised to tackle problems defined over real-valued domains. Recent experiments on Boolean functions showed that the concept of computed prediction extends beyond real values and it can also be applied to solve more typical classifier system benchmarks such as Boolean multiplexer and parity functions. So far however, no result has been presented for other well known classifier system benchmarks, i.e., binary multistep problems such as the woods environments. In this paper, we apply XCS with computed prediction to woods environments and show that computed prediction can also tackle this class of problems. Our results demonstrate that (i) XCS with computed prediction converges to optimality faster than XCS, (ii) it solves problems that may be too difficult for XCS and (iii) it evolves solutions that are more compact than those evolved by XCS.

The high-level synthesis process involves three interdependent and NP-complete optimization problems: (i) the operation scheduling, (ii) the resource allocation, and (iii) the controller synthesis. Evolutionary algorithms have been effectively applied to high level synthesis in presence conflicting design objectives for finding good tradeoffs in the design space. However, so far the design space exploration has been performed using single-objective evolutionary algorithms with an ad hoc fitness function to achieve the desired tradeoff between the objectives. Recently we proposed a framework based on multi-objective genetic algorithms to perform a fully automated design space exploration. In this paper we focus on the choice of the solution representations that can be used to perform the design space exploration with multi-objective genetic algorithms. In particular we consider two specific representations and compare them on a set of benchmark problems. Our results suggest that they have different biases on the search space that make them more effective in different problems and design subspaces. Accordingly, we present a preliminary investigation on a new representation that exploits the advantages of both of them.

We investigate rule matching in learning classifier systems for problems involving binary and real inputs. We consider three rule encodings: the widely used character-based encoding, a specificity-based encoding, and a binary encoding used in Alecsys. We compare the performance of the three algorithms both on matching alone and on typical test problems. The results on matching alone show that the population generality influences the performance of the matching algorithms based on string representations in different ways. Character-based encoding becomes slower and slower as generality increases, specificity-based encoding becomes faster and faster as generality increases. The results on typical test problems show that the specificity-based representation can halve the time required for matching but also that binary encoding is about ten times faster on the most difficult problems. Moreover, we extend specificity-based encoding to real-inputs and propose an algorithm that can halve the time require for matching real inputs using an interval-based representation.

XCS with computed action, briefly XCSCA, is a recent extension of XCS to tackle problems involving a large number of discrete actions. In XCSCA the classifier action is computed with a parameterized function learned in a supervised fashion. In this paper, we introduce XCSCAsvm that extends XCSCA using support vector machines to compute classifier action. We compared XCSCAsvm and XCSCA on the learning of several binary functions. The experimental results show that XCSCAsvm reaches the optimal performance faster than XCSCA.

The learning in a niche based learning classifier system depends both on the complexity of the problem space and on the number of available actions. In this paper, we introduce a version of XCS with computed actions, briefly XCSCA, that can be applied to problems involving a large number of actions. We report experimental results showing that XCSCA can evolve accurate and compact representations of binary functions which would be challenging for typical learning classifier system models.

In this paper we introduce XCSF with support vector prediction: the problem of learning the prediction function is solved as a support vector regression problem and each classifier exploits a Support Vector Machine to compute the prediction. In XCSF with support vector prediction, XCSFsvm, the genetic algorithm adapts classifier conditions, classifier actions, and the SVM kernel parameters. We compare XCSF with support vector prediction to XCSF with linear prediction on the approximation of four test functions. Our results suggest that XCSF with support vector prediction compared to XCSF with linear prediction (i) is able to evolve accurate approximations of more difficult functions, (ii) has better generalization capabilities and (iii) learns faster.

We analyze generalization in XCSF and introduce three improvements. We begin by showing that the types of generalizations evolved by XCSF can be influenced by the input range. To explain these results we present a theoretical analysis of the convergence of classifier weights in XCSF which highlights a broader issue. In XCSF, because of the mathematical properties of the Widrow-Hoff update, the convergence of classifier weights in a given subspace can be slow when the spread of the eigenvalues of the autocorrelation matrix associated with each classifier is large. As a major consequence, the system's accuracy pressure may act before classifier weights are adequately updated, so that XCSF may evolve piecewise constant approximations, instead of the intended, and more efficient, piecewise linear ones. We propose three different ways to update classifier weights in XCSF so as to increase the generalization capabilities of XCSF: one based on a condition-based normalization of the inputs, one based on linear least squares, and one based on the recursive version of linear least squares. Through a series of experiments we show that while all three approaches significantly improve XCSF, least squares approaches appear to be best performing and most robust. Finally we show how XCSF can be extended to include polynomial approximations.

We extend XCSF with neural prediction and replace the linear prediction function used in XCSF with a feedforward multilayer neural network. Each classifier exploits a neural network to approximate the payoff surface associated to the target problem while the genetic algorithm adapts both classifier conditions, classifier actions, and the network structure. We compare XCSF with neural prediction to XCSF with linear prediction. Our results show that XCSF with neural prediction, XCSFNN, can outperform XCSF.

This paper investigates reinforcement learning (RL) in XCS. First, it formally shows that XCS implements a method of generalized RL based on linear approximators, in which the usual input mapping function translates the state-action space into a niche relative fitness space. Then, it shows that, although XCS has always been related to standard RL, XCS is actually a method of averaging RL. More precisely, XCS with gradient descent can be actually derived from the typical update of averaging RL. It is noted that the use of averaging RL in XCS introduces an intrinsic preference toward classifiers with a smaller fitness in the niche. It is argued that, because of the accuracy pressure in XCS, this results in an additional preference toward specificity. A very simple experiment is presented to support this hypothesis. The same approach is applied to XCS with computed prediction (XCSF) and similar conclusions are drawn.

This paper introduces XCSF extended with tile coding prediction: each classifier implements a tile coding approximator; the genetic algorithm is used to adapt both classifier conditions (i.e., to partition the problem) and the parameters of each approximator; thus XCSF evolves an ensemble of tile coding approximators instead of the typical monolithic approximator used in reinforcement learning. The paper reports a comparison between (i) XCSF with tile coding prediction and (ii) plain tile coding. The results show that XCSF with tile coding always reaches optimal performance, it usually learns as fast as the best parametrized tile coding, and it can be faster than the typical tile coding setting. In addition, the analysis of the evolved tile coding ensembles shows that XCSF actually adapts local approximators following what is currently considered the best strategy to adapt the tile coding parameters in a given problem.

We study how different prediction update algorithms influence the performance of XCSF. We consider three classical parameter estimation algorithms (NLMS, RLS, and Kalman filter) and four gain adaptation algorithms (K1, K2, IDBD, and IDD). The latter have been shown to perform comparably to the best algorithms (RLS and Kalman), but they have a lower complexity. We apply these algorithms to update classifier prediction in XCSF and compare the performances of the seven versions of XCSF on a set of real functions. Our results show that the best known algorithms still perform best: XCSF with RLS and XCSF with Kalman perform significantly better than the others. In contrast, when added to XCSF, gain adaptation algorithms perform comparably to NLMS, the simplest estimation algorithm, the same used in the original XCSF. Nevertheless, algorithms that perform similarly generalize differently. For instance: XCSF with Kalman filter evolves more compact solutions than XCSF with RLS and gain adaptation algorithms allow better generalization than NLMS.

Computed prediction represents a major shift in learning classifier system research. XCS with computed prediction, based on linear approximates, has been applied so far to function approximation, to single step problems involving continuous payoff functions, and to multi step problems. In this paper we take this new approach in a different direction and apply it to the learning of Boolean functions - a domain characterized by highly discontinuous 0/1000 payoff functions. We also extend it to the case of computed prediction based on functions, borrowed from neural networks, that may be more suitable for 0/1000 payoff problems: the perceptron and the sigmoid. The results we present show that XCSF with linear prediction performs optimally in typical Boolean domains and it allows more compact solutions evolving classifiers that are more general compared with XCS. In addition, perceptron based and sigmoid based prediction can converge slightly faster than linear prediction while producing slightly more compact solutions

We apply XCS with computed prediction (XCSF) to tackle multistep reinforcement learning problems involving continuous inputs. In essence we use XCSF as a method of generalized reinforcement learning. We show that in domains involving continuous inputs and delayed rewards XCSF can evolve compact populations of accurate maximally general classifiers which represent the optimal solution to the target problem. We compare the performance of XCSF with that of tabular Q-learning adapted to the continuous domains considered here. The results we present show that XCSF can converge much faster than tabular techniques while producing more compact solutions. Our results also suggest that when exploration is less effective in some areas of the problem space, XCSF can exploit effective generalizations to extend the evolved knowledge beyond the frequently explored areas. In contrast, in the same situations, the convergence speed of tabular Q-learning worsens.

XCSF is the extension of XCS in which classifier prediction is computed as a linear combination of classifier inputs and a weight vector associated to each classifier. XCSF can exploit classifiers' computable prediction to evolve accurate piecewise linear approximations of functions. In this paper, we take XCSF one step further and show how XCSF can be easily extended to allow polynomial approximations. We test the extended version of XCSF on various approximation problems and show that quadratic/cubic approximations can be used to significantly improve XCSF's generalization capabilities.

XCSF extends the typical concept of learning classifier systems through the introduction of computed classifier prediction. Initial results show that XCSF's computed prediction can be used to evolve accurate piecewise linear approximations of simple functions. In this paper, we take XCSF one step further and apply it to typical reinforcement learning problems involving delayed rewards. In essence, we use XCSF as a method of generalized (linear) reinforcement learning to evolve piecewise linear approximations of the payoff surfaces of typical multistep problems. Our results show that XCSF can easily evolve optimal and near optimal solutions for problems introduced in the literature to test linear reinforcement learning methods.