Abstract While reinforcement learning (RL) algorithms have generated impressive strategies for a wide range of tasks, the performance improvements in continuous-domain, real-world problems do not follow the same trend. Poor exploration and quick convergence to locally optimal solutions play a dominant role. Advanced RL algorithms attempt to mitigate this issue by introducing exploration signals during the training procedure. This successful integration has paved the way to introduce signals from the intrinsic exploration branch. ACRE algorithm is a framework that concretely describes the conditions for such an integration, avoiding transforming the Markov decision process into time varying, and as a result, making the whole optimization scheme brittle and susceptible to instability. The key distinction of ACRE lies in the way of handling and storing both extrinsic and intrinsic rewards. ACRE is an off-policy, actor-critic style RL algorithm that separately approximates the forward novelty return. ACRE is shipped with a Gaussian mixture model to calculate the instantaneous novelty; however, different options could also be integrated. Using such an effective early exploration, ACRE results in substantial improvements over alternative RL methods, in a range of continuous control RL environments, such as learning from policy-misleading reward signals. Open-source implementation is available here: https://github.com/athakapo/ACRE .

In this article, the problem of real-time robot exploration and map building (active SLAM) is considered. A single stereo vision camera is exploited by a fully autonomous robot to navigate, localize itself, define its surroundings, and avoid any possible obstacle in the aim of maximizing the mapped region following the optimal route. A modified version of the so-called cognitive-based adaptive optimization algorithm is introduced for the robot to successfully complete its tasks in real time and avoid any local minima entrapment. The method’s effectiveness and performance were tested under various simulation environments as well as real unknown areas with the use of properly equipped robots.

Further deterioration of the already burdened traffic conditions is expected within the following years, especially in high population density urban regions. To cope with such problem, centralized and decentralized adaptive optimization techniques have already been proposed in literature; introducing inefficient performance though, due to the highly stochastic dynamics involved, scaling and/or model unavailability problems, as well as data transmission limitations. To confront such problems, L4GCAO, a novel, model-free, decentralized, adaptive optimization approach, has been developed for maximizing the system's overall performance, by calibrating the parameters of a given signal control strategy through decentralized self-learning elements (agents). This paper considers a realistic simulation scenario where the parameters of a signal control strategy applied at each network intersection are calibrated, to study the performance of L4GCAO. For comparison purposes, the thoroughly evaluated and verified centralized optimization counterpart approach of L4GCAO namely CAO - has also been adopted herein. The results of the study indicate that both CAO and L4GCAO present quite similar potential for improving the overall performance metric considered, with respect to a well-designed fixed time control strategy used as reference point.

Several disastrous incidents from earthquakes have been recorded in recent and past human life. Despite the improvements in structural stability and vibration resilience, heavy structures still suffer from construction cost problems which usually hinder their potential investment. Vibration active control techniques present a great potential for reducing the anti-seismic protection costs. So far, existing building vibration control strategies are unable to provide a reliable operation able to reject the evolving uncertain non-linear dynamics that grow as the amplitude of the exogenous ground disturbance increases. This paper applies a vibration active control optimization methodology in applications involving large structures. A simulation model of the structure is used to optimize, in an offline manner, the total structure displacement metric. Simulation experiments demonstrate that the adopted approach namely Automated Fine-Tuning Cognitive Adaptive Optimization (AFT-CAO) - can effectively deal with seismic dynamics both in low and large seismic cases. AFT-CAO was proven capable to provide efficient control decisions that well-established LQR cannot outperform. The structure model used for the simulation tests consists by three vertically interconnected masses, each connected to an external lateral spring with an adjustable applied restoring force.

Managing grid-connected charging stations for fleets of electric vehicles leads to an optimal control problem where user preferences must be met with minimum energy costs (e.g., by exploiting lower electricity prices through the day, renewable energy production, and stored energy of parked vehicles). Instead of state-of-the-art charging scheduling based on open-loop strategies that explicitly depend on initial operating conditions, this paper proposes an approximate dynamic programming feedback-based optimization method with continuous state space and action space, where the feedback action guarantees uniformity with respect to initial operating conditions, while price variations in the electricity and available solar energy are handled automatically in the optimization. The resulting control action is a multi-modal feedback, which is shown to handle a wide range of operating regimes, via a set of controllers whose action that can be activated or deactivated depending on availability of solar energy and pricing model. Extensive simulations via a charging test case demonstrate the effectiveness of the approach.

The well-known feedback ramp metering algorithm ALINEA can be applied for local ramp metering or used as a key component in a coordinated ramp metering system. ALINEA uses real-time occupancy measurements from the ramp-flow merging area that may be at most few hundred meters downstream of the metered on-ramp nose. In many practical cases, however, bottlenecks with smaller capacity than the merging area may exist further downstream for various reasons, which suggests using measurements from those further downstream bottlenecks rather than from the merging area. This paper addresses the local ramp metering problem in such a downstream bottleneck case. Theoretical analysis indicates that ALINEA may lead to a poorly damped closed-loop behavior in this case, but PI-ALINEA, which is a suitable proportional-integral (PI) extension of ALINEA, can lead to satisfactory control performance. The stability of the closed-loop ramp metering system with PI-ALINEA is rigorously proved by Lyapunov stability arguments. The root locus method is also employed to analyze the linearized closed-loop system performance of ALINEA and PI-ALINEA with and without a downstream bottleneck to provide insights on both controllers' performance. Simulation studies are conducted using a macroscopic traffic flow model to demonstrate that the ramp metering performance of ALINEA indeed deteriorates in the distant downstream bottleneck case, whereas a significant improvement is obtained using PI-ALINEA. Moreover, with its control parameters appropriately tuned, PI-ALINEA is found to be universally applicable to a range of distances between the on-ramp and downstream bottlenecks. This indicates that little fine-tuning would be necessary in field applications.

The authors introduce a multifunctional control research test bench and its data management and interface system. They implemented a building monitoring, control, and interface system (MCIS) used for evaluation of complex building energy systems, for optimization of such energy systems and for carrying out control research experiments. In this paper they explain the MCIS' architecture and functionality, they present its research potential and three different control research use cases for building energy systems and heating, ventilation, and air conditioning (HVAC) systems; model-assisted control parameter fine tuning, model-based predictive control, and demonstration of adaptive control algorithms. The authors outline prerequisites for implementation, demonstration, and evaluation of new developed control algorithms and strategies. These prerequisites are fulfilled via system expansions providing a flexible demonstration bench.

The SFly project is an EU-funded project, with the goal to create a swarm of autonomous vision controlled micro aerial vehicles. The mission in mind is that a swarm of MAV's autonomously maps out an unknown environment, computes optimal surveillance positions and places the MAV's there and then locates radio beacons in this environment. The scope of the work includes contributions on multiple different levels ranging from theoretical foundations to hardware design and embedded programming. One of the contributions is the development of a new MAV, a hexacopter, equipped with enough processing power for onboard computer vision. A major contribution is the development of monocular visual SLAM that runs in real-time onboard of the MAV. The visual SLAM results are fused with IMU measurements and are used to stabilize and control the MAV. This enables autonomous flight of the MAV, without the need of a data link to a ground station. Within this scope novel analytical solutions for fusing IMU and vision measurements have been derived. In addition to the realtime local SLAM, an offline dense mapping process has been developed. For this the MAV's are equipped with a payload of a stereo camera system. The dense environment map is used to compute optimal surveillance positions for a swarm of MAV's. For this an optimiziation technique based on cognitive adaptive optimization has been developed. Finally, the MAV's have been equipped with radio transceivers and a method has been developed to locate radio beacons in the observed environment.

This paper deals with the problem of deploying a team of flying robots to perform surveillance coverage missions over a terrain of arbitrary morphology. In such missions, a key factor for the successful completion is the knowledge of the terrain's morphology. In this paper, we introduce a two-step centralized procedure to align optimally a swarm of flying vehicles for the aforementioned task. Initially, a single robot constructs a map of the area of interest using a novel monocular-vision-based approach. A state-of-the-art visual-SLAM algorithm tracks the pose of the camera while, simultaneously, building an incremental map of the surrounding environment. The map generated is processed and serves as an input in an optimization procedure using the cognitive adaptive methodology initially introduced in [1], [2]. The output of this procedure is the optimal arrangement of the robot team, which maximizes the monitored area. The efficiency of our approach is demonstrated using real data collected from aerial robots in different outdoor areas.

In this paper, we study the problem of optimizing (fine-tuning) the design parameters of large-scale traffic control systems that are composed of distinct and mutually interacting modules. This problem usually requires a considerable amount of human effort and time to devote to the successful deployment and operation of traffic control systems due to the lack of an automated well-established systematic approach. We investigate the adaptive fine-tuning algorithm for determining the set of design parameters of two distinct mutually interacting modules of the traffic-responsive urban control (TUC) strategy, i.e., split and cycle, for the large-scale urban road network of the city of Chania, Greece. Simulation results are presented, demonstrating that the network performance in terms of the daily mean speed, which is attained by the proposed adaptive optimization methodology, is significantly better than the original TUC system in the case in which the aforementioned design parameters are manually fine-tuned to virtual perfection by the system operators.

The recently developed traffic signal control strategy known as traffic-responsive urban control (TUC) requires availability of a fixed signal plan that is sufficiently efficient under undersaturated traffic conditions. To drop this requirement, the well-known Webster procedure for fixed-signal control derivation at isolated junctions is appropriately employed for real-time operation based on measured flows. It is demonstrated via simulation experiments and field application that the following hold: 1) The developed real-time demand-based approach is a viable real-time signal control strategy for undersaturated traffic conditions. 2) It can indeed be used within TUC to drop the requirement for a prespecified fixed signal plan. 3) It may, under certain conditions, contribute to more efficient results, compared with the original TUC method.

In this paper, the problem of positioning a team of mobile robots for a surveillance task in a non-convex environment with obstacles is considered. The robots are equipped with global positioning capabilities (for instance they are equipped with GPS) and visual sensors able to monitor the surrounding environment. Furthermore, they are able to communicate one with each other. The goal is to maximize the area monitored by the team, by identifying the best configuration of the team members. Due to the non-convex nature of the problem, an analytical solution can not be obtained. The proposed method is based on a new cognitive-based, adaptive optimization algorithm (CAO). This method allows getting coordinated and scalable controls to accomplish the task, even when the obstacles are unknown and the team is heterogeneous, i.e. each robot is equipped with a different type of visual sensor. Extensive simulations are presented to show the efficiency of the proposed approach.

Learning mechanisms that operate in unknown environments should be able to efficiently deal with the problem of controlling unknown dynamical systems. Many approaches that deal with such a problem face the so-called exploitation-exploration dilemma where the controller has to sacrifice efficient performance for the sake of learning "better" control strategies than the ones already known: during the exploration period, poor or even unstable closed-loop system performance may be exhibited. In this paper, we show that, in the case where the control goal is to stabilize an unknown dynamical system by means of state feedback, exploitation and exploration can be concurrently performed without the need of sacrificing efficiency. This is made possible through an appropriate combination of recent results developed by the author in the areas of adaptive control and adaptive optimization and a new result on the convex construction of control Lyapunov functions for nonlinear systems. The resulting scheme guarantees arbitrarily good performance in the regions where the system is controllable. Theoretical analysis as well as simulation results on a particularly challenging control problem verify such a claim.

A major barrier preventing the wide employment of mobile networks of robots in tasks such as exploration, mapping, surveillance, and environmental monitoring is the lack of efficient and scalable multi-robot passive and active sensing (estimation) methodologies. The main reason for this is the absence of theoretical and practical tools that can provide computationally tractable methodologies which can deal efficiently with the highly nonlinear and uncertain nature of multi-robot dynamics when employed in the aforementioned tasks. In this paper, a new approach is proposed and analyzed for developing efficient and scalable methodologies for a general class of multi-robot passive and active sensing applications. The proposed approach employs an estimation scheme that switches among linear elements and, as a result, its computational requirements are about the same as those of a linear estimator. The parameters of the switching estimator are calculated off-line using a convex optimization algorithm which is based on optimization and approximation using Sum-of-Squares (SoS) polynomials. As shown by rigorous arguments, the estimation accuracy of the proposed scheme is equal to the optimal estimation accuracy plus a term that is inversely proportional to the number of estimator's switching elements (or, equivalently, to the memory storage capacity of the robots' equipment). The proposed approach can handle various types of constraints such as communication and computational constraints as well as obstacle avoidance and maximum speed constraints and can treat both problems of passive and active sensing in a unified manner. The efficiency of the approach is demonstrated on a 3D active target tracking application employing flying robots.

Despite the continuous advances in the fields of intelligent control and computing, the design and deployment of efficient large scale nonlinear control systems (LNCSs) requires a tedious fine-tuning of the LNCS parameters before and during the actual system operation. In the majority of LNCSs the fine-tuning process is performed by experienced personnel based on field observations via experimentation with different combinations of controller parameters, without the use of a systematic approach. The existing adaptive/neural/fuzzy control methodologies cannot be used towards the development of a systematic, automated fine-tuning procedure for general LNCS due to the strict assumptions they impose on the controlled system dynamics; on the other hand, adaptive optimization methodologies fail to guarantee an efficient and safe performance during the fine-tuning process, mainly due to the fact that these methodologies involve the use of random perturbations. In this paper, we introduce and analyze, both by means of mathematical arguments and simulation experiments, a new learning/adaptive algorithm that can provide with convergent, an efficient and safe fine-tuning of general LNCS. The proposed algorithm consists of a combination of two different algorithms proposed by Kosmatopoulos (2007 and 2008) and the incremental-extreme learning machine neural networks (I-ELM-NNs). Among the nice properties of the proposed algorithm is that it significantly outperforms the algorithms proposed by Kosmatopoulos as well as other existing adaptive optimization algorithms. Moreover, contrary to the algorithms proposed by Kosmatopoulos , the proposed algorithm can operate efficiently in the case where the exogenous system inputs (e.g., disturbances, commands, demand, etc.) are unbounded signals.

In a recent series of articles with largely identical contents and results, some claims are raised about the pertinence and performance of the well-known and widely field-applied local ramp metering algorithm ALINEA and of some extended versions thereof. The expressed claims are based on simulation results with a self-made microscopic simulator. This paper shows that the produced simulation results and derived conclusions are based on an insufficient understanding of the feedback character of the ALINEA algorithm, which led to an inappropriate application of the method. More specifically, the mainstream measurement that feeds ALINEA was misplaced so that any occurring congestion could not be monitored; this renders ALINEA blind to the traffic conditions under control and negates the very notion of feedback.

In this paper, the problem of stabilization of unknown nonlinear dynamical systems is considered. An adaptive feedback law is constructed that is based on the switching adaptive strategy proposed by the author and uses linear-in-the-weights neural networks accompanied with appropriate robust adaptive laws in order to estimate the time-derivative of the control Lyapunov function (CLF) of the system. The closed-loop system is shown to be stable; moreover, the state vector of the controlled system converges to a ball centered at the origin and having a radius that can be made arbitrarily small by increasing the high gain K and the number of neural network regressor terms. No growth conditions on the nonlinearities of the system are imposed with the exception that such nonlinearities are sufficiently smooth. Finally, we mention that neither the system dynamics or the CLF of the system need to be known in order to apply the proposed methodology.

It is known that the problem of learning the shape parameters of unknown surfaces that are in contact with a robot end-effector can be formulated as a nonlinear parameter estimation problem and an extended Kalman filter can be applied in order to estimate the surface shape parameters. In this paper, we show that the problem of learning the shape parameters of unknown contact surfaces can be formulated as a linear parameter estimation problem and thus globally convergent learning laws can be applied. Moreover, we show that by using appropriate neural network approximators, the unknown surfaces can be learned even if there are no force measurements, i.e., the robot is not provided with any force or tactile sensors.

A mathematical analysis of a class of learning vector quantization (LVQ) algorithms is presented. Using an appropriate time-coordinate transformation, we show that the LVQ algorithms under consideration can be transformed into linear time-varying stochastic difference equations. Using this fact, we apply stochastic Lyapunov stability arguments, and we prove that the LVQ algorithms under consideration do indeed converge, provided that some appropriate conditions hold.

Several continuous-time and discrete-time recurrent neural network models have been developed and applied to various engineering problems. One of the difficulties encountered in the application of recurrent networks is the derivation of efficient learning algorithms that also guarantee the stability of the overall system. This paper studies the approximation and learning properties of one class of recurrent networks, known as high-order neural networks; and applies these architectures to the identification of dynamical systems. In recurrent high-order neural networks, the dynamic components are distributed throughout the network in the form of dynamic neurons. It is shown that if enough high-order connections are allowed then this network is capable of approximating arbitrary dynamical systems. Identification schemes based on high-order network architectures are designed and analyzed.

The capabilities of recurrent high order neural networks (RHONNs), whose synapses are adjusted according to the learning law proposed in Kosmatopoulos and Christodoulou (1992), and Koostmatopoulos, Christodoulou, and Ioannou (1993) are examined in 1) spatiotemporal pattern learning, recognition, and reproduction and 2) stochastic dynamical system identification problems. The mathematical model describing the stochastic disturbances that affect the spatiotemporal patterns or the system dynamics is quite general, and includes both additive and multiplicative stochastic disturbances. Under an extensive mathematical analysis, the authors show that, for any selection of the neural network's high order terms, the prediction error converges to zero exponentially fast. Extensions are also made to the case where the energy coordinate equivalent (ECE) RHONN's are used.>