Motivated by applications in large-scale and multi-agent reinforcement learning, we study the non-asymptotic performance of stochastic approximation (SA) schemes with delayed updates under Markovian sampling. While the effect of delays has been extensively studied for optimization, the manner in which they interact with the underlying Markov process to shape the finite-time performance of SA remains poorly understood. In this context, our first main contribution is to show that under time-varying bounded delays, the delayed SA update rule guarantees exponentially fast convergence of the \emph{last iterate} to a ball around the SA operator’s fixed point. Notably, our bound is \emph{tight} in its dependence on both the maximum delay $\tau_{max}$, and the mixing time $\tau_{mix}$. To achieve this tight bound, we develop a novel inductive proof technique that, unlike various existing delayed-optimization analyses, relies on establishing uniform boundedness of the iterates. As such, our proof may be of independent interest. Next, to mitigate the impact of the maximum delay on the convergence rate, we provide the first finite-time analysis of a delay-adaptive SA scheme under Markovian sampling. In particular, we show that the exponent of convergence of this scheme gets scaled down by $\tau_{avg}$, as opposed to $\tau_{max}$ for the vanilla delayed SA rule; here, $\tau_{avg}$ denotes the average delay across all iterations. Moreover, the adaptive scheme requires no prior knowledge of the delay sequence for step-size tuning. Our theoretical findings shed light on the finite-time effects of delays for a broad class of algorithms, including TD learning, Q-learning, and stochastic gradient descent under Markovian sampling.

Edge networks call for communication efficient (low overhead) and robust distributed optimization (DO) algorithms. These are, in fact, desirable qualities for DO frameworks, such as federated edge learning techniques, in the presence of data and system heterogeneity, and in scenarios where inter-node communication is the main bottleneck. Although computationally demanding, Newton-type (NT) methods have been recently advocated as enablers of robust convergence rates in challenging DO problems where edge devices have sufficient computational power. Along these lines, in this work$w$e propose Q-SHED, an original NT algorithm for DO featuring a novel bit-allocation scheme based on incremental Hessian eigenvectors quantization. The proposed technique is integrated with the recent SHED algorithm, from which it inherits appealing features like the small number of required Hessian computations, while being bandwidth-versatile at a bit-resolution level. Our empirical evaluation against competing approaches shows that Q-SHED can reduce by up to 60% the number of communication rounds required for convergence.

BACKGROUND: The ability to rapidly adapt to adverse environmental conditions represents the key of success of many pathogens and, in particular, of Mycobacterium tuberculosis. Upon exposition to heat shock, antibiotics or other sources of stress, appropriate responses in terms of genes transcription and proteins activity are activated leading part of a genetically identical bacterial population to express a different phenotype, namely to develop persistence. When the stress response network is mathematically described by an ordinary differential equations model, development of persistence in the bacterial population is associated with bistability of the model, since different emerging phenotypes are represented by different stable steady states. RESULTS: In this work, we develop a mathematical model of SigE stress response network that incorporates interactions not considered in mathematical models currently available in the literature. We provide, through involved analytical computations, accurate approximations of the system's nullclines, and exploit the obtained expressions to determine, in a reliable though computationally efficient way, the number of equilibrium points of the system. CONCLUSIONS: Theoretical analysis and perturbation experiments point out the crucial role played by the degradation pathway involving RseA, the anti-sigma factor of SigE, for coexistence of two stable equilibria and the emergence of bistability. Our results also indicate that a fine control on RseA concentration is a necessary requirement in order for the system to exhibit bistability.

In this article, we investigate an innovative solution, to implement high sampling frequency industrial control by means of networked embedded systems connected via WiFi. The basic idea relies on a co-design approach for the control application, which is then able to adapt its sampling period, as well as to tune the Wi-Fi parameters, according to the feedback coming from the network. To this end, we implemented a cross-layer architecture acting at both application and data-link layers, which features a robust frame-delay state estimator, a time-efficient communication policy, and a specific tuning of the critical protocol parameters. Suitable hardware-in-the-loop experiments have been carried out exploiting two different embedded systems available off-the-shelf. The preliminary results, obtained from an extensive experimental campaign, are encouraging since they show that the proposed architecture enables industrial control applications requiring a sampling rate up to 1000 Hz, even in presence of communication impairments.

Networked embedded systems for industrial control based on wireless support provide several advantages over wired counterparts, but often reveal unsuitable for the most demanding industrial control applications, such as advanced manufacturing and cooperative robotics. Data exchange over IEEE 802.11 networks may theoretically represent an appropriate solution for time-critical applications, provided that unreliability and non- determinism issues are properly handled. In this respect, this paper hence proposes an original solution, based on a cross- layer approach, to allow the realization of high-speed industrial control-over-Wi-Fi networked embedded systems. The proposal implements a novel robust frame-delay state estimator, a time efficient communication policy, and a specific tuning of critical protocol parameters. Suitable hardware-in-the-loop experiments have been carried out implemented exploiting two different embedded systems. Preliminary results show that the proposed architecture enables industrial control applications requiring a sampling rate of up to 1 kHz, even in presence of non negligible communication errors.

We consider the problem of classifying anomalous occupancy sensor behavior in connected indoor lighting systems. Anomalous occupancy sensor behavior may occur in the form of either a high number of false alarms (type-1 anomalies) or missed detections (type-2 anomalies). Two anomaly discovery scenarios are considered: one, in which no anomalies exist post-deployment, and two, in which both anomaly types are found together with normally functional sensors. We address the problem of classifying anomalies that may occur subsequently using a machine learning approach. Under scenario 1, we consider a one class random forest classifier to determine whether an occupancy signal is normal or not. In scenario 2, we consider a supervised random forest classifier to determine whether the detection signal of an occupancy sensor is normal, or exhibits type-1 or type-2 anomalies. We devise occupancy signal features in time and frequency domains to perform 2-class classification in scenario 1, and 3-class classification in scenario 2. The proposed method is evaluated using motion sensor data from an office building, and is shown to have higher true positive rate and a lower false positive rate in comparison to an unsupervised k -means method and a random forest classifier with a single signal energy feature.

Wireless power transfer networks (WPTNs) are composed of dedicated energy transmitters (ETs) that charge energy receivers (ERs) via radio frequency waves. A safe-charging WPTN should keep electromagnetic radiation below predetermined limits meanwhile maximizing the transmitted power. In this paper, we consider this requirement as an optimization problem: the maximization of harvested power by ERs subject to the electro-magnetic safety constraints. In order to provide an approximated solution to this problem, we introduce a dual ascent-like distributed charging algorithm that enables ETs to work without global information and satisfy safety constraints asymptotically. We provide an in-depth theoretical analysis of our algorithm which is supported by numerical simulations.

This paper deals with the problem of estimating in a collaborative way the gradient and the Hessian matrix of an unknown signal via noisy measurements collected by a group of robots. We propose symmetric formations with a reduced number of robots for both the two-dimensional (2-D) and the three-dimensional (3-D) cases, such that the gradient and Hessian of the signal are estimated at the center of the formation via simple computation on local quantities independently of the orientation of the formation. If only gradient information is required, the proposed formations are suitable for mobile robots that need to move in circular motion. We also provide explicit bounds for the approximation error and for the noise perturbation that can be used to optimally scale the formation radius. Numerical simulations illustrate the performance of the proposed strategy for source seeking against alternative solutions available in the literature and show how Hessian estimation can provide faster convergence even in the presence of noisy measurements.

We consider control of multiple luminaires with a central controller and occupancy and light sensors co-located at the luminaires. The sensors periodically provide local occupancy state and illumination information to the central controller. Using this sensor feedback, the central controller determines the dimming levels of the luminaires so as to adapt artificial light output to changing daylight levels and occupancy conditions, in an energy efficient way. We propose a multivariable feedback controller based on optimizing a cost function with a component corresponding to power consumption and another corresponding to illumination errors at the light sensors. The constraints are that the attained illumination value at the light sensors be no smaller than the reference set-points and that the dimming levels of luminaires are within physical limits. We compare the performance of the proposed controller with a simple stand-alone reference controller. We show via simulations in an open-plan office lighting system that the proposed controller has better performance in terms of achieving the reference set-points.

This paper considers the classification problem using support vector (SV) machines and investigates how to maximally reduce the size of the training set without losing information. Under separable data set assumptions, we derive the exact conditions stating which observations can be discarded without diminishing the overall information content. For this purpose, we introduce the concept of potential SVs, i.e., those data that can become SVs when future data become available. To complement this, we also characterize the set of discardable vectors (DVs), i.e., those data that, given the current data set, can never become SVs. Thus, these vectors are useless for future training purposes and can eventually be removed without loss of information. Then, we provide an efficient algorithm based on linear programming that returns the potential and DVs by constructing a simplex tableau. Finally, we compare it with alternative algorithms available in the literature on some synthetic data as well as on data sets from standard repositories.

In this paper, we study sensor fusion for the attitude estimation of micro aerial vehicles (MAVs), in particular mechanical flying insects. First, following a geometric approach, a dynamic observer is proposed which estimates attitude based on kinematic data available from different and redundant bio-inspired sensors such as halteres, ocelli, gravitometers, magnetic compass and light polarization compass. In particular, the traditional structure of complementary filters, suitable for multiple sensor fusion, is specialized to the Lie group of rigid body rotations SO(3). Then, a numerical implementation of the filter is provided for the specific case of inertial/magnetic navigation, i.e. when gravitometers, magnetometer and gyroscopes are available. Finally, the filter performance is experimentally tested via a 3 degrees-of-freedom robotic flapper and a custom-made set of inertial/magnetic sensors. Experimental results show good agreement, upon proper tuning of the filter, between the actual kinematics of the robotic flapper and the kinematics reconstructed from the inertial/magnetic sensors via the proposed filter.

In this paper, we consider the problem of estimating the state of a dynamical system from distributed noisy measurements. Each agent constructs a local estimate based on its own measurements and on the estimates from its neighbors. Estimation is performed via a two stage strategy, the first being a Kalman-like measurement update which does not require communication, and the second being an estimate fusion using a consensus matrix. In particular we study the interaction between the consensus matrix, the number of messages exchanged per sampling time, and the Kalman gain for scalar systems. We prove that optimizing the consensus matrix for fastest convergence and using the centralized optimal gain is not necessarily the optimal strategy if the number of exchanged messages per sampling time is small. Moreover, we show that although the joint optimization of the consensus matrix and the Kalman gain is in general a non-convex problem, it is possible to compute them under some relevant scenarios. We also provide some numerical examples to clarify some of the analytical results and compare them with alternative estimation strategies.

This paper considers control and estimation problems where the sensor signals and the actuator signals are transmitted to various subsystems over a network. In contrast to traditional control and estimation problems, here the observation and control packets may be lost or delayed. The unreliability of the underlying communication network is modeled stochastically by assigning probabilities to the successful transmission of packets. This requires a novel theory which generalizes classical control/estimation paradigms. The paper offers the foundations of such a novel theory.

This paper considers the problem of pursuit evasion games (PEGs), where the objective of a group of pursuers is to chase and capture a group of evaders in minimum time with the aid of a sensor network. The main challenge in developing a real-time control system using sensor networks is the inconsistency in sensor measurements due to packet loss, communication delay, and false detections. We address this challenge by developing a real-time hierarchical control system, namedLochNess, which decouples the estimation of evader states from the control of pursuers via multiple layers of data fusion. The multiple layers of data fusion convert noisy, inconsistent, and bursty sensor measurements into a consistent set of fused measurements. Three novel algorithms are developed forLochNess: multisensor fusion, hierarchical multitarget tracking, and multiagent coordination algorithms. The multisensor fusion algorithm converts correlated sensor measurements into position estimates, the hierarchical multitarget tracking algorithm based on Markov chain Monte Carlo data association (MCMCDA) tracks an unknown number of targets, and the multiagent coordination algorithm coordinates pursuers to chase and capture evaders using robust minimum-time control. The control systemLochNessis evaluated in simulation and successfully demonstrated using a large-scale outdoor sensor network deployment.

Several recent studies have addressed the suitability of current wireless technologies for industrial communications which, frequently, are requested to cope with severe timing constraints. Most of the analysis that have lately appeared in the literature have focused on the performance offered by the lower layers of the communication stack. However, in order to obtain a complete picture, it is important to analyze how this gets coupled with the features of higher layer protocols typically employed by industrial communication applications. In this paper, we investigate the performance obtained by an application layer protocol, derived from those currently employed by wired fieldbuses, running over COTS devices based on two popular wireless communication standards, namely, IEEE 802.15.4 and IEEE 802.11. In particular, we present a mapping of the application service elements onto the services offered by the mentioned wireless systems and discuss some possible design choices. A prototype of the application layer protocol is then implemented for each considered wireless technology and the performance figures evaluated by means of experimental measurements.

In this paper, we present the design of the flight control algorithms for flapping wing micromechanical flying insects (MFIs). Inspired by the sensory feedback and neuromotor structure of insects, we propose a similar top-down hierarchical architecture to achieve high performance despite the MFIs' limited on-board computational resources. The flight stabilization problem is formulated as high-frequency periodic control of an underactuated system. In particular, we provide a methodology to approximate the time-varying dynamics caused by the aerodynamic forces with a time-invariant model using averaging theory and a biomimetic parametrization of the wing trajectories. This approximation leads to a simpler dynamical model that can be identified using experimental data from the on-board sensors and the voltage inputs to the wing actuators. The overall control law is a periodic proportional output feedback. Simulations, including sensor and actuator models, demonstrate stable flight in hovering mode.

This paper presents the mathematical modeling of flapping flight inch-size micro aerial vehicles (MAVs), namely micromechanical flying insects (MFIs). The target robotic insects are electromechanical devices propelled by a pair of independent flapping wings to achieve sustained autonomous flight, thereby mimicking real insects. In this paper, we describe the system dynamic models which include several elements that are substantially different from those present in fixed or rotary wing MAVs. These models include the wing-thorax dynamics, the flapping flight aerodynamics at a low Reynolds number regime, the body dynamics, and the biomimetic sensory system consisting of ocelli, halteres, magnetic compass, and optical flow sensors. The mathematical models are developed based on biological principles, analytical models, and experimental data. They are presented in the Virtual Insect Flight Simulator (VIFS) and are integrated together to give a realistic simulation for MFI and insect flight. VIFS is a software tool intended for modeling flapping flight mechanisms and for testing and evaluating the performance of different flight control algorithms.

Multiple-target tracking is a canonical application of sensor networks as it exhibits different aspects of sensor networks such as event detection, sensor information fusion, multi-hop communication, sensor management and decision making. The task of tracking multiple objects in a sensor network is challenging due to constraints on a sensor node such as short communication and sensing ranges, a limited amount of memory and limited computational power. In addition, since a sensor network surveillance system needs to operate autonomously without human operators, it requires an autonomous tracking algorithm which can track an unknown number of targets. In this paper, we develop a scalable hierarchical multiple-target tracking algorithm that is autonomous and robust against transmission failures, communication delays and sensor localization error.

In this work we consider the problem of pursuit evasion games (PEGs) where a group of pursuers is required to detect, chase and capture a group of evaders with the aid of a sensor network in minimum time. Differently from standards PEGs where the environment and the location of evaders is unknown and a probabilistic map is built based on the pursuer’s onboard sensors, here we consider a scenario where a sensor network, previously deployed in the region of concern, can detect the presence of moving vehicles and can relay this information to the pursuers. Here we propose a general framework for the design of a hierarchical control architecture that exploits the advantages of a sensor network by combining both centralized and decentralized real-time control algorithms. We also propose a coordination scheme for the pursuers to minimize the time-to-capture of all evaders. In particular, we focus on PEGs with sensor networks orbiting in space for artificial space debris detection and removal.

In this paper, we study attitude stabilization strategies via output sensor feedback for micro aerial vehicles (MAVs), inch-size robots capable of autonomous flight. In order to compensate for the size and power limitations of MAVs, we introduce the ocelli and halteres, the body orientation and rotation sensing mechanisms used by flying insects. The analysis and simulations of these sensors show the feasibility of using such biologically inspired approaches to build biomimetic gyroscopes and angular position detectors. Finally, attitude stabilization techniques based on these sensors are proposed and successfully tested on an aerodynamic model for a micromechanical flying insect (NIFI). To the authors' knowledge, this is the first attempt in using output feedback from biomimetic devices with ocelli and halteres to achieve attitude stabilization in MAVs.

This paper describes recent developments on the model identification and attitude control system for a micromechanical flying insect (MFI). We include recently developed dynamical models for the thorax actuators and the various sensor models. Wing kinematic parameterization scheme was designed to generate feasible wing motions to decouple the body torques under the constraints of the thorax model. A nominal state-space LTI model in hover was identified through linear estimation and a LQR controller was designed to achieve stable hovering and steering maneuvers. Simulation results show satisfactory performance comparable to that of the real insects.

Four prototypes of biomimetic sensors have been designed and implemented for flight control of a robotic flying insect. The ocelli use four photodiodes to detect changes in light intensity in the surrounding. The halteres use piezo-actuated vibrating structures to sense body rotational velocities via the Coriolis forces. The optic flow sensors consist of linear arrays of elementary motion detectors (EMDs) to register optic flows. The magnetic field sensor uses three metal loops to detect changes in the magnetic field. Despite simplicity and novelty, the preliminary tests on these devices showed promising performance for using such biomimetic sensors on a robotic flying insect.

Sensor networks are gaining a central role in the research community. This paper addresses some of the issues arising from the use of sensor networks in control applications. Classical control theory proves to be insufficient in modeling distributed control problems where issues of communication delay, jitter, and time synchronization between components are not negligible. After discussing our hardware and software platform and our target application, we review useful models of computation and then suggest a mixed model for design, analysis, and synthesis of control algorithms within sensor networks. We present a hierarchical model composed of continuous time-trigger components at the low level and discrete event-triggered components at the high level.

This paper describes recent development on the design of the flight control system for a micromechanical flying insect (MFI), a 10-25 mm (wingtip-to-wingtip) device capable of sustained autonomous flight. High level attitude control is considered. Based on our previous work, in which the complex time-varying component of aerodynamic forces are treated as external disturbances, a nominal state-space linear time-invariant model in hover is developed through linear estimation. The identified model is validated through the virtual insect flight simulator (VIFS), and is used to design feedback controllers for the MFI. A LQG controller is designed and compared with a PD controller. The identification scheme provides a more systematic way of treating aerodynamic modeling errors, and the controllers designed based on the identified model shows better overall performance in simulation. Another advantage of this approach is that measurement of the instantaneous aerodynamic forces is not necessary, thus simplifies the experimental setup for the real MFI.

Body rotation and orientation sensing mechanisms used by flying insects are introduced and their mathematical models are presented. The analysis and simulations of these models showed the feasibility of using such biologically inspired approaches to build biomimetic gyroscopes and angular position detectors. Further, an approximate rigid body model for the insect body dynamics is developed so that attitude stabilization techniques for a flying robotic insect can be tested to illustrate the utility of these novel sensor architectures. To the authors' knowledge, this is the first attempt in using output feedback from biomimetic devices such as ocelli and halters to achieve attitude stabilization.

Describes results on the design and simulation of a flight control system for the micromechanical flying insect (MFI), a 10-25 mm (wingtip-to-wingtip) device eventually capable of sustained autonomous flight. The biologically inspired system architecture results in a hierarchical structure of different control methodologies, which give the possibility to plan complex missions from a sequence of simple flight modes and maneuvers. As a case study, a stabilizing hovering control scheme is presented and simulated with VIFS, a software simulator for insect flight.

Presents the design of the software simulator, VIFS, for insect flight. In particular, it is intended to estimate flight control algorithms and performance for a micromechanical flying insect (MFI), a 10-25 mm (wingtip-to-wingtip) device eventually capable of sustained autonomous flight. The VIFS is an end-to-end tool composed of several modular blocks which model actuator dynamics, wing aerodynamics, body motion, visual and inertial sensors, environment perception, and control algorithms. A 3D virtual environment simulation is also developed as a visualization tool. We present the current state of art of its implementation and preliminary results.