The ubiquitous deployment of robots across diverse domains, from industrial automation to personal care, underscores their critical role in modern society. However, this growing dependence has also revealed security vulnerabilities. An attack vector involves the deployment of malicious software (malware) on robots, which can cause harm to robots themselves, users, and even the surrounding environment. Machine learning approaches, particularly supervised ones, have shown promise in malware detection by building intricate models to identify known malicious code patterns. However, these methods are inherently limited in detecting unseen or zero-day malware variants as they require regularly updated massive datasets that might be unavailable to robots. To address this challenge, we introduce RoboGuardZ, a novel malware detection framework based on zero-shot learning for robots. This approach allows RoboGuardZ to identify unseen malware by establishing relationships between known malicious code and benign behaviors, allowing detection even before the code executes on the robot. To ensure practical deployment in resource-constrained robotic hardware, we employ a unique parallel structured pruning and quantization strategy that compresses the RoboGuardZ detection model by 37.4% while maintaining its accuracy. This strategy reduces the size of the model and computational demands, making it suitable for real-world robotic systems. We evaluated RoboGuardZ on a recent dataset containing real-world binary executables from multi-sensor autonomous car controllers. The framework was deployed on two popular robot embedded hardware platforms. Our results demonstrate an average detection accuracy of 94.25% and a low false negative rate of 5.8% with a minimal latency of 20 ms, which demonstrates its effectiveness and practicality.

Zero-day vulnerabilities pose a significant challenge to robot cyber-physical systems (CPS). Attackers can exploit software vulnerabilities in widely-used robotics software, such as the Robot Operating System (ROS), to manipulate robot behavior, compromising both safety and operational effectiveness. The hidden nature of these vulnerabilities requires strong defense mechanisms to guarantee the safety and dependability of robotic systems. In this paper, we introduce RoboCop, a cyber-physical attack detection framework designed to protect robots from zero-day threats. RoboCop leverages static software features in the pre-execution analysis along with runtime state monitoring to identify attack patterns and deviations that signal attacks, thus ensuring the robot’s operational integrity. We evaluated RoboCop on the F1-tenth autonomous car platform. It achieves a 93% detection accuracy against a variety of zero-day attacks targeting sensors, actuators, and controller logic. Importantly, in on-robot deployments, it identifies attacks in less than 7 seconds with a 12% computational overhead.

Designing distributed control systems (DCS) is challenging because we need to balance the application control performance with the real-time hardware resource costs. Compared to the abundance of automated tools for monolithic systems, managing the design and implementation of DCS still heavily relies on human expertise. To address this challenge, we propose a human-augmenting design tool that suggests and explores tradeoffs in co-optimizing controller simplification and multiprocessor distributed task scheduling with a simple dial. Our work creatively combines sparsity-based optimal controller simplification with task consolidation, to achieve an optimal control performance on the smallest distributed hardware footprint. Our unique task consolidator adapts to various real-time scheduling algorithms and allocates control tasks to the fewest nodes under the utilization bounds of the chosen real-time scheduler. To account for real-time guarantees when assigning the controller, we introduce an improved worst-case execution time (WCET) model based on a mixed Weibull distribution. This new WCET model provides state-of-the-art accuracy of WCET based upon fewer data samples, thus reducing design-time effort and improving run-time performance. With our tool, designers can virtually simulate the theoretical optimum, physically download it, or experimentally explore alternative tradeoffs between system cost and controller performance. In a real-world implementation, our tool reduced 60% of hardware costs by trading off merely 0.14% of control performance. By automating the “simplification – demux – WCET estimation – task consolidation” pipeline, our tool allows control engineers to balance control performance and system cost directly, speeding up manual distributed controller design by 16 times.

This article focuses on the state feedback control design problem for a specific class of strict-feedback fractional-order nonlinear systems with unknown state time-varying delays, external disturbance, and limited observability. A state feedback control scheme is proposed based on the introduction of a reduced-order observer. The design of this control scheme utilizes Lyapunov-Krasovskii functional with static gain, leading to the development of a novel delay-independent and memoryless control strategy. The main objective is to guarantee the boundedness of all signals in the closed-loop system and ensure asymptotic stability. The effectiveness of the proposed control scheme is validated through simulation and experimental tests, demonstrating their capability to address the control challenges posed by the considered system models in neural networks, robot control and a real-time oscillator applications. Simulation and empirical results illustrate the improved performance achieved by the control strategy over two existing adaptive control schemes in terms of lower-steady-state error, faster system responsiveness and increased stability (milder oscillation) in the state responses. The proposed control scheme offers a practical way of achieving desired control objectives, providing valuable insights for researchers and practitioners in the field of control engineering.

Ultrasound imaging is a commonly used modality for several diagnostic and therapeutic procedures. However, the diagnosis by ultrasound relies heavily on the quality of images assessed manually by sonographers, which diminishes the objectivity of the diagnosis and makes it operator-dependent. The supervised learning-based methods for automated quality assessment require manually annotated datasets, which are highly labour-intensive to acquire. These ultrasound images are low in quality and suffer from noisy annotations caused by inter-observer perceptual variations, which hampers learning efficiency. We propose an UnSupervised UltraSound image Quality assessment Network, US2QNet, that eliminates the burden and uncertainty of manual annotations. US2QNet uses the variational autoencoder embedded with the three modules, pre-processing, clustering and post-processing, to jointly enhance, extract, cluster and visualize the quality feature representation of ultrasound images. The pre-processing module uses filtering of images to point the network's attention towards salient quality features, rather than getting distracted by noise. Post-processing is proposed for visualizing the clusters of feature representations in 2D space. We validated the proposed framework for quality assessment of the urinary bladder ultrasound images. The proposed framework achieved 78% accuracy and superior performance to state-of-the-art clustering methods. The project page with source codes is available at https://sites.google.com/view/US2QNet.

Ultrasound is a vital imaging modality utilized for a variety of diagnostic and interventional procedures. However, an expert sonographer is required to make accurate maneuvers of the probe over the human body while making sense of the ultrasound images for diagnostic purposes. This procedure requires a substantial amount of training and up to a few years of experience. In this paper, we propose an autonomous robotic ultrasound system that uses Bayesian Optimization (BO) in combination with the domain expertise to predict and effectively scan the regions where diagnostic quality ultrasound images can be acquired. The quality map, which is a distribution of image quality in a scanning region, is estimated using Gaussian process in BO. This relies on a prior quality map modeled using expert's demonstration of the high-quality probing maneuvers. The ultrasound image quality feedback is provided to BO, which is estimated using a deep convolution neural network model. This model was previously trained on database of images labelled for diagnostic quality by expert radiologists. Experiments on three different urinary bladder phantoms validated that the proposed autonomous ultrasound system can acquire ultrasound images for diagnostic purposes with a probing position and force accuracy of 98.7% and 97.8%, respectively.

The promise of individualized care for improving animal welfare demands real-time continuous monitoring of animals. While technology has helped crop agriculture realize the goals of precision care, animal agriculture is still lacking domain-adapted technology. In this work, we present a novel Criticality-Aware Self-Powered in-vivo sensing Edge for pRecision animal agriculture, CASPER. Enabling real-time monitoring of a suite of biomarkers while scavenging power from both thermal and physiological sources, CASPER promises unprecedented adaptability, range, and life-cycle for such an edge node. Field deployments show that CASPER generates 30mW of power with a surplus of 9.08mW, during the ultra-low power mode. The criticality-aware control is proven to capture deviation in trends of biomarker activity that would be missed in fixed-interval transmission. Hence proving the validity and effectiveness of CASPER.

Smart devices have entered all spheres of modern living, from monitoring the steps we walk to managing refrigerator inventory, ushering in the dawn of a new urban experience. The kitchen is the heart of the home; a place to share, care for and nurture the family unit, but also a place seeing the greatest impact from the introduction of smart devices. The smart sensing and remote control capability of smart appliances have enabled great physical convenience for users but have had less impact on cognitive conveniences. While such devices can sensewhatthey are working with, they fail to understandwhothey are working for, leaving much of the burden of trivial planning and decision making to humans with less personalized services. Hence, we introduce TupperwareEarth, a knowledge-based ontological semantic network for the “Internet of Kitchen Things” with the aim of reducing physical as well as cognitive loads of humans in cooking tasks. Also, we present a testbed for exploring kitchen innovation and validating the effectiveness of TupperwareEarth that combines intelligent kitchen storage containers, Smart Tupperware, and existing smart kitchen appliances through an Internet of Things network and a user-friendly front-end interface, Tuppy. Using this testbed, the quantitative user studies show a 33% reduction in average food preparation time and qualitative user surveys show that 75% of the users observed a significant reduction in cognitive loads, thereby validating the cognitive conveniences granted by TupperwareEarth.

Everyday contact-rich tasks, such as peeling, cleaning, and writing, demand multimodal perception for effective and precise task execution. However, these present a novel challenge to robots as they lack the ability to combine these multimodal stimuli for performing contact-rich tasks. Learning-based methods have attempted to model multi-modal contact-rich tasks, but they often require extensive training examples and task-specific reward functions which limits their practicality and scope. Hence, we propose a generalizable model-free learning-from-demonstration framework for robots to learn contact-rich skills without explicit reward engineering. We present a novel multi-modal sensor data representation which improves the learning performance for contact-rich skills. We performed training and experiments using the real-life Sawyer robot for three everyday contact-rich skills – cleaning, writing, and peeling. Notably, the framework achieves a success rate of 100% for the peeling and writing skill, and 80% for the cleaning skill. Hence, this skill learning framework can be extended for learning other physical manipulation skills.

This paper describes the optimization of a neuromorphic architecture for printable organic neurons as part of an ongoing project to develop Form + Function 4-D Printing. The previously proposed architecture prioritizes simplicity and massive redundancy for a printable analog neural network consisting of only one transistor plus memristors for synapses, per neuron, but sacrifices negative synaptic weights. This paper demonstrates an optimization technique to minimize the insertion of inverting amplifiers to realize a minimal approximating set of negative weights. This helps to develop a compact, printable and accurate neuromorphic computer to bring new function to the 3-D printing of form in multi-functional robotic materials. An example robotic skin is developed with the ability to compute the centroid of touch "compiled into the skin" to an average accuracy of 9.19% in comparison to an unconstrained Artificial Neural Network (ANN). The presented soft robotic skin is fabricated by hand using conventional silicon components, but serves as a proof-of-concept for radical new capabilities in Form + Function 4-D Printing.

Telesurgery can be hindered by high-latency and low-bandwidth communication networks, often found in austere settings. Even delays of less than one second are known to negatively impact surgeries. To tackle the effects of connectivity associated with telerobotic surgeries, we propose the DESERTS framework. DESERTS provides a novel simulator interface where the surgeon can operate directly on a virtualized reality simulation and the activities are mirrored in a remote robot, almost simultaneously. Thus, the surgeon can perform the surgery uninterrupted, while high-level commands are extracted from his motions and are sent to a remote robotic agent. The simulated setup mirrors the remote environment, including an alpha-blended view of the remote scene. The framework abstracts the actions into atomic surgical maneuvers (surgemes) which eliminate the need to transmit compressed video information. This system uses a deep learning based architecture to perform live recognition of the surgemes executed by the operator. The robot then executes the received surgemes, thereby achieving semi-autonomy. The framework’s performance was tested on a peg transfer task. We evaluated the accuracy of the recognition and execution module independently as well as during live execution. Furthermore, we assessed the framework’s performance in the presence of increasing delays. Notably, the system maintained a task success rate of 87% from no-delays to 5 seconds of delay.

The paper discusses how robots enable occupant-safe continuous protection for students when schools reopen. Conventionally, fixed air filters are not used as a key pandemic prevention method for public indoor spaces because they are unable to trap the airborne pathogens in time in the entire room. However, by combining the mobility of a robot with air filtration, the efficacy of cleaning up the air around multiple people is largely increased. A disinfection co-robot prototype is thus developed to provide continuous and occupant-friendly protection to people gathering indoors, specifically for students in a classroom scenario. In a static classroom with students sitting in a grid pattern, the mobile robot is able to serve up to 14 students per cycle while reducing the worst-case pathogen dosage by 20%, and with higher robustness compared to a static filter. The extent of robot protection is optimized by tuning the passing distance and speed, such that a robot is able to serve more people given a threshold of worst-case dosage a person can receive.

The beneficial aspects of fully-actuated multirotor UAVs, provided by nonparallel rotor configuration, are increasingly being recognized and utilized to great benefit in high-precision applications. Full six-degree-of-freedom force control, higher control bandwidth and improved disturbance rejection prove valuable. However, the cant angle will cause great multirotor dihedral effect and significantly affects blade flapping, which decreases the flight performance of nonparallel actuated UAVs. Therefore, this paper presents a novel aerodynamic model for fully-actuated hexrotor UAVs while considering the aerodynamic effects caused due to tilt angled propeller configurations. In the proposed aerodynamic model, the significance of multirotor dihedral effect, defined as an aerodynamic coefficient proportional to the relative linear velocity of the UAV, is modeled for nonparallel actuators. Additionally, the modeling for blade flapping effect for cant angled propellers is provided to accurately model the aerodynamics. Wind tunnel experiments were conducted to characterize the aerodynamic constants for multirotor dihedral effect, blade flapping effect and air friction. Experimental results are presented to validate the proposed aerodynamic model on a fully-actuated hexrotor UAV (Purdue’s Dexterous Hexrotor). Lastly, the multirotor dihedral effect and blade flapping effect at different cant angles and at different wind speeds are analyzed.

In austere environments, teleoperated surgical robots could save the lives of critically injured patients if they can perform complex surgical maneuvers under limited communication bandwidth. The bandwidth requirement is reduced by transferring atomic surgical actions (referred to as “surgemes”) instead of the low-level kinematic information. While such a policy reduces the bandwidth requirement, it requires accurate recognition of the surgemes. In this paper, we demonstrate that transfer learning across surgical tasks can boost the performance of surgeme recognition. This is demonstrated by using a network pre-trained with peg-transfer data from Yumi robot to learn classification on debridement on data from Taurus robot. Using a pre-trained network improves the classification accuracy achieves a classification accuracy of 76% with only 8 sequences in target domain, which is 22.5% better than no-transfer scenario. Additionally, ablations on transfer learning indicate that transfer learning requires 40% less data compared to no-transfer to achieve same classification accuracy. Further, the convergence rate of the transfer learning setup is significantly higher than the no-transfer setup trained only on the target domain.

Many real-world time-sensitive and high-stake applications (e.g., surgical, rescue, and recovery robotics) exhibit sequential nature; thus, applying Recurrent Neural Network (RNN)-based sequential models is an attractive approach to detect robotic activity. One limitation of such approaches is data scarcity. As a result, limited training samples may lead to over-fitting, producing incorrect predictions during deployment. Nevertheless, abundant domain knowledge may still be available, which may help formulate logic constraints. In this paper, we propose a novel way to integrate domain knowledge into RNN-based sequential prediction. We build a Markov Logic Network (MLN)-based classifier that automatically learns constraint weights from data. We propose two methods to incorporate this MLN-based prediction: (i) PriorLayer, in which the values of the hidden layer of the RNN are combined with weights learned from logic constraints in an additional neural network layer, and (ii) Conflation, in which class probabilities from RNN predictions and constraint weights are combined based on the conflation of class probabilities. We evaluate robotic activity classification methods on a simulated OpenAI Gym environment and a real-world DESK dataset for surgical robotics. We observe that our proposed MLN-based approaches boost the performance of LSTM-based networks. In particular, MLN boosts the accuracy of LSTM from 71% to 84% on the Gym dataset and from 68% to 72% on the Taurus robot dataset. Furthermore, MLN (i.e., PriorLayer) shows regularization capability where it improves accuracy in initial LSTM training while avoiding over-fitting early, thus improves the final classification accuracy on unseen data. The code is available at https://github.com/masud99r/prediction-with-logic-constraints.

Inspection for structural properties (surface stiffness and coefficient of restitution) is crucial for understanding and performing aerial manipulations in unknown environments, with little to no prior knowledge on their state. Inspection-on-the-fly is the uncanny ability of humans to infer states during manipulation, reducing the necessity to perform inspection and manipulation separately. This paper presents an infrastructure for inspection-on-the-fly method for aerial manipulators using hybrid physical interaction control. With the proposed method, structural properties (surface stiffness and coefficient of restitution) can be estimated during physical interactions. A three-stage hybrid physical interaction control paradigm is presented to robustly approach, acquire and impart a desired force signature onto a surface. This is achieved by combining a hybrid force/motion controller with a model-based feed-forward impact control as intermediate phase. The proposed controller ensures a steady transition from unconstrained motion control to constrained force control, while reducing the lag associated with the force control phase. And an underlying Operational Space dynamic configuration manager permits complex, redundant vehicle/arm combinations. Experiments were carried out in a mock-up of a Dept. of Energy exhaust shaft, to show the effectiveness of the inspection-on-the-fly method to determine the structural properties of the target surface and the performance of the hybrid physical interaction controller in reducing the lag associated with force control phase.

Datasets are an essential component for training effective machine learning models. In particular, surgical robotic datasets have been key to many advances in semi-autonomous surgeries, skill assessment, and training. Simulated surgical environments can enhance the data collection process by making it faster, simpler and cheaper than real systems. In addition, combining data from multiple robotic domains can provide rich and diverse training data for transfer learning algorithms. In this paper, we present the DESK (DExterous Surgical SKills) dataset. It comprises a set of surgical robotic skills collected during a surgical training task using three robotic platforms: the Taurus II robot, Taurus II simulated robot, and the YuMi robot. This dataset was used to test the idea of transferring knowledge across different domains (e.g. from Taurus to YuMi robot) for a surgical gesture classification task with seven gestures/surgemes. We explored two different scenarios: 1) No transfer and 2) Domain transfer (simulated Taurus to real Taurus and YuMi robots). We conducted extensive experiments with three supervised learning models and provided baselines in each of these scenarios. Results show that using simulation data during training enhances the performance on the real robots, where limited real data is available. In particular, we obtained an accuracy of 55% on the real Taurus data using a model that is trained only on the simulator data, but that accuracy improved to 82% when the ratio of real to simulated data was increased to 0.18 in the training set.

Humans generally teach their fellow collaborators to perform tasks through a small number of demonstrations, often followed by episodes of coaching that tune and refine the execution during practice. Adopting a similar framework for teaching robots through demonstrations makes teaching tasks highly intuitive and imitating the refinement of complex tasks through coaching improves the efficacy. Unlike traditional Learning from Demonstration (LfD) approaches which rely on multiple demonstrations to train a task, we present a novel one-shot learning from demonstration approach, augmented by coaching, to transfer the task from task expert to robot. The demonstration is automatically segmented into a sequence of a priori skills (the task policy) parametrized to match task goals. During practice, the robotic skills self-evaluate their performances and refine the task policy to locally optimize cumulative performance. Then, human coaching further refines the task policy to explore and globally optimize the net performance. Both the self-evaluation and coaching are implemented using reinforcement learning (RL) methods. The proposed approach is evaluated using the task of scooping and unscooping granular media. The self-evaluator of the scooping skill uses the realtime force signature and resistive force theory to minimize scooping resistance similar to how humans scoop. Coaching feedback focuses modifications to sub-domains of the action space, using RL to converge to desired performance. Thus, the proposed method provides a framework for learning tasks from one demonstration and generalizing it using human feedback through coaching achieving a success rate of ≈90%.

In the future, deployable, teleoperated surgical robots can save the lives of critically injured patients in battlefield environments. These robotic systems will need to have autonomous capabilities to take over during communication delays and unexpected environmental conditions during critical phases of the procedure. Understanding and predicting the next surgical actions (referred as “surgemes”) is essential for autonomous surgery. Most approaches for surgeme recognition cannot cope with the high variability associated with austere environments and thereby cannot “transfer” well to field robotics. We propose a methodology that uses compact image representations with kinematic features for surgeme recognition in the DESK dataset. This dataset offers samples for surgical procedures over different robotic platforms with a high variability in the setup. We performed surgeme classification in two setups: 1) No transfer, 2) Transfer from a simulated scenario to two real deployable robots. Then, the results were compared with recognition accuracies using only kinematic data with the same experimental setup. The results show that our approach improves the recognition performance over kinematic data across different domains. The proposed approach produced a transfer accuracy gain up to 20% between the simulated and the real robot, and up to 31% between the simulated robot and a different robot. A transfer accuracy gain was observed for all cases, even those already above 90%.

Network connectivity maintenance is essential for efficient robotic team operations. Achieving robust robotic ad hoc network connectivity requires a capable link maintenance mechanism, especially if the network experiences expected intermittent connectivity. Although various routing protocols for wireless ad hoc networks have been proposed, they deal with the message routing and the link maintenance problems separately, resulting in additional overhead costs and long network latency. These limitations motivate us to develop a new routing mechanism for a robotic network, which we called Meta-Routing. Meta-Routing expands current routing protocols to include not only the normal routing of packets, but also the maintenance of broken links in robotic networks. This paper presents a method to achieve Meta-Routing by controlling robot motion based on gradient descent method augmented with the radio frequency (RF) environment recognition method. The RF recognition method utilizes hidden Markov models (HMMs) to categorize "RF shadows" into a set of RF obstacle primitives from partial information. The motion control algorithm utilizes the gradient estimations augmented with HMM results through RF recognition to determine the robot movement direction for achieving connectivity maintenance. The numerical experimental results demonstrate promising RF recognition and gradient estimations results as well as confirm their abilities in robot motion control for link maintenance, reduction of the total path cost and network latency for achieving Meta-Routing.

Previous work in the area of gesture production, has made the assumption that machines can replicate humanlike gestures by connecting a bounded set of salient points in the motion trajectory. Those inflection points were hypothesized to also display cognitive saliency. The purpose of this paper is to validate that claim using electroencephalography (EEG). That is, this paper attempts to find neural signatures of gestures (also referred as placeholders) in human cognition, which facilitate the understanding, learning and repetition of gestures. Further, it is discussed whether there is a direct mapping between the placeholders and kinematic salient points in the gesture trajectories. These are expressed as relationships between inflection points in the gestures trajectories with oscillatory mu rhythms (8-12 Hz) in the EEG. This is achieved by correlating fluctuations in mu power during gesture observation with salient motion points found for each gesture. Peaks in the EEG signal at central electrodes (motor cortex; C3/Cz/C4) and occipital electrodes (visual cortex; O3/Oz/O4) were used to isolate the salient events within each gesture. We found that a linear model predicting mu peaks from motion inflections fits the data well. Increases in EEG power were detected 380 and 500ms after inflection points at occipital and central electrodes, respectively. These results suggest that coordinated activity in visual and motor cortices is sensitive to motion trajectories during gesture observation, and it is consistent with the proposal that inflection points operate as placeholders in gesture recognition.

User's intentions may be expressed through spontaneous gesturing, which have been seen only a few times or never before. Recognizing such gestures involves one shot gesture learning. While most research has focused on the recognition of the gestures themselves, recently new approaches were proposed to deal with gesture perception and production as part of the recognition problem. The framework presented in this work focuses on learning the process that leads to gesture generation, rather than treating the gestures as the outcomes of a stochastic process only. This is achieved by leveraging kinematic and cognitive aspects of human interaction. These factors enable the artificial production of realistic gesture samples originated from a single observation, which in turn are used as training sets for state-of-the-art classifiers. Classification performance is evaluated in terms of recognition accuracy and coherency; the latter being a novel metric that determines the level of agreement between humans and machines. Specifically, the referred machines are robots which perform artificially generated examples. Coherency in recognition was determined at 93.8%, corresponding to a recognition accuracy of 89.2% for the classifiers and 92.5% for human participants. A proof of concept was performed towards the expansion of the proposed one shot learning approach to adaptive learning, and the results are presented and the implications discussed.

This paper presents a new numeric estimation for dynamic performance of fully-actuated UAV platforms with nonparallel actuation mechanisms. Multi-rotor UAV designs with tilted rotors can achieve fully-actuated performance (net thrust that spans the 6-DoF force-torque space) by tilting the rotors in both cant and dihedral. These orthogonal rotations have different implications on the intrinsic stability of the platform. A numeric estimation of the multi-rotor performances is proposed with an optimization approach to configure the nonparallel thrusters with respect to desired tasks. The new model is applied to Purdues Dexterous Hexrotor UAV and CyPhy Works LVL 1 platform, to illustrate the multi-rotor estimation and optimization. Estimation of these fully-actuated UAVs include precise physical interaction with the environment (Dexterous Hexrotor), hovering efficiency, and influence operating altitude. Experimental results are presented to validate the optimized result with a contact based force control task in preparation for a demo at the DOE Gaseous Diffusion Plant.

We proposed ReFrESH in our previous publication. It is a self-adaptive infrastructure aimed at managing the performance of multi-robot systems through dynamically diagnosing and maintaining unexpected issues of modules. To integrate ReFrESH and robotic application-level software more conveniently, it is necessary to develop a module design framework to support implementation of self-adaptive real-time software. To this end, based on the port-based object abstraction and port-automation theory, we propose the Extended Port-Based Object (E-PBO). E-PBO has two main advantages: (1) it builds the basis of a programming model to provide specific, yet flexible, guidelines to robotics application engineers for creating and integrating software modules; (2) it forms the basis of a self-adaption model to provide specific methods for evaluating the running task configuration and estimating the new but non-running task configuration (if required) without interfering with the running configuration. E-PBO has been incorporated into the Port-Based Object Real-Time Operating System (PBO/RT) and applied to a visual servoing robotic application, which is demonstrated here.

Mobile robots are being employed far more often in extreme environments, such as urban search and rescue, with greater levels of autonomy; yet recent studies on field robotics show that numerous failure modes affect the reliability of the robot in meeting mission objectives. Therefore, fault tolerance is increasingly important for field robots operating in unpredictable environments to ensure safety and effectiveness of the system. This paper demonstrates a self-adaptation framework, ReFrESH, that contains mechanisms for fault detection and fault mitigation. The goal of ReFrESH is to provide diagnosable and maintainable infrastructure support, built into a real-time operating system, to manage task performance in the presence of unexpected uncertainties. ReFrESH augments the port-based object framework by attaching evaluation and estimation mechanisms to each functional component so that the robot can easily detect and locate faults. In conjunction, a task level decision mechanism interacts with the fault detection elements in order to generate and choose an optimal approach to mitigating faults. Moreover, to increase flexibility of the fault tolerance, ReFrESH provides self-adaptation support for both software and hardware functionality. To our knowledge, this is the first framework to support both software and hardware self-adaptation. A demonstrative application of ReFrESH illustrates its applicability through a target tracking task deployed on a mobile robot system.

This paper presents an anthropomorphic robotic bear for the exploration of human-robot interaction including verbal and non-verbal communications. This robot is implemented with a hybrid face composed of a mechanical faceplate with 10 DOFs and an LCD-display-equipped mouth. The facial emotions of the bear are designed based on the description of the Facial Action Coding System as well as some animal-like gestures described by Darwin. The mouth movements are realized by synthesizing emotions with speech. User acceptance investigations have been conducted to evaluate the likability of these facial behaviors exhibited by the eBear. Multiple Kernel Learning is proposed to fuse different features for recognizing user's facial expressions. Our experimental results show that the developed Bear-Like robot can perceive basic facial expressions and provide emotive conveyance towards human beings.

This paper presents a method for proactive robot communication connectivity maintenance based on electromagnetic field (EMF) recognition and signal strength (SS) gradient estimation for mobile robots. To achieve these goals in an efficient manner, we combine EMF recognition method and gradient descent of SS measurements into a proactive robot motion control algorithm in a way that maintains connectivity among mobile robots in the presence of a radio frequency (RF) obstacle. The EMF recognition method utilizes hidden Markov models (HMMs) for learning EMF environments based on SS measurements. The proposed motion control algorithm uses the EMF recognition and gradient method results to drive the robots towards favorable locations in which robots can communicate. The numerical simulation demonstrates promising EMF recognition, robot motion control results and confirms their abilities in proactive robot motion control for connectivity maintenance.

This paper presents a robust method for 3D object rotation estimation using spherical harmonics representation and the unit quaternion vector. The proposed method provides a closed-form solution for rotation estimation without recurrence relations or searching for point correspondences between two objects. The rotation estimation problem is casted as a minimization problem, which finds the optimum rotation angles between two objects of interest in the frequency domain. The optimum rotation angles are obtained by calculating the unit quaternion vector from a symmetric matrix, which is constructed from the two sets of spherical harmonics coefficients using eigendecomposition technique. Our experimental results on hundreds of 3D objects show that our proposed method is very accurate in rotation estimation, robust to noisy data, missing surface points, and can handle intra-class variability between 3D objects.

Structured Computational Polymers (SCP) is a concept of layered class of active material that can sense its environment and, due to its cognitive capabilities, react “intelligently” to those changes. In such a material, we envision semiconducting polymer based sensing, actuation, and information processing for on-board decision making to be combined into one active material. This paper describes incremental steps taken towards developing such a multifunctional active material, concentrating on distributed forms of actuation and cognition, with an intermediate goal of utilizing SCP as a “skin” of a soft robot - a robot, made of flexible materials, which is not bounded by its rigid structure and can adjust to its changing environment - with its sensing, cognition, and actuation embedded in the shape. We demonstrate, via experiment and rudimentary simulation, the feasibility of utilizing water hammer as a form of directed, distributed actuation. We also show that distributed form of cognition can be realized via a novel concept termed Synthetic Neural Network (SNN), which is a type of organic neuromorphic architecture modeled after Artificial Neural Network. SNN, based on a single-transistor-single-memristor-per-input for an individual neuron, can approximate the sigmoidal activation function with an accuracy of about 3%. A simulation of the SNN is shown to accurately predict the directionality of water hammer propulsion with an accuracy of 7.2 percent.

Artificial Intelligence (AI) has made tremendous progress since it was first postulated in the 1950s. However, AI systems are primarily emulated on serial machine hardware that result in high power consumption, especially when compared to their biological counterparts. Recent interest in neuromorphic architectures aims to more directly emulate biological information processing to achieve substantially lower power consumption for appropriate information processing tasks. We propose a novel way of realizing a neuromorphic architecture, termed Synthetic Neural Network (SNN), that is modeled after conventional artificial neural networks and incorporates organic bistable devices as circuit elements that resemble the basic operation of a binary synapse. Via computer simulation we demonstrate how a single synthetic neuron, created with only a single transistor, a single-bistable-device-per-input, and two resistors, exhibits a behavior of an artificial neuron and approximates the sigmoidal activation function. We also show that, by increasing the number of bistable devices per input, a single neuron can be trained to behave like a Boolean logic AND or OR gate. To validate the efficacy of our design, we show two simulations where SNN is used as a pattern classifier of complicated, non-linear relationships based on real-world problems. In the first example, our SNN is shown to perform the trained task of directional propulsion due to water hammer effect with an average error of about 7.2%. The second task, a robotic wall following, resulted in SNN error of approximately 9.6%. Our simulations and analysis are based on the performance of organic electronic elements created in our laboratory.

Biological neural networks are spectacularly more energy efficient than currently available man-made, transistor-based information processing units. Additionally, biological systems do not suffer catastrophic failures when subjected to physical damage, but experience proportional performance degradation. Hardware neural networks promise great advantages in information processing tasks that are inherently parallel or are deployed in an environment where the processing unit might be susceptible to physical damage. This paper, intended for hardware neural network applications, presents analysis of performance degradation of various architectures of artificial neural networks when subjected to `stuck-at-0' and `stuck-at-1' faults. This study aims to determine if a fixed number of neurons should be kept in a single or multiple hidden layers. Faults are administered to input and hidden layer(s) and analysis of unoptimized and optimized, feedforward and recurrent networks, trained with uncorrelated and correlated data sets is conducted. A comparison of networks with single, dual, triple, and quadruple hidden layers is quantified. The main finding is that `stuck-at-0' faults administered to input layer result in least performance degradation in networks with multiple hidden layers. However, for `stuck-at-0' faults occurring to cells in hidden layer(s), the architecture that sustains the least damage is that of a single hidden layer. When `stuck-at-1' errors are applied to either input or hidden layers, the network(s) that offer the most resilience are those with multiple hidden layers. The study suggests that hardware neural network architecture should be chosen based on the most likely type of damage that the system may be subjected to, namely damage to sensors or the neural network itself.

We propose a digital video stabilization process using information that the scale-invariant feature transform (SIFT) provides for each frame. We use a fuzzy clustering scheme to separate the SIFT features representing global motion from those representing local motion. We then calculate the global orientation change and translation between the current frame and the previous frame. Each frame's translation and orientation is added to an accumulated total, and a Kalman filter is applied to estimate the desired motion. We provide experimental results from five video sequences using peak signal-to-noise ratio (PSNR) and qualitative analysis.

Statically and dynamically reconfigurable robot mechanisms have been extensively studied by a number of researchers. Control formulations have been proposed for specific mechanisms and some researchers have tried to build unified frameworks for general robot control. This paper reports an extension of the differential drive mechanism that we call the Heterogeneous Differential Drive. Bridging the gap between differential drive mechanisms and skid-steered mechanisms, the heterogeneous differential drive permits the modular combination of different types of actuators with different capabilities under one unified framework. It is a step toward a unified framework for actuators we call Heterogeneous Drive Mechanisms that permits reconfigurable mechanisms with homogeneous or heterogeneous components. The heterogeneous differential drive is a theoretical class of vehicles that lies in the gray area between pure differential drive vehicles and pure skid steered vehicles, yet represents either at the extremes. The heterogeneous differential drive also provides the basis for our preliminary development of the heterogeneous drive. This paper develops the kinematic model of the heterogeneous differential drive from the kinematic model of the differential drive formulation and describes an example mechanism.

For certain applications, field robotic systems require small size for cost, weight, access, stealth or other reasons. Small size results in constraints on critical resources such as power, space (for sensors and actuators), and computing cycles, but these robots still must perform many of the challenging tasks of their larger brethren. The need for advanced capabilities such as machine vision, application-specific sensing, path planning, self localization, etc. is not reduced by small-scale applications, but needs may vary with the task. As a result, when resources are constrained, it is prudent to configure the robot for the task at hand; both hardware and software. We are developing a reconfigurable computing subsystem for resource-constrained robots that allows rapid deployment of statically configured hardware and software for a specific task. The use of a field programmable gate array (FPGA) provides flexibility in hardware for both sensor interfacing and hardware-accelerated computation. In this paper, we describe a static reconfiguration architecture we call the morphing bus that allows the rapid assembly of sensors and dedicated computation through reusable hardware and software modules. It is a novel sensor bus in the fact that no bus interface circuitry is required on the sensor side - the bus "morphs" to accommodate the signals of the sensor

Limbs are an attractive approach to certain niche robotic applications, such as urban search and rescue, that require both small size and the ability to locomote through highly rubbled terrain. Unfortunately, a large number of degrees of freedom implies there is a large space of non- optimal locomotion trajectories (gaits), making gait adaptation critical. On the other hand, these extra degrees of freedom open many possibilities for active sensing of the terrain, which is essential information for adapting the gait. In previous work, we developed a metric for terrain classification that makes use of the loping body motion (i.e. gait bounce) during locomotion. In this work we present a framework for evolving gaits to better differentiate the gait bounce signal across terrains. This framework includes a limb/terrain interaction model that estimates gait bounce based on established models of wheel/terrain interaction, and an objective function that can be optimized for terrain discriminability. Additional objective functions for improved locomotion are presented, as well as culling agents that help guide the evolution process away from real-world impossibilities.

We present a novel technique for the detection of thrown objects and other free-flying bodies in video sequences. Our method runs in real-time and was designed to be used as a component in a deployed surveillance system. We detect regions of interesting motion that fit certain size, compactness and speed criteria, and use the expectation maximization algorithm to detect objects on parabolic trajectories over a short time window. The system was shown to successfully detect thrown objects of various sizes in a large test set of indoor and outdoor videos.

For controlling robots in an urban search and rescue (USAR) application, a wearable joystick is presented with improved sensing capability as well as a giant magneto-resistance (GMR) sensor model for use with rare-earth magnets. Scientists have been studying a variety of existing human/robot interface devices to control USAR robots in a disaster. Due to the stresses involved in USAR environments, the selection of an appropriate interface device out of the numerous interactive devices available has to be carefully considered. Furthermore, the total burden to the user of human/robot interface devices in USAR tasks includes not only the periods of interaction, but also the burden of transporting and remotely setting up the devices. The wearable joystick presented is developed with the design goal of minimizing total encumbrances. The features of this wearable joystick include easy and wire-free installation into regular gloves. An improved hardware structure for the sensor pad and the alignment of magnets is described that completely wraps the wrist. This band-type mechanism provides more robust data acquisition than previous prototypes. To evaluate performance, time-to-complete tests are performed, with a comparison to a metric for path tortuosity. The fractal dimension of the resulting path is analyzed to represent the degree of control the user has over the interface device. Experimental results are provided from both computer screen tess and real USAR robot driving tests.

This paper presents a novel technique for camera tampering detection. It is implemented in real-time and was developed for use in surveillance and security applications. This method identifies camera tampering by detecting large differences between older frames of video and more recent frames. A buffer of incoming video frames is kept and three different measures of image dissimilarity are used to compare the frames. After normalization, a set of conditions is tested to decide if camera tampering has occurred. The effects of adjusting the internal parameters of the algorithm are examined. The performance of this method is shown to be extremely favorable in real-world settings.

This paper presents a new design of a two-axis MEMS (microelectromechanical systems) capacitive force sensor with strict linearity and a new sensor calibration method for micro-sensors. Precise calibration of multi-axis micro force sensors is difficult for several reasons, including the need to apply many known force vectors at precise orientations at the micro force scale, and the risk of damaging the small, fragile MEMS device. In this paper the shape from motion method is introduced for micro force sensors resulting in a rapid and effective calibration technique. Structural-electrostatic coupled field simulations are conducted in order to optimize the sensor design, which is calibrated with the shape from motion method as well as the least squares method for comparison purposes. Calibration results demonstrate that the shape from motion method is an effective, practical, and accurate method for calibrating multiaxis micro force sensors

We present a new terrain classification technique both for effective, autonomous locomotion over natural, unknown terrains and for the qualitative analysis of terrains for exploration and mapping. Our straight-forward approach requires a single camera with little processing of visual information. Specifically, we derived a gait bounce measure from visual servoing errors that result from vehicle-terrain interactions during normal locomotion. Characteristics of the terrain, such as roughness and compliance, manifest themselves in the spatial patterns of this signal and can be extracted using pattern classification techniques. For legged robots, different limb-terrain interactions generate gait bounce signals with different information content, thus deliberate limb motions can effect higher information content (i.e. the robot is an active sensor of terrain class). Segmentation of the gait cycle based on the limb-terrain interaction isolates portions of the gait bounce signal with high information content. The decoding of, then sequencing of, this content from each cycle segment yields a robust classification of terrain type from known benchmarks. To extract this spatio-temporal pattern of the gait bounce signal, we developed a meta-classifier using discriminant analysis and hidden Markov model. We present the gait bounce derivation. We demonstrate the viability of terrain classification for legged vehicles using gait bounce with a rigorous study of more than 700 trials, obtaining 84% accuracy. We describe how terrain classification can be used for gait adaptation, particularly in relation to an efficiency metric. We also demonstrate that our technique is generally applicable to other locomotion mechanisms such as wheels and treads.

Micro-positioning devices are increasingly being made of parallel manipulators due to their superior stiffness characteristics. This paper explores a variant of the classical six degrees-of-freedom Stewart-Gough platforms for use in micro-positioning applications. Stiffness values of both these mechanisms are computed and compared for relative gain in theoretical stiffness achieved. An over-sized version of the platform was built for educational purposes.

We have investigated the performance of simultaneously estimating the 3D motion and structure for navigation when the scale information is obtained by utilizing the cooperative efforts of multiple robots. The method determines the relative positions of robots by tracking a specific geometric feature that is part of their structure, and then uses the extended Kalman filter to estimate the motion and structure. For implementation we used two CRAWLER Scouts, and performed several experiments to explore the effects of cooperative running of robots on the motion estimation.

A custom version of the TerminatorBot is described for core bored inspection during search-and-rescue operations. "Core bored inspection" refers to visual inspection of a void by passing a small camera through an access hole into the void. This is the classic "camera-on-a-stick" approach. Sometimes the access hole occurs naturally. Sometimes a suspected void has no access hole. To gain access, a hole is bored through the rubble with a coring tool, hence the term "core-bored inspection". In either case, the camera, once inside, can articulate to look around, but is limited to fine-of-sight. Occlusions can prevent a thorough inspection or force using/boring another hole. A small, agile robotic device could augment the use of such cameras. We propose the TerminatorBot as a prototype limbed robot for studying such applications.

Sequencing sensorimotor primitives to achieve complex behaviors can simplify programming of robotic systems. Using programming by demonstration to code the component primitives can further simplify the process. Learning methods employed in programming by demonstration require comprehensive data sets, which place a significant burden on the user during demonstration. We present a generalized method whereby training sets can be automatically filtered, freeing the user from knowledge of the underlying learning method. We achieve this by first capturing the characteristic behavior for a demonstrated task, then determining a measure of distance from that behavior. With this information, data sets can be analyzed to determine whether a particular moment of demonstration is "good" and should be included in the final training set. Results from programming by demonstration of left wall-following on a mobile platform are presented. Additionally, we present a method for on-line performance analysis that takes advantage of the characteristic behavior identified in the filtering process.

We present a method to evaluate the performance of an eigenvector learned sensorimotor primitive for mobile robots. At runtime, the learning system projects sensor data onto the eigenspace using eigenvectors determined in training. The result of the projection is a set of sensor values and actuator values. We developed an error metric based on comparing the projected values with the actual sensor values. When the system performs closely to how it was trained, the difference between projected and actual sensors is small and hence the error metric is small. The error increases as the performance degrades. This method is not task specific and can be used for any eigenvector learned primitive. Two example applications of the error metric are shown using wall following skills for a mobile robot. First, the metric is used as a transition cue for multiprimitive sequential tasks. Second, the error metric is used to create an adaptive system that chooses the best performing skill.

A novel, centimeter-scale crawling robot has been developed to address applications in surveillance, search-and-rescue, and planetary exploration. This places constraints on size and durability that minimizes the mechanism. As a result, a dual-use design employing two arms for both manipulation and locomotion was conceived. In a complementary fashion, this paper investigates the dual-use of visual servoing error. Visual servoing can be used by a mobile robot for homing and tracking. But because ground-based mobile robots are inherently planar, the control methodology (steering) is one-dimensional. The two-dimensional nature of image-based servoing leaves additional information content to be used in other contexts. We explore this information in the context of classifying terrain conditions. An outline for gait adaptation based on this is suggested for future work.

Presents a miniature robotic system ("scout") useful for reconnaissance and surveillance missions. A large number of scout robots are deployed and controlled by humans and/or larger "ranger" robots. The specially designed and constructed scouts are extremely small (roughly 116cc volume) yet are readily deployable (by tossing or launching), have multiple mobility modes, have multiple sensing capabilities, can transmit and receive data and instructions, and have a limited capability for autonomous action. The rangers are significantly larger vehicles, based on a commercial-off-the-shelf platform, augmented with scout launchers, radios, and additional sensors. Together, the scouts and rangers form a hierarchical team capable of carrying out complex missions in a wide variety of environments.

A novel, miniature robot designed to use its two arms for both manipulation and locomotion is described. Intended for military and civilian surveillance and search-and-rescue applications, the robot must be small, rugged, and lightweight, hence the desire for dual-use. The robot consists of two, three-degree-of-freedom arms that can stow completely inside the 75 mm diameter cylindrical body for ballistic deployment. This paper describes the mechanism and design motivation as well as two novel locomotion gaits and a third conventional gait.

In this paper we present an active video module that consists of a miniature video sensor, a wireless video transmitter and a pan-tilt mechanism driven by micromotors. The video module is part of a miniature mobile robot that is projected to areas of the environment to be surveyed. A single-chip CMOS video sensor and miniature brushless DC gearmotors are used to comply with restrictions imposed by the robotic system in terms of payload weight volume and power consumption. Different types of actuation are analyzed for compatibility with a mesoscale robotic system. Applications of an active video module are discussed.

This paper describes a demonstration-based programming system in which a mobile robot observes the actions of a human performing a multi-step task. From these observations, the robot determines which of its pre-learned capabilities are required to replicate the task and in what sequence they must be ordered. The focus of this paper is on the hidden Markov model method used to learn and classify the actions as "gestures". A preliminary system demonstration is also described in which the robot observes the human performing a block distribution task. During the demonstration, the robot actively follows the demonstrator to maintain its vantage point and to infer spatial relationships.

Explores gesture-based programming as a paradigm for programming robotic agents. Gesture-based programming is a form of programming by human demonstration that focuses the development of robotic systems on task experts rather than programming experts. The technique relies on the existence of previously acquired robotic skills (which we call "sensorimotor primitives") which we hope to develop to into the robotic equivalent of skills acquired by humans through everyday experiences. The interpretation of the human's demonstration and subsequent matching to robotic primitives is a qualitative problem that we approach with a community of skilled agents. A simple manipulative task and a variant of that task are programmed to demonstrate the system.

Positioning an eye-in-hand robotic system with respect to a static target is a challenging research problem since it involves recognition of the object and the desired pose at which alignment is to occur, planning a trajectory for the robot to attain this pose, and careful calibration of the system and the environment. In this paper we introduce a unified framework based on image morphing, to address the above problems and apply it to the task of translational and rotational alignment of an eye-in-hand system to planar objects or planar projections of 3D objects. In our method the desired manipulator pose for each object is defined and stored as a view of the object taken from this pose. The identity of an unknown object in the workspace is established by morphing its image to the views in the database, and using a quantification of the morph as a dissimilarity measure. The synthetic images generated during the morph are used guide an eye-in-hand system to the desired pose. The framework can accommodate partially occluded or deformable targets and smooth trajectories can be generated since an arbitrary number of intermediate images can be used.

The tropism system cognitive architecture provides an intuitive formalism for colonies of agents, either hardware or software. We present a fine-grained implementation of the architecture on a colony of software agents for the interpretation of human tactile gestures for robotic trajectory specification and modification. The fine-grained nature of the architecture and the use of the port-based object framework for agent instantiation allows the manual construction of a capable agent set that is reconfigurable and reusable across different gesture-based interaction tasks.

In this paper we summarize recent research results from a new technique of sensor calibration called shape-from-motion calibration. We first present the basic technique, which is an eigenspace analysis, and show that it includes the rigor of least squares without the full burden of measuring all the applied inputs. Next we present new research that removes another constraint of the calibration technique and extends the robustness to cover systems with slight nonlinearities.

We present the design of a modular tactile sensor and actuator system for observing human demonstrations of contact tasks. The system consists of three interchangeable parts: an intrinsic tactile sensor for measuring net force/torque, an extrinsic tactile sensor for measuring contact distributions, and a tactile actuator for displaying tactile distributions. The novel components are the extrinsic sensor and tactile actuator which are "inside-out symmetric" to each other and employ an electrorheological gel for actuation.

Gesture-based programming is a new paradigm to ease the burden of programming robots. By tapping in to the user's wealth of experience with contact transitions, compliance, uncertainty and operations sequencing, we hope to provide a more intuitive programming environment for complex, real-world tasks based on the expressiveness of nonverbal communication. A requirement for this to be accomplished is the ability to interpret gestures to infer the intentions behind them. As a first step toward this goal, this paper presents an application of distributed perception for inferring a user's intentions by observing tactile gestures. These gestures consist of sparse, inexact, physical "nudges" applied to the robot's end effector for the purpose of modifying its trajectory in free space. A set of independent agents-each with its own local, fuzzified, heuristic model of a particular trajectory parameter observes data from a wristforce/torque sensor to evaluate the gestures. The agents then independently determine the confidence of their respective findings and distributed arbitration resolves the interpretation through voting.