In multi-robot systems, the successful execution of tasks typically depends on predefined instructions. However, existing approaches encounter substantial challenges in dynamic environments, particularly in autonomously reasoning, generating task instructions, and allocating tasks. These challenges are further exacerbated by the need to address complex temporal, spatial, and heterogeneous task constraints. To address these limitations, inspired by the success of the Large Language Models (LLMs) in natural language understanding and logical inference, this paper proposes an environment-driven and LLM-guided task inference and allocation framework with dual-system temporal logics. The framework consists of three key modules: the Environment Module, which employs environment LTL to continuous monitor and verify environmental resource constraints; the Inference Module, leveraging LLMs for autonomous generation and verification of robotic tasks in response to resource changes; and the Robot Module, which explores the feasible task allocation for the multi-robot system. When the resources in the environment do not satisfy the specification, the Inference Module is used to analyze and infer the feasible actions to be executed by the multi-robot system, so as to alleviate the environmental resource problem. Experimental results demonstrate the scalability, efficiency, and autonomy of our framework across varying task environments and robot configurations.

Existing multirobot task allocation (MRTA) research primarily assumes an environment with known geometric and semantic information. However, in most real-world scenarios, semantic information, such as the locations and classifications of landmarks, is often uncertain. This uncertainty is exacerbated by dynamic requirements, like the sudden addition or removal of tasks, making it challenging for robots to respond reactively. Moreover, tasks in MRTA typically involve multiple constraints, including temporal requirements, diverse capabilities, varying resource needs, and inter-task dependencies. To address these challenges, we propose a reactive task allocation framework for heterogeneous multirobot systems that accounts for temporal requirements and multiple task constraints described by $\mathrm {LTL^{\mathcal {R}}}$ . Our approach assumes an environment with known geometry but unknown semantic landmarks. To efficiently solve task allocations, we encode $\mathrm {LTL^{\mathcal {R}}}$ along with the system's states into a proposed planning decision tree for exploration. Upon detecting a relevant semantic landmark, the reactive multiconstraint planning decision tree (RMC-PDT) is triggered for re-planning. Extensive experiments validate three key features of our method: 1) efficient reactive planning; 2) multiconstraint task solving; and 3) scalability.

Efficiently searching for multiple objects in a partially known environment, where only the names and locations of landmarks are available, presents significant challenges. Existing search algorithms in the literature fail to fully utilize prior knowledge to improve search efficiency, and exhibit significantly diminished efficiency when extended to multiobject search. To address these limitations, we propose an inference-based multi-object reactive search framework. This framework utilizes the COMET inference model to reason about co-occurrence values between the target objects and known landmarks, thereby enhancing search efficiency. These co-occurrence values are integrated into a reactive temporal logic motion planning strategy, which allows the robot search for multiple objects with temporal logic constraints specified by LTL and adapt dynamically if the inferred reasoning differs from the actual object arrangement encountered during the search. Extensive simulations were conducted to evaluate the feasibility and efficiency of the proposed motion planning algorithm. Results demonstrate that the integration of commonsense reasoning with reactive temporal logic planning significantly improves multi-object search efficiency. Project website: https://sites.google.com/view/imors.

Human-robot interaction plays a critical role in scientific experiments by ensuring efficient and reliable execution of experimental tasks. To achieve successful task completion, robots must adapt in real time to unexpected task variations, external disturbances, and safety constraints. In this work, we propose a reactive task and motion planning framework designed to address these challenges. By formulating interaction tasks using Linear Temporal Logic (LTL), our approach introduces Planning Decision Tree and Augmented Planning Decision Tree approach to dynamically adjust task sequences in response to environmental changes. At the execution layer, we employ a Model Predictive Path Integral controller, which ensures both efficient and safe control. Additionally, the planning interface effectively coordinates the planning and execution layers, ensuring strict adherence to experimental task specifications. The effectiveness of the proposed reactive planning framework is demonstrated through physical experiments using a 7-DoFs robot. Project website: https://sites.google.com/view/rtlp-iros/

Designing a whole-body controller for loco-manipulation in unstructured real-world environments remains a formidable challenge. Previous approaches have primarily focused on extending the workspace of robotic arms while maintaining quadrupedal landing postures. However, these methods fail to fully exploit the mobility of legged robots. To address these limitations, we propose a unified framework for collision-free loco-manipulation in real-world applications. The framework comprises two key modules: (1) a Loco-manipulation Motion Prior, which generates loco-manipulation skill trajectories via Trajectory Optimization (TO), and (2) a Collision-free Manipulation module using a Model Predictive Path Integral (MPPI)-based trajectory generator and a vector-based trajectory follower. Extensive experiments have been conducted in both simulation and real-world scenarios to evaluate our framework’s tracking accuracy, whole-body coordination, and workspace expansion capabilities. Supplementary and videos are available at: https://sites.google.com/view/loco-mani-amp/

Formal languages effectively outline robots’ task specifications, yet current temporal logic struggles to balance semantic expression with solution speed. To address this challenge, we propose extended predicate-based temporal logic (E-pTL), augmenting conventional linear temporal logic with more expressive atomic predicates to reflect the time, space, and order attributes of task and represent complex tasks with dynamic propositions, time windows, and even relative time window. This approach, blending automata-based task abstraction and extended predicates, offers enhanced expressiveness and conciseness for intricate specifications. To cope with E-pTL, we introduce a novel planning framework, the Planning Decision Tree (PDT). PDT incrementally builds a tree through automata and system state searches, recording potential task plans. The proposed pruning method can reduce the exploration space. This method swiftly handles complex temporal tasks defined by E-pTL. Rigorous analysis confirms PDT-based planning’s feasibility (ensuring satisfactory planning aligned with task specifications) and completeness (guaranteeing a feasible solution if available). Moreover, PDT-based planning proves efficient, with solution times approximately linearly proportional to automaton states squared. Extensive simulations and experiments validate its effectiveness and efficiency. Note to Practitioners—Temporal logic, as a formal language, has been widely applied to describe the task of robotic systems. However, linear temporal logic (LTL) struggles with the explicit time constraints, while other temporal logics, e.g., signal temporal logic (STL), cannot compactly represent task progress via an automaton. Consequently, current solutions for complex task with temporal logic specification heavily lean on optimization-based approaches, which are time-consuming and impractical for real-time systems. This work presents a novel approach for fast motion planning in dynamic environments with extended predicate-based temporal logic (E-pTL). The proposed E-pTL not only enhances semantic expression but also can be checked by an automaton, simplifying progress checks. However, existing automata-based methods encounter limitations in dynamic tasks. Graph search methods require replanning at each time step and lack completeness guarantees, while sampling methods can’t efficiently handle the feasibility of each atomic proposition. In contrast, the developed planning decision tree (PDT)-based framework incrementally searches for automata and validates propositions without a transition system, significantly reducing the search space. The suggested pruning method further trims the search space without compromising feasibility. Overall, PDT enables rapid, reliable planning, outperforming optimization-based methods like ILP/MILP and proving well-suited for real-time applications.

Human-robot collaboration plays an important role in intelligent manufacturing. However, the main challenge is how the robot can make online reactive changes to the plan based on the observed human behavior to ensure the completion of user-defined tasks. Such a challenge is further exacerbated if eye-in-hand manipulation is considered since the local field of the camera view cannot capture global observations. Different from existing planning approaches that separate the perception and planning modules, and make strong assumptions about perception abilities, we develop a framework of real-time local reactive planning that enables the robot to quickly adapt its actions if necessary through its limited perception of surroundings using an eye-in-hand camera. Specifically, we develop a locally observable transition system (LOTS) and interpretably express the task using linear temporal logic (LTL). To improve the grasping performance using local visual perception, we propose a high-resolution grasp network (HRG-Net) that achieves state-of-the-art results on multiple datasets (99.50% in Cornell and 97.50% in Jacquard and 96% in Graspnet-1Billion) for the task. A physical experiment using a 7DoF Franka Emika Panda robot demonstrates the effectiveness of the reactive planning framework.Note to Practitioners—Intelligent manufacturing often requires the human operator to work collaboratively with the robot in a shared workspace. Due to possible (assistive or non-assistive) interference of human operators, it is highly desired that the robot can perceive human behaviors and react properly to ensure task accomplishment. Hence, this work is particularly motivated to develop a reactive planning framework that relies on real-time local visual perception (i.e., eye-in-hand camera) to quickly react to its dynamic surroundings and replan its motion when necessary. In future work, rather than using the observed human behavior, we will investigate how to predict human intentions to further improve human-robot collaboration.

Unsupervised domain adaptive semantic segmentation aims to transfer knowledge from the annotated source domain to the unlabeled target domain. Recently, self-training methods have gained substantial attention, which leverage high-confidence predictions in the target domain as pseudo labels for supervision. However, limited exploration of intra-class variations across domains, including significant visual differences within each category, has led to misalignment between feature distribution across domains. In this article, we present a unified non-parametric distance-based online clustering method to efficiently maintain multiple centroid-based prototypes within each category subspace instead of one prototype for each category subspace, which enables prototypes to possess the capacity for richer feature representation. Then, considering the variance across different dimensions of a feature representation, we then extend the prototypes from centroid-based ones to distribution-based ones. Specifically, each subspace is modeled using a Gaussian mixture model which includes several anisotropic Gaussian distributions, aimed at prioritizing discriminative dimensions and obtaining a finer measurement of the pixel-to-prototype similarity. Meanwhile, a category-aware feature space is achieved through pixel-to-prototype contrastive learning to ensure the compactness of pixel features in the same subcategory and drive the separation between pixel features of different subcategories. What's more, multi-resolution features are utilized to promote diversity and robustness among intra-class prototypes. Experiments validate the competitiveness of our two prototype-based methods against existing state-of-the-art methods, with a mIoU of 76.8% on GTA$\rightarrow$Cityscapes, 68.4% on Synthia$\rightarrow$Cityscapes, 54.5% on Cityscapes$\rightarrow$DarkZurich and 56.4% on Cityscapes$\rightarrow$ACDC. Notably, our method is able to seamlessly integrate with existing UDA methods.

Safe reinforcement learning (RL) has shown great potential for building safe general-purpose robotic systems. While many existing works have focused on post-training policy safety, it remains an open problem to ensure safety during training as well as to improve exploration efficiency. Motivated to address these challenges, this work develops shielded planning guided policy optimization (SPPO), a new model-based safe RL method that augments policy optimization algorithms with path planning and shielding mechanism. In particular, SPPO is equipped with shielded planning for guided exploration and efficient data collection via model predictive path integral (MPPI), along with an advantage-based shielding rule to keep the above processes safe. Based on the collected safe data, a task-oriented parameter optimization (TOPO) method is used for policy improvement, as well as the observation-independent latent dynamics enhancement. In addition, SPPO provides explicit theoretical guarantees, i.e., clear theoretical bounds for training safety, deployment safety, and the learned policy performance. Experiments demonstrate that SPPO outperforms baselines in terms of policy performance, learning efficiency, and safety performance during training.

Learning-based policy optimization methods have shown great potential for building general-purpose control systems. However, existing methods still struggle to achieve complex task objectives while ensuring policy safety during learning and execution phases for black-box systems. To address these challenges, we develop data-driven safe policy optimization (D2SPO), a novel reinforcement learning (RL)-based policy improvement method that jointly learns a control barrier function (CBF) for system safety and a linear temporal logic (LTL) guided RL algorithm for complex task objectives. Unlike many existing works that assume known system dynamics, by carefully constructing the data sets and redesigning the loss functions of D2SPO, a provably safe CBF is learned for black-box dynamical systems, which continuously evolves for improved system safety as RL interacts with the environment. To deal with complex task objectives, we take advantage of the capability of LTL in representing the task progress and develop LTL-guided RL policy for efficient completion of various tasks with LTL objectives. Extensive numerical and experimental studies demonstrate that D2SPO outperforms most state-of-the-art (SOTA) baselines and can achieve over 95% safety rate and nearly 100% task completion rates. The experiment video is available at https://youtu.be/2RgaH-zcmkY.

As a focal point of research in various fields, human body language understanding has long been a subject of intense interest. Within this realm, the exploration of emotion recognition through the analysis of facial expressions, voice patterns, and physiological signals holds significant practical value. Compared with unimodal approaches, multimodal emotion recognition models leverage complementary information from vision, acoustic, and language modalities to robust perceive the human sentiment attitudes. However, the heterogeneity among modality signals leads to significant domain shifts, posing challenges for achieving balanced fusion. In this article, we propose a Domain-Separated Bottleneck Attention (DBA) Fusion Framework for human multimodal emotion recognition with lower computational complexity. Specifically, we partition each modality into two distinct domains: the invariant/private domain. The invariant domain contains crucial shared information, while the private domain aims to capture modality-specific representations. For the decomposed features, we introduce two sets of bottleneck cross-attention modules to effectively utilize the complementarity between domains to reduce redundant information. In each module, we interweave two Fusion Adapter blocks into the Self-Attention Transformer backbone. Each Fusion Adapter block integrates a small group of latent tokens as bridges for inter-modal and inter-domain interactions, mitigating the adverse effects of modality distribution differences and lowering computational costs. Extensive experimental results demonstrate that our method outperforms State-of-the-Art (SOTA) approaches across three widely used benchmark datasets.

Reactive planning enables the robots to deal with dynamic events in uncertain environments. However, existing methods heavily rely on the predefined hard-coded robot behaviors, e.g, a pre-coded temporal logic formula that specifies how robot should react. Little attention has been paid for autonomous generation of reactive tasks specifications during the runtime. As a first attempt towards this goal, this work develops a real-time decision-making and motion planning framework. It allows the robot to follow a global task planned offline while taking proactive decisions and generating temporal logic specifications for local reactive tasks when encountering dynamic events. Specifically, inspired by the causal knowledge graph, a proposition graph is developed, based on which the decision module encode the environment and the task as the Boolean logic and linear temporal logic (LTL), respectively. Based on the established proposition graph and perceived environment, the agent can autonomously generate an LTL formula to realize the local temporary task. A joint sampling algorithm is then developed, in which the automaton states of local and global task are jointly considered to generate a feasible planning that satisfies both global and local tasks. Experiments demonstrate the effectiveness of the proposed decision-making and motion planning.

Visual-based robot pose estimation is a fundamental challenge, involving the determination of the camera’s pose with respect to a robot. Conventional methods for camera-to-robot pose calibration rely on fiducial markers to establish keypoint correspondences. However, these approaches exhibit significant variability in accuracy and robustness, particularly in 2D keypoint detection. In this work, we present an end-to-end pose estimation approach that achieves camera-to-robot calibration using monocular images and keypoint information. Our method employs a two-level nested U-shaped architecture, featuring a bottom-level residual U-block to extract richer contextual information from diverse receptive fields to enhance keypoint refinement. By incorporating the perspective-n-point (PnP) algorithm and leveraging 3D robot joint keypoints, we establish correspondence of 3D coordinate points between the robot’s coordinate system and the camera’s coordinate system, facilitating accurate pose estimation. Experimental evaluations encompass real-world and synthetic datasets, demonstrating competitive results across three distinct robot manipulators.

While reinforcement learning (RL) attracts increasing research attention, maximizing the return while keeping the agent safe at the same time remains an open problem. Motivated to address this challenge, this work proposes a new Fast and Safe Policy Optimization (FSPO) algorithm, which consists of three steps: the first step involves reward improvement update, the second step projects the policy to the neighborhood of the baseline policy to accelerate the optimization process, and the third step addresses the constraint violation by projecting the policy back onto the constraint set. Such a projection-based optimization can improve the convergence and learning performance. Unlike many existing works that require convex approximations for the objectives and constraints, this work exploits a first-order method to avoid expensive computations and high dimensional issues, enabling fast and safe policy optimization, especially for challenging tasks. Numerical simulation and physical experiments demonstrate that FSPO outperforms existing methods in terms of safety guarantees and task completion rate.

Affordance grounding aims to localize the interaction regions for the manipulated objects in the scene image according to given instructions, which is essential for Embodied AI and manipulation tasks. A key challenge in affordance grounding is enabling the agent to understand human instructions, identify usable tools in the environment, and determine how to use them to complete the task. Most recent works primarily support simple action labels as input instructions for localizing affordance regions, failing to capture complex human objectives. Moreover, these approaches typically identify affordance regions of only a single object in object-centric images, ignoring the object context and struggling to localize affordance regions of multiple objects in complex scenes for practical applications. To address this concern, for the first time, we introduce a new task of affordance grounding based on natural language instructions, extending it from previously using simple labels for complex human instructions. For this new task, we propose a new framework, WorldAfford. We design a novel Affordance Reasoning Chain-of-Thought Prompting to reason about affordance knowledge from LLMs more precisely and logically. Subsequently, we use SAM and CLIP to localize the objects related to the affordance knowledge in the image. We identify the affordance regions of the objects through an affordance region localization module. To benchmark this new task and validate our framework, an affordance grounding dataset, LLMaFF, is constructed. We conduct extensive experiments to verify that WorldAfford performs state-of-the-art on the previous AGD20K and the new LLMaFF dataset. In particular, WorldAfford can localize the affordance regions of multiple objects and provide an alternative when objects in the environment cannot fully match the given instruction. Our Project page: https://worldafford.github.io/.

This works exploits soft constraints in linear temporal logic task planning to enhance the agent’s capability in handling potentially conflicting or even infeasible tasks. Different from most existing works that focus on sticking to the original plan and trying to find a relaxed plan if the workspace does not permit, we augment the soft constraints to represent possible candidate sub-tasks that can be selected to fulfill the global task. Specifically, a hierarchical temporal logic specification is developed to represent LTL tasks with soft constraints and preferences. The hierarchical structure consists of an outer and inner layer, where the outer layer uses co-safe LTL to specify the task-level specifications and the inner layer specifies the low-level task-related atomic propositions via soft constraints. To cope with the hierarchical temporal logic specification, a hierarchical iterative search (HIS) algorithm is developed, which incrementally searches feasible atomic propositions and automaton states, and returns a task plan with minimum cost. Rigorous analysis shows that HIS based planning is feasible (i.e., the generated plan is applicable and satisfactory with respect to the task specification) and optimal (i.e, with minimum cost). Extensive simulation demonstrates the effectiveness of the proposed soft task planning approach.

Empowering legged robots with agile maneuvers is a great challenge. While existing works have proposed diverse control-based and learning-based methods, it remains an open problem to endow robots with animal-like perception and athleticism. Towards this goal, we develop an End-to-End Legged Perceptive Parkour Skill Learning (LEEPS) framework to train quadruped robots to master parkour skills in complex environments. In particular, LEEPS incorporates a vision-based perception module equipped with multi-layered scans, supplying robots with comprehensive, precise, and adaptable information about their surroundings. Leveraging such visual data, a position-based task formulation liberates the robot from velocity constraints and directs it toward the target using innovative reward mechanisms. The resulting controller empowers an affordable quadruped robot to successfully traverse previously challenging and unprecedented obstacles. We evaluate LEEPS on various challenging tasks, which demonstrate its effectiveness, robustness, and generalizability. Supplementary and videos are available at: https://sites.google.com/view/leeps

Reinforcement Learning (RL) based methods have been increasingly explored for robot learning. However, RL based methods often suffer from low sampling efficiency in the exploration phase, especially for long-horizon manipulation tasks, and generally neglect the semantic information from the task level, resulted in a delayed convergence or even tasks failure. To tackle these challenges, we propose a Temporal-Logic-guided Hybrid policy framework (HyTL) which leverages three-level decision layers to improve the agent’s performance. Specifically, the task specifications are encoded via linear temporal logic (LTL) to improve performance and offer interpretability. And a waypoints planning module is designed with the feedback from the LTL-encoded task level as a high-level policy to improve the exploration efficiency. The middle-level policy selects which behavior primitives to execute, and the low-level policy specifies the corresponding parameters to interact with the environment. We evaluate HyTL on four challenging manipulation tasks, which demonstrate its effectiveness and interpretability. Our project is available at: https://sites.google.com/view/hytl-0257/.

Wearable walking exoskeletons show great potentials in helping patients with neuro musculoskeletal stroke. Key to the successful applications is the design of effective walking trajectories that enable smooth walking for exoskeletons. This work proposes a walking planning method based on the divergent component of motion to obtain a stable joint angle trajectory. Since periodic and nonperiodic disturbances are ubiquitous in the repeating walking motion of an exoskeleton system, a major challenge in the walking control of wearable exoskeleton is the joint angle drift problem, that is, the joint angle motion trajectories are not necessarily periodic due to the presence of disturbance. To address this challenge, this work develops an adaptive repetitive control strategy to guarantee that the motion trajectories of joint angle are repetitive. In particular, by treating the disturbance as system uncertainties, an adaptive controller is designed to compensate for the uncertainties based on an integral-type Lyapunov function. A fully saturated learning approach is then developed to achieve asymptotic tracking of repetitive walking trajectories. Extensive experiments are carried out to demonstrate the effectiveness of the tracking performance.

It is an interesting open problem to enable robots to efficiently and effectively learn long-horizon manipulation skills. Motivated to augment robot learning via more effective exploration, this work develops task-driven reinforcement learning with action primitives (TRAPs), a new manipulation skill learning framework that augments standard reinforcement learning algorithms with formal methods and parameterized action space (PAS). In particular, TRAPs uses linear temporal logic (LTL) to specify complex manipulation skills. LTL progression, a semantics-preserving rewriting operation, is then used to decompose the training task at an abstract level, informs the robot about their current task progress, and guides them via reward functions. The PAS, a predefined library of heterogeneous action primitives, further improves the efficiency of robot exploration. We highlight that TRAPs augments the learning of manipulation skills in both learning efficiency and effectiveness (i.e., task constraints). Extensive empirical studies demonstrate that TRAPs outperforms most existing methods.

Transparent objects are widely used in industrial automation and daily life. However, robust visual recognition and perception of transparent objects have always been a major challenge. Currently, most commercial-grade depth cameras are still not good at sensing the surfaces of transparent objects due to the refraction and reflection of light. In this work, we present a transformer-based transparent object depth estimation approach from a single RGB-D input. We observe that the global characteristics of the transformer make it easier to extract contextual information to perform depth estimation of transparent areas. In addition, to better enhance the fine-grained features, a feature fusion module (FFM) is designed to assist coherent prediction. Our empirical evidence demonstrates that our model delivers significant improvements in recent popular datasets, e.g., 25% gain on RMSE and 21% gain on REL compared to previous state-of-the-art convolutional-based counterparts in ClearGrasp dataset. Extensive results show that our transformer-based model enables better aggregation of the object's RGB and inaccurate depth information to obtain a better depth representation. Our code and the pre-trained model are available at https://github.com/yuchendoudou/TODE.

Fast motion planning is of great significance, espe-cially when a timely mission is desired. However, the complexity of motion planning can grow drastically with the increase of environment details and mission complexity. This challenge can be further exacerbated if the tasks are coupled with the desired locations in the environment. To address these issues, this work aims at fast motion planning problems with temporal logical specifications. In particular, we develop a hierarchical decoupling framework that consists of three layers: the high-level task planner, the decoupling layer, and the low-level motion planner. The decoupling layer is designed to bridge the high and low layers by providing necessary information exchange. Such a framework enables the decoupling of the task planner and path planner, so that they can run independently, which significantly reduces the search space and enables fast planing in continuous or high-dimension discrete workspaces. In addition, the implicit constraint during task-level planning is taken into account, so that the low-level path planning is guaranteed to satisfy the mission requirements. Numerical simulations demonstrate at least one order of magnitude speed up in terms of computational time over existing methods.

Learning-based path planning is becoming a promising robot navigation methodology due to its adaptability to various environments. However, the expensive computing and storage associated with networks impose significant challenges for their deployment on low-cost robots. Motivated by this practical challenge, we develop a lightweight neural path planning architecture with a dual input network and a hybrid sampler for resource-constrained robotic systems. Our architecture is designed with efficient task feature extraction and fusion modules to translate the given planning instance into a guidance map. The hybrid sampler is then applied to restrict the planning within the prospective regions indicated by the guide map. To enable the network training, we further construct a publicly available dataset with various successful planning instances. Numerical simulations and physical experiments demonstrate that, compared with baseline approaches, our approach has nearly an order of magnitude fewer model size and five times lower computational while achieving promising performance. Besides, our approach can also accelerate the planning convergence process with fewer planning iterations compared to sample-based methods.

Dynamic movement primitives (DMPs) are a flexible trajectory learning scheme widely used in motion generation of robotic systems. However, existing DMP-based methods mainly focus on simple go-to-goal tasks. Motivated to handle tasks beyond point-to-point motion planning, this work presents temporal logic guided optimization of motion primitives, namely$\mathbf{PI}^{\mathbf{BB}-\mathbf{TL}}$algorithm, for complex manipulation tasks with user preferences. In particular, weighted truncated linear temporal logic (wTLTL) is incorporated in the$\mathbf{PI}^{\mathbf{BB}-\mathbf{TL}}$algorithm, which not only enables the encoding of complex tasks that involve a sequence of logically organized action plans with user preferences, but also provides a convenient and efficient means to design the cost function. The black-box optimization is then adapted to identify optimal shape parameters of DMPs to enable motion planning of robotic systems. The effectiveness of the$\mathbf{PI}^{\mathbf{BB}-\mathbf{TL}}$algorithm is demonstrated via simulation and experiment.

The exoskeleton is mainly used by subjects who suffer muscle injury to enhance motor ability in the daily life environment. Previous research seldom considers extending human collaboration skills to human-robot collaborations. In this article, two models, that is: 1) the following the better model and 2) the interpersonal goal integration model, are designed to facilitate the human-human collaborative manipulation in tracking a moving target. Integrated with dual-arm exoskeletons, these two models can enable the robot to successfully perform target tracking with two human partners. Specifically, the manipulation workspace of the human-exoskeleton system is divided into a human region and a robot region. In the human region, the human acts as the leader during cooperation, while, in the robot region, the robot takes the leading role. A novel region-based Barrier Lyapunov function (BLF) is then designed to handle the change of leader roles between the human and the robot and ensures the operation within the constrained human and robot regions when driving the dual-arm exoskeleton to track the moving target. The designed adaptive controller ensures the convergence of tracking errors in the presence of region switches. Experiments are performed on the dual-arm robotic exoskeleton for the subject with muscle damage or some degree of motor dysfunctions to evaluate the proposed controller in tracking a moving target, and the experimental results demonstrate the effectiveness of the developed control.

Many previous works of soft wearable exoskeletons (exosuit) target at improving the human locomotion assistance, without considering the impedance adaption to interact with the unpredictable dynamics and external environment, preferably outside the laboratory environments. This article proposes a novel hierarchical human-in-the-loop paradigm that aims to produce suitable assistance powers for cable-driven lower limb exosuits to aid the ankle joint in pushing off the ground. It includes two primary loop layers: impedance learning in the external loop and human-in-the-loop adaptive management in the inner loop. Considering unknown terrains, its impedance model can be transferred to a quadratic programming problem with specified constraints, which a designed primal-dual optimization prototype then solves. Then, the presented impedance learning strategy is introduced to regulate the impedance model with the adaptive assistant powers for humans on different terrains. An adaptive controller is designed in the inner loop to balance the nonlinearities and compliance existing in the human-exosuit coexistence, while the robust mechanism compensates for disturbances to facilitate trajectory management without employing the general regressor. The advantage of the proposed technique over conventional solutions with fixed impedance parameters is that it can improve human walking performance over different terrains. Experiments demonstrate the significance of the approach.

In intelligent driving, situational assessment (SA) is an important technology, which helps to improve the cognitive ability of intelligent vehicles in the environment. Uncertainty analysis is very significant in situation assessment. This article proposes an SA method based on uncertainty risk analysis. Under uncertain conditions, according to the random environment model and Gaussian distribution model, the collision probability between multiple vehicles is estimated by comprehensive trajectory prediction. The proposed method considers collision probabilities of different prediction points within and outside the prediction range and obtains long-term accurate prediction results. The method is suitable for the situation risk assessment of sensor systems in the presence of unexpected dynamic obstacles, sensor failures or communication losses in traffic, and different environmental sensing accuracy. The experimental results show that in the dynamic traffic environment, the proposed scenario assessment method can not only accurately predict and assess the situation risks within the prediction range, but also provide accurate scenario risk assessment outside the prediction range.

This paper presents a model-free reinforcement learning (RL) algorithm to synthesize a control policy that maximizes the satisfaction probability of complex tasks, which are expressed by linear temporal logic (LTL) specifications. Due to the consideration of environment and motion uncertainties, we model the robot motion as a probabilistic labeled Markov decision process (PL-MDP) with unknown transition probabilities and probabilistic labeling functions. The LTL task specification is converted to a limit deterministic generalized Büchi automaton (LDGBA) with several accepting sets to maintain dense rewards during learning. The novelty of applying LDGBA is to construct an embedded LDGBA (E-LDGBA) by designing a synchronous tracking-frontier function, which enables the record of non-visited accepting sets of LDGBA at each round of the repeated visiting pattern, to overcome the difficulties of directly applying conventional LDGBA. With appropriate dependent reward and discount functions, rigorous analysis shows that any method, which optimizes the expected discount return of the RL-based approach, is guaranteed to find the optimal policy to maximize the satisfaction probability of the LTL specifications. A model-free RL-based motion planning strategy is developed to generate the optimal policy in this paper. The effectiveness of the RL-based control synthesis is demonstrated via simulation and experimental results.

Singularity issue has long been a concern of the task-space control design for Euler-Lagrange systems. In classical task-space controls, robots are often assumed to operate in the task space, where singularities do not exist. Such an assumption limits their potential applications in various workspaces. To address the potential singularity issue associated with Euler-Lagrange systems, this article proposes an adaptive fuzzy-region-based control for Euler-Lagrange systems with kinematically singular configurations. Singular regions are described by the potential energy function. The proposed controller includes a joint-space control, which is active when the system approaches singular regions, and a task-space control, which is used to track the desired trajectory. Therefore, the system can smoothly transit from singular regions to nonsingular regions or can achieve singularity avoidance during the tracking task. In order to achieve singularity avoidance while reducing control effort, the coefficients of the potential energy function are adjusted dynamically based on the designed fuzzy system. Rigorous analysis shows that singularity issues can be properly handled, and the asymptotic stability of the system is ensured. Experiments are conducted to demonstrate the effectiveness of the proposed controller.

Leader-follower controllability on signed multiagent networks is investigated in this paper. Specifically, we consider a dynamic signed multiagent network, where the agents interact via neighbor-based Laplacian feedback and the network allows positive and negative edges to capture cooperative and competitive interactions among agents. The agents are classified as either leaders or followers, thus forming a leader-follower signed network. To enable full control of the leader-follower signed network, controllability ensured leader group selection approaches are investigated in this paper, that is, identifying a small subset of nodes in the signed network, such that the selected nodes are able to drive the network to a desired behavior, even in the presence of antagonistic interactions. In particular, graphical characterizations of the controllability of signed networks are first developed based on the investigation of the interaction between network topology and agent dynamics. Since signed path and cycle graphs are basic building blocks for a variety of networks, the developed topological characterizations are then exploited to develop leader selection methods for signed path and cycle graphs to ensure leader-follower controllability. Along with illustrative examples, heuristic algorithms are also developed showing how leader selection methods developed for path and cycle graphs can be potentially extended to more general signed networks. In contrast to existing results that mainly focus on unsigned networks, this paper characterizes controllability and develops leader selection methods for signed networks. In addition, the developed results are generic, in the sense that they are not only applicable to signed networks but also to unsigned networks, since unsigned networks are a particular case of signed networks that only contain positive edges.

For human-robot co-manipulation by robotic exoskeletons, the interaction forces provide a communication channel through which the human and the robot can coordinate their actions. In this article, an optimization approach for reshaping the physical interactive trajectory is presented in the co-manipulation tasks, which combines impedance control to enable the human to adjust both the desired and the actual trajectories of the robot. Different from previous studies, the proposed method significantly reshapes the desired trajectory during physical human-robot interaction (pHRI) based on force feedback, without requiring constant human guidance. The proposed scheme first formulates a quadratically constrained programming problem, which is then solved by neural dynamics optimization to obtain a smooth and minimal-energy trajectory similar to the natural human movement. Then, we propose an adaptive neural-network controller based on the barrier Lyapunov function (BLF), which enables the robot to handle the uncertain dynamics and the joint space constraints directly. To validate the proposed method, we perform experiments on the exoskeleton robot with human operators for co-manipulation tasks. The experimental results demonstrate that the proposed controller could complete the co-manipulation tasks effectively.

Recent advances in fine manipulation have led to increased interest in both scientific research works and engineering applications. Robot manipulation at a level approaching human skills is gaining attention in both industrial and individual services. A major challenge in fine manipulation is the unavoidable uncertainties and unpredictable conditions encountered in dynamic and unstructured application environments. The employment of biologically inspired (bioinspired) embodiments in fine manipulation shows significant advantages in tackling such problems. The aim of bioinspired embodiment is to improve fine manipulation of robotic systems utilizing the knowledge gained from natural systems with biomimetic methods. Such a method includes sensing, planning, and execution. This article provides a comprehensive survey of the current state of bioinspired technologies in fine manipulation, and outlines new challenges and some potential directions.

A decentralized controller is designed for leader-based synchronization of communication-delayed networked agents. The agents have heterogeneous dynamics modeled by uncertain, nonlinear Euler-Lagrange equations of motion affected by heterogeneous, unknown, exogenous disturbances. The developed controller requires only one-hop (delayed) communication from network neighbors and the communication delays are assumed to be heterogeneous, uncertain, and time-varying. Each agent uses an estimate of communication delay to provide feedback of estimated recent tracking error. Simulation results are provided to demonstrate the improved performance of the developed controller over other popular control designs.

The risk-aware path planning problem is considered, which aims to locate a target in a congested urban environment and facilitate aid in the decision making on target interdiction. The target is modeled as a ground vehicle moving randomly within a road network and following traffic rules. To locate the target, a heterogeneous sensor network composed of passive sensors (e.g., Static traffic cameras and mobile human observers) and active sensors (e.g., A UAV) is tasked to cooperatively search for the target. A sample-based Bayesian filter is developed to fuse various sensor measurements to estimate the target state. To facilitate the decision making on target interdiction, a notion of risk is considered, which evaluates the incurred loss of target interdiction at certain locations based on incomplete information of target state and urban factors (e.g., The proximity to critical areas such as populated shopping malls, schools, military, or government buildings). As opposed to the static traffic cameras and the randomly walking human observers that passively provide target measurements, the UAV actively plans its path, based on mutual information, to maximize the in formativeness of future measurements. In contrast to classical target tracking that only focuses on reducing the uncertainty of target state, the risk is encoded in the particle weights to guide the motion of UAV to improve target state estimation and, ultimately, reduce the risk of decision on target interdiction. Simulation results are provided to demonstrate the integrated sensing framework and the risk-aware path planning algorithm.

This paper considers stabilizing a nonlinear system with input constraints using switched control. The first result is that, for a given control Lyapunov function (CLF), a stabilizable set of initial conditions can be found. The paper then considers switching between multiple CLFs to increase the space of stabilizing inputs which also increases the set of stabilizable initial conditions. An algorithm is presented that identifies a subset of the known CLFs that may be used at the current time to produce stabilizing control inputs. While there are existing results for switched system stabilization using CLFs, stabilization for systems with input constraints is an original contribution. Sum of squares (SOS) methods are discussed for generating multiple CLFs, and examples are provided.

In this paper, context-aware communication that stabilizes a class of nonlinear networked control systems (NCS) operating over a bandwidth-limited shared-channel is proposed to reduce network traffic. The NCS under consideration includes feedback and feed forward communication channels, and is assumed to be perturbed by an unmodeled, nonlinear disturbance. Informational value of the output states can be analyzed in the context of system stability using model-based approaches. When fused with event-based triggering, these context-aware communication policies result in a periodic feedback and feed forward signals that guarantee uniformly ultimately bounded tracking of output states of an uncertain perturbed system along the desired time-varying trajectory. The proposed NCS is validated using extensive simulation results for nonlinear coupled MIMO systems.

A human-RRT (Rapidly-exploring Random Tree) collaborative algorithm is presented for path planning in urban environments. The well-known RRT algorithm is modified for efficient planning in cluttered, yet structured urban environments. To engage the expert human knowledge in dynamic replanning of autonomous vehicles, a graphical user interface is developed that enables interaction with the automated RRT planner in real-time. The interface can be used to invoke standard planning attributes such as way areas, space constrains, and waypoints. In addition, the human can draw desired trajectories using the touch interface for the RRT planner to follow. Based on new information and evidence collected by human, state-dependent risk or penalty to grow paths based on an objective function can also be specified using the interface.