It remains a formidable challenge to accurately recognize motion intentions of patients thus to control hand exoskeletons according to their volition. Current methods primarily focus on recognition of limited patient’s motion intentions, with the purpose of controlling preconfigured gestures of a hand exoskeleton for grasping objects. These methods exhibit a marked shortfall when encountering scenarios that are unexpected or not designed in advance, such as non-preprogrammed hand movements and object manipulation tasks. To tackle this issue, large language model (LLM) and speech recognition technology are employed in this study to allow the patient to control a hand exoskeleton at will. In particular, two LLMs are tailored to formulate codes of either generating non-preprogrammed gestures or dealing with unencountered objects. Additionally, an incremental learning framework is proposed to enable patients to perform both predefined and non-predefined operation tasks by integrating a natural language parser with the two LLM-based learners. The natural language parser can directly control the hand exoskeleton to perform predefined operations tasks from prestored command set, while the LLM-based learners can incrementally expand the control command set so as to enhance adaptability of the hand exoskeleton to complex activities over daily use. This study is a pioneering work in the field of hand exoskeletons, which will revolutionize the way to control hand exoskeletons. Furthermore, the proposed framework can be easily generalized to any other robots by modifying the prompt of customized LLMs, which provides a new idea to achieve autonomous learning in robotics.Note to Practitioners—The motivation of this article is to tackle the challenge of intention recognition for performing activities of daily living (ADLs) by stroke patients using a multi-degree of freedom hand exoskeleton. Existing methods for intention recognition so far can only be used for several tasks that are predefined in advance, thus none of them allow patients to control the hand exoskeleton completely at will. To surpass this limitation, an LLM-enabled incremental learning framework that integrates a hand exoskeleton controller with Large Language Model (LLM) is proposed and validated in this study. The framework offers patients an intuitive interface via voice interaction and enables patients to perform not only predefined operation tasks by the hand exoskeleton controller but also non-predefined ones that can be learned from the LLM. As a result, the hand exoskeleton controller continues to learn from the LLM, therefore is gradually able to perform all tasks in daily life. This pioneering study paves a new way in building patient-controlled hand exoskeletons with autonomous intelligence that can deal with non-predefined operation tasks in unstructured environments.

Accurate and prompt speed grasp intention recognition is crucial in online human-robot interaction (HRI). However, dynamic grasping relying on complete motion for high recognition accuracy will lead to an unavoidable delay in real-time prediction. To address this issue, we propose a Spatial Gating and Temporal Weighting Early Grasp Prediction (STEGP) method that utilizes incomplete dynamic grasping data from sliding windows to reduce the time delay for reliable robot teleoperation. The proposed method comprises a synergy-based feature extraction module, a spatial gating classification module, and a time-decay weighting fusion prediction module. The spatial-temporal mechanism with gating units effectively classifies sequential movements, achieving performance comparable to that of Transformers but being much easier to implement. Integrating a time-decay weighting frame enables reliable early prediction even with incomplete data. gMLP is chosen for the classification of hand dynamic grasping because of its high accuracy, realizing 93.83% accuracy for 33 grasping categories. The prediction tests demonstrated 85.4% accuracy, with the first 25% grasp completion across 28 subjects. Online robotic teleoperation grasp experiments achieved a 57.4% reduction in time delay and a 93.3% success rate.

It has been clinically proven that exoskeletons are effective self-training rehabilitation or daily living assistance devices for patients with hand dysfunctions. However, exoskeleton-assisted hand exercises with high degrees-of-freedom are considered as challenging tasks because the digit space, especially the thumb, cannot accommodate enough actuators. In this article, we report a tendon-driven soft hand exoskeleton with a hybrid configuration for thumb actuation. The soft hand exoskeleton system uses the least number of actuators to realize full degrees-of-freedom actuation for all digits. It is tested on a stroke patient with hemiplegia and a healthy subject. The experimental results show that the hand exoskeleton could assist the stroke patient to accomplish various training tasks, such as thumb encircling, grasping, pinching, releasing, and writing. It was found that digit trajectories and joint angle changes of the stroke patient were close to those of the healthy subject. Especially, the range of motion of the stroke patient shows significant improvement with the hand exoskeleton assistance compared to that without the hand exoskeleton assistance. The research in this article paves the way to develop fully actuated soft hand exoskeleton that can be eventually integrated with an electroencephalogram or electromyography for self-training rehabilitation or daily living assistance.

The drift distorts the atomic force microscopy (AFM) images as the time taken to acquire a complete AFM image is relatively long (a few minutes). As the AFM image is used as a reference for most manipulation mechanisms, the image distorted by drift will cause problems for AFM-based manipulation because the displayed positions of the objects under nanomanipulation do not match their actual locations. The drift during manipulation, similarly, will further exacerbate the mismatch between the displayed positions and the actual locations. Such mismatch is a major hurdle to achieve automation in AFM-based nanomanipulation. Without proper compensation, manipulation based on a wrong displayed location of the object often fails. In this paper, we present an algorithm to identify and eliminate the drift-induced distortion in the AFM image by applying a strategic local scan method. Briefly, after an AFM image is captured, the entire image is divided into several parts along vertical direction. A quick local scan is performed in each part of the image to measure the drift value in that very part. In this manner, the drift value is calculated in a small local area instead of the global image. Thus, the drift can be more precisely estimated and the actual position of the objects can be more accurately identified. In this paper, we also present the strategy to constantly compensate the drift during manipulation. By applying local scan on a single fixed feature in the AFM image frequently, the most current positions of all objects can be displayed in the augmented reality for real-time visual feedback.

Because of the presence of thermal drift, AFM (atomic force microscopy) images are always contaminated. Such contamination is one of the major hampers to achieve accurate and efficient AFM based nanomanipulation. Based on contaminated images, the manipulation operations often fail. In this paper, we apply a local scan method to identify and compensate the thermal drift contamination in the AFM image. After an AFM image is captured, the entire image is divided into several parts along y direction. A local scan is immediately performed in each part of the image to calculate the drift value at that very part. In this manner, the drift value is calculated in a small local area instead of the global image. Thus, the drift can be more precisely estimated and the image can be more accurately recovered, which lead to improved accuracy for AFM imaging and enhanced productivity for AFM based nanomanipulation.

One of the main roadblocks to Atomic Force Microscope (AFM) based nanomanipulation is lack of real time visual feedback. Although the model based visual feedback can partly solve this problem, its unguaranteed reliability due to the inaccurate models in nano-environment still limits the efficiency of AFM based nanomanipulation. This paper introduce a Real-time Fault Detection and Correction (RFDC) method to improve the reliability of the visual feedback. By utilizing Kalman filter and local scan technologies, the RFDC method not only can real-time detect the fault display caused by the modeling error, but also can on-line correct it without interrupting manipulation. In this way, the visual feedback keeps consistent with the true environment changes during manipulation, which makes several operations being finished without a image scanning in between. The theoretical study and the implementation of the RFDC method are elaborated in this paper. Experiments of manipulating nano-particles have been carried out to demonstrate the effectiveness and efficiency of the proposed method.

Atomic Force Microscope (AFM) has been used as a manipulation tool for a decade. The problem of lacking real time visual feedback still limits its efficiency and hinders its wide application. Although the model based visual feedback can partly solve this problem, due to the complexity of nano environment, it is difficult to use a model to accurately describe the object’s behavior. The modeling error will give the operator a false feedback and lead to a failed manipulation. In this paper, a strategy for visual feedback error on-line detection and correction is proposed to solve this problem. As the real time force information is a key factor for this strategy, an adaptable end effector is employed to accurately measure the interaction force between the probe and the nano-objects, and the system error is also compensated to improve the accuracy of interaction force measurement. Based on the true real time force information, an extended Kalman filter is developed to online detect whether there is a false feedback. Once a false feedback is detected, an optimal searching pattern is generated to get the real manipulation result in a short time. With the assistance of this strategy, the false visual feedback can be realtime detected and corrected without interrupting manipulation. Complex manipulation task can be finished without being interrupted by a new image scan. Experiments of manipulating nano-particles are performed to verify the effectiveness of this strategy, which demonstrated the improved efficiency of the AFM based nano-assembly system.

An atomic force microscope (AFM) explores the topography of a sample surface using a micro-sized flexible cantilever, which works as a flexible robot arm. The flexible cantilever is controlled to keep vibrating when an AFM works in the tapping mode. The cantilever is modeled as a flexible beam instead of a point mass system in this paper. The nonlinear interaction force between the tip and sample surface is also modeled. A simulation environment is developed to analyze the dynamics of cantilevers using the flexible beam model. Simulation results confirm that the flexible beam model can represent the system more accurately than the point-mass model. It has been shown that lower modes are more sensitive to changes of surface topography or surface materials when the cantilever is driven to vibrate at a higher harmonic mode. At the same time, this simulation environment also provides a more accurate way to validate the design of a new AFM probe and AFM controller than simulation packages which use the point-mass model

Nanoassembly using atomic force microscopy (AFM) is a promising technique for nanomanufacturing. Most AFM-based nanoassembly schemes are implemented either manually using haptic devices or in an interactive way between the users and the atomic force microscope images. These schemes are time consuming and inefficient. Therefore, the computer-aided design (CAD)-guided automated nanoassembly using AFM is desirable for nanomanufacturing. In this paper, a general framework for CAD-guided automated nanoassembly using AFM is developed. Based on the CAD model of a nanostructure, the manipulation paths for both nanoparticles and nanorods are generated automatically. A local scanning method is developed to compensate for the random drift that may cause the failure of the nanoassembly. The experimental results demonstrate that the developed general framework can be employed to manufacture nanostructures efficiently. The research work opens a door to the CAD-guided automated nanomanufacturing using AFM. Note to Practitioners-Atomic force microscope (AFM)-based nanoassembly will lead to potential breakthroughs in manufacturing new revolutionary industrial products because many potential nanostructures and nanodevices are asymmetric, which cannot be manufactured using self-assembly only. In order to increase the efficiency and accuracy of AFM-based nanoassembly, automated computer-aided-design (CAD)-guided nanoassembly is desirable to manufacture nanostructures and nanodevices. Based on the CAD model, the environment model and the model of the nanoobjects, collision-free paths are generated to control the AFM tip to manipulate nanoobjects. A local scanning method is developed to obtain the actual position of each nanoobject to compensate for the random drift. Since the building materials of nanostructures and nanodevices may include nanoparticles, nanorods, nanowires, nanotubes, etc., automated path planning algorithms are developed for both nanoparticles and nanorods. The experimental results show that the developed general framework can be used to manufacture nanostructures more efficiently.

Nanomanufacturing using Atomic Force Microcopy has been widely investigated. Most of nanomanipulation schemes go through the scan-design-manipulation-scan cycle manually which is time consuming and inefficient. Automated AFM tip path planning is desirable for nanomanufacturing, but does not receive much attention. In this paper, a CAD guided automated nanomanufacturing system is presented. Based on the CAD model of a nanostructure, the manipulation paths for both nanoparticles and nanorods are generated automatically. A local scanning method is developed to compensate for the random drift that may cause the failure of the nano-assembly. The experimental results demonstrate that the developed algorithm can be employed to manufacture nanostructures efficiently. The research work opens a door to the CAD guided automated nanomanufacturing.

Atomic force microscopy is a powerful and widely used imaging technique that can visualize single molecules both in air and solution. Using the AFM tip as the end effector, it can be modified into a nano-robot which can manipulate objects in nanoscale. By functionalizing the nano-robot end effector with antibodies, atomic force microscopy is able to identify specific types of receptors on cells’ membrane in an image much as fluorescent tags do in optical microscopy but with higher resolution. After the single receptors have been identified, it becomes possible to manipulate these biological macromolecules in their physiological environment. This new technology open a promising way to study the function of biological macromolecules individually.

Nanomanipulation using Atomic Force Microscope (AFM) has been extensively investigated for many years. However, control of tip position during nanomanipulation is still a major issue because of the deformation of the cantilever caused by manipulation force. The softness of the conventional cantilevers also cause the failure of the manipulation of relatively large and sticky nano-object because the tip can easily slip over the nano-object. In this paper, an active atomic force microscopy probe is used to solve these problems by changing the cantilever’s flexibility or rigidity through different control strategies in imaging and manipulation modes respectively. During imaging mode, the active probe is controlled to bend up with respect to the interaction force between the tip and samples, thus making the tip response faster and increase the imaging speed. During manipulation mode, the active probe is controlled to bend down with respect to the interaction force between tip and the samples; thus increasing its nominal rigidity to avoid tip slipping over object. A detailed model of the active probe is presented in this paper and the controller designed based on the proposed active probe model is also implemented on the augmented reality system, which is an AFM based nanomanipulation system with both real-time visual and haptic feedback. The simulation results for the control strategies and the preliminary experimental results for the AFM based nanmomanipulation verified the validity of the model and effectiveness of the controller.

Atomic force microscope (AFM) based nanomanipulation has been extensively investigated for many years. However, the efficiency and accuracy of the AFM based nanomanipulation is still a major issue due to the nonlinearities and uncertainties in nanomanipulation operations. The deformation of the cantilever caused by manipulation force, in our experience, is one of the most major nonlinearities and uncertainties. It causes difficulties in precisely controlling the tip position, which will cause the tip to miss the position of the object. In order to solve this problem, the traditional approach is to use a rigid cantilever. However, this will significantly reduce the sensitivity of the force feeling during the manipulation, which is essential for achieving an efficient and reliable nanomanipulation. An active AFM probe is used to solve this problem by directly controlling the cantilever's flexibility or rigidity during manipulation. An infinite dimensional model of the active probe is developed. Control of the active probe employing an optimal LQR control law is also implemented. The experimental results have verified the theoretical model and demonstrated that the precise position control and high sensitive interaction force measurement can be achieved simultaneously.

Assembly of nano-structures involves manipulation of nanoparticles, nano-rods, nanowires and nanotubes. Modelling the behavior of a nano-rod or a nanotube pushed by an AFM tip is much more complex than that of a nano-particle because in the case of the nano-particle usually only translation occurs while for the nano-rod and nanotube both translational and rotational motion occurs during manipulation. In this work, the behavior of nano-rods under pushing is theoretically analyzed and the interaction among tip, substrate and nano-rods has been modelled. Based on these models, the real-time interactive forces are used to update the AFM image. The real-time visual display combined with the real-time force feedback provides an augmented reality environment in which the operator not only can feel the interaction forces but can also observe the real-time changes of the nano-environment. The new developed augmented reality system capable of manipulating not only nanoparticles but also nano-rods makes nano-assembly using AFM based nanomanipulation system feasible and applicable.

Using AFM as a nanomanipulation tool has been discussed for more than a decade. However, its lack of real-time visual feedback during manipulation has hindered its wide application. Fortunately, this problem has been overcome by our recently developed augmented reality system. Not only can this new system provide real-time force feedback but real-time visual feedback during nanomanipulation. In order to build this augmented reality system, a lot of parameters are used to model the force interaction, measure the forces, and control the tip motion. In this paper, calibration of all parameters used in the augmented reality system is discussed in details.

The development of nanomanufacturing technologies will lead to potential breakthroughs in manufacturing of new industrial products. Nanomanufacturing by manipulating nanoparticles using an atomic force microscope is desirable to manufacture asymmetric nanodevices and nanostructures. The complexity of nanomanufacturing requires positioning, manipulating and assembling nanoparticles. Typical manual nanomanipulation is time-consuming and inefficient. Automated path planning is desirable for nanomanufacturing, but does not receive much attention. In this paper, a general framework is developed to manufacture nanostructures and nanodevices. An automated tool path planning algorithm is presented. Simulations are performed to test the generated paths. The generated paths are also implemented to manipulate nanoparticles to manufacture nanostructures automatically. The simulation and experimental results are consistent. The general framework can also be extended to manipulate other nanoobjects.

DNA electronics circuits require an efficient way to accurately position and individually manipulate DNA molecules. The recent development of atomic force microscopy (AFM) seems to be a promising solution. We have recently developed an AFM based augmented reality system. This new system can provide both real-time force feedback and real-time visual feedback during nanomanipulation. We have shown that nano-imprinting and manipulation of nano-particles and nano-rods can be easily performed under assistance of the augmented reality system. In this research, the system's ability is extended to manipulation of DNA molecules. Using a polynomial fitting method, the deformation of DNA molecules is displayed in real time in the augmented reality system during manipulation. Indeed, DNA molecules adopt many different structures including kinks, bends, bulges and distortions. These different structures and inappropriate physical contacts may result in the controversy of DNA conductivity reported over the last decade. The AFM based nanomanipulation system can be used either as a nanolithography tool to make small gap electrodes or a nanomanipulation tool to elongate, deform and cut DNA molecules. The measurement of the conductivity of DNA molecules in their different shapes and structures is a promising method to find conclusive evidences, which verify the electrical conductivity of DNA molecules.

The use of atomic force microscope (AFM) as a nanomanipulator has been evolving for various kinds of nanomanipulation tasks. Due to the bow effect of the piezo scanner of the AFM, the AFM space is different from the Cartesian space. In this paper, different 3-D nanomanipulation tasks using AFM such as nanolithography, pushing and cutting are discussed. 3-D path planning are performed directly in the AFM space and the 3-D paths are generated based on the 3-D topography information of the surface represented in the AFM space. This approach can avoid the mappings between the AFM space and Cartesian space in planning. By following the generated motion paths, the tip can either follow the topography of the surface or move across the surface by avoiding collision with bumps. Nanomanipulation using this method can be considered as the "true" 3-D operations since the cantilever tip can be controlled to follow any desired 3-D trajectory within the range of AFM space. The experimental study shows the effectiveness of the planning and control scheme.

Previous studies on nanomanipulation using Atomic Force Microscope (AFM) go through the scan-design-manipulation-scan cycle, in which an operator does not have any real-time visual feedback during manipulation. In this paper, a simple model of tip-substrate-object interactive forces has been presented for the qualitative and quantitative analysis. Based on this model, the real-time tip-substrate-object interactive forces are used to update the AFM images in order to provide the operator with real-time visual feedback. The real-time visual display combining with real-time force feedback provides an augmented reality environment, in which the operator not only can feel the real-time 3-D interaction forces but also observe the real-time changes of the nano-environment.

Calibration is indispensable for automatic micromanipulation. In this paper, using the height difference between different focus planes detected from the microscope and the coordinates of the probe measured from encoders, systematic methods to calibrate the relative position and orientation between the tools and the microscope are developed. Using the least square error (LSE) method to process the data measured, the influence of various positioning errors can be reduced. Experiments are performed to calibrate a micromanipulation system using the proposed methods. The calibration results are validated by cross verification through further experiments.