Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.

Mild cognitive impairment (MCI) is an early stage of cognitive decline that significantly increases the risk of dementia, making early interventions crucial for maintaining cognitive health in older adults. Our research takes a co-design approach to understand, design, and evaluate how socially assistive robots and a virtual human can promote lifestyle changes for people with MCI (pwMCI), potentially improving their cognitive health. Through an iterative refinement process, we aim to develop adaptable, user-friendly, and sustainable technologies that foster long-term engagement across diverse settings. Ultimately, our goal is to improve cognitive health, quality of life, emotional well-being, and loneliness in pwMCI.

Sign language is an essential communication tool, however, it can be highly challenging for non-deaf students to learn. We propose a Pepper robot based sign language teaching assistant called SignPepper, with the ability to communicate in both spoken and sign language. Using Whisper speech to text and Llama 3.3, SignPepper can engage in two way spoken lessons, with the ability to physically demonstrate signs to students. Furthermore, using 3D convolutional neural networks trained on sign language recognition, SignPepper can watch, analyze and give personalized feedback on students attempts at performing newly learned signs in real-time; including hand-based error localization.

Monocular depth estimation is a rudimentary problem for robotic perception systems and downstream applications. However, depth estimation from a single image is an inherently ill-posed problem due to data loss related to projection from 3D to 2D. Recent studies address the discrepancy between camera parameters by using learning-based methods and unifying the camera model to canonical camera space or bipolar representations, thus addressing the problem of training a metric depth model over different datasets with different camera parameters. In addition, the previous study, OrchardDepth, introduced the sparse-dense depth consistency loss function to learn the dense depth distribution through the city autonomous driving scene to improve model performance in the orchard. Instead of enforcing strict consistency between the sparse and dense depth, this work introduced the KL divergence to encourage the network to adapt to the depth distributions of different sensors and penalize deviations from reliable regions while tolerating errors in unreliable areas. Furthermore, we further enhance the depth consistency loss by integrating bins into the supervised discretised depth distribution. This method significantly improves the robustness and performance of our previous method. In addition, it improves the absolute relative error in the orchard dataset by 17.3% and 16.2% in contrast to SILog Loss and OrchardDepth baseline, respectively. Thus enhancing the new training paradigm for depth estimation in the orchard scene.

This paper presents a two-stage facial expression mimicking system using the EveR-4 H22 robot, designed to improve Human-Robot Interaction (HRI) by accurately replicating human emotions. The system follows a two-step process: blendshape extraction, followed by optimized mapping functions that translate human expressions into the robot’s parameters. The parameter training employed Gradient Descent with Regularization, using the Adam Optimizer for 1 million iterations on the custom-labeled data with different emotional categories such as Sad, Fearful, Anger and more. Experimental results show improvements in emotional accuracy, with significant training outcomes that reduced Regularized L2 loss by 1,182 times, capable of accurately mimicking unseen facial emotional expression of an unseen individual. It holds potential applications in domains such as healthcare and customer service, as well as automating generation of demographically broader spectrum of emotional expressions.

Isolated sign language recognition is a challenging task involving the learning of complex relationships between spatial and temporal features. Due to the high complexity and relatively small datasets available, state-of-the-art methods often adopt language modeling and convolutional neural network based multimodal designs, achieving high accuracy at the cost of significant architectural complexity. Conceptually simpler, transformers have gained widespread adoption in related computer vision tasks, outperforming 3D convolutional network competitors. However, due to a lack of training data, video transformers struggle with sign language recognition and have not demonstrated competitive accuracy compared to 3D convolutional neural network designs. We introduce DeepSign, a family of vision transformer based sign language recognition models with superior performance to 3D convolutional neural network designs. Through careful model ablation we select the UniFormerV2 and VideoMAE V2 architectures and perform mixture of dataset pretraining. Our strongest model DeepSign UniFormerV2-L achieves state-of-the-art on the WLASL100 and MSASL100 benchmarks, producing 92.64% and 94% top-1 accuracies respectively. Armed with VideoMAE V2’s powerful pretrained backbone, DeepSign ViT base offers greater efficiency for a small accuracy tradeoff. We hope DeepSign will help advance future sign language research by providing strong foundational models to kickstart experiments.

Sign languages are widely used forms of communication by the deaf and hearing impaired communities. Due to a lack of qualified teachers, robot sign language teaching systems have been proposed, aiming to aid in sign language education. In this paper, we conduct a study utilizing the SignPepper system that we developed; a humanoid Pepper robot based system with capabilities in sign demonstration, verbal communication, and sign language recognition. Specifically, SignPepper adopts a 3D convolutional neural network trained on 100 American sign language signs, Whisper for speech recognition, ChatGPT 4o for context phrasing and Pepper’s built-in text-to-speech functionality. Our study consisted of 33 participants split into two groups, 18 participants were taught sign language by SignPepper whilst the other 15 were shown the same signs as videos. The same sign recognition neural network is used for evaluating the recall accuracy of students in both groups. Survey results showed the SignPepper group had higher sign recall accuracy, greater interest in learning more sign language and greater engagement. However, comfort during the lesson and comfort towards robotic platforms as teaching platforms was lower than the video group. The SignPepper group also rated instruction clarity as slightly lower. Our results indicate that the physical dexterity limits of the Pepper robot platform are a major limitation; as performance of signs may not match human experts exactly. Student comfort is also an area which requires future improvements. Nevertheless, SignPepper demonstrates strong viability for the adoption of robotic sign language teaching systems.

Reinforcement Learning (RL) has been widely used to solve tasks where the environment consistently provides a dense reward value. However, in real-world scenarios, rewards can often be poorly defined or sparse. Auxiliary signals are indispensable for discovering efficient exploration strategies and aiding the learning process. In this work, inspired by intrinsic motivation theory, we postulate that the intrinsic stimuli of novelty and surprise can assist in improving exploration in complex, sparsely rewarded environments. We introduce a novel sample-efficient method able to learn directly from pixels, an image-based extension of TD3 with an autoencoder called NaSA-TD3. The experiments demonstrate that NaSA-TD3 is easy to train and an efficient method for tackling complex continuous-control robotic tasks, both in simulated environments and real-world settings. NaSA-TD3 outperforms existing state-of-the-art RL image-based methods in terms of final performance without requiring pre-trained models or human demonstrations.

Apple fruitlet thinning is critical in cultivating high-quality apples, requiring an expert workforce to manage the orchard. The thinning process requires precise mapping of fruitlet clusters across the tree branches to manage the desired load for each tree. This paper presents Archie Snr, which was developed to autonomously assess the current load of the tree and thin the excess apples as an expert thinner would. The platform has been extensively evaluated in a real-world commercial orchard. The results show the platform can generate an average load count accuracy of 82.1% with a recall of 93.3%. The system was then able to successfully thin 66.14% of the fruitlets from the canopy.

Cane pruning grapevines is a complex manual task requiring expert vine assessment to determine which canes to prune. This paper presents Archie Jnr, which was developed to autonomously assess the structure of the vine and prune the lower-quality canes as an expert pruner would. The platform has been extensively evaluated in a real-world commercial vineyard using a three-cane pruning method. The results show the effectiveness of the vision system for generating accurate assessments of a vine’s canes. The platform is also shown to be capable of successfully pruning 71.1% of the 311 total canes that required pruning across 25 vines.

Multiple modalities can boost accuracy in the difficult task of Sign Language Recognition (SLR), however, each modality does not necessarily contribute the same quality of information. Current multi-modal approaches assign the same importance weightings to each modality, or set weightings based on unproven heuristics. This paper takes a systematic approach to find the optimal weights by performing grid search. Firstly, we create a multi-modal version of the RGB only WLASL100 data with additional hand crop and skeletal pose modalities. Secondly, we create a 3D CNN based weighted multi-modal sign language network (WMSLRnet). Finally, we run various grid searches to find the optimal weightings for each modality. We show that very minor adjustments in the weightings can have major effects on the final SLR accuracy. On WLASL100, we significantly outperform previous networks of similar design, and achieve high accuracy in SLR without highly complex pre-training schemes or extra data.

Model Free Reinforcement Learning (MFRL) has shown significant promise for learning dexterous robotic manipulation tasks, at least in simulation. However, the high number of samples, as well as the long training times, prevent MFRL from scaling to complex real-world tasks. Model- Based Reinforcement Learning (MBRL) emerges as a potential solution that, in theory, can improve the data efficiency of MFRL approaches. This could drastically reduce the training time of MFRL, and increase the application of RL for real- world robotic tasks. This article presents a study on the feasibility of using the state-of-the-art MBRL to improve the training time for two real-world dexterous manipulation tasks. The evaluation is conducted on a real low-cost robot gripper where the predictive model and the control policy are learned from scratch. The results indicate that MBRL is capable of learning accurate models of the world, but does not show clear improvements in learning the control policy in the real world as prior literature suggests should be expected.

Over the last decade, a plethora of soft robotic devices have been proposed for the execution of complex grasping and dexterous manipulation tasks. Tasks requiring such increased dexterity are typically executed using fully-actuated, rigid end-effectors equipped with sophisticated sensing and controlled with complex control laws. The new class of soft robotic devices offers an alternative to the traditional end-effectors and facilitates the development of robotic grasping and manipulation solutions that are lightweight, safe to interact with, affordable, and easy to use and control. Within the class of soft robotic grippers and hands, promising recent developments were made in ultra-affordable, even disposable mechanisms based on origami and kirigami structures. This paper proposes a new kirigami-inspired robotic gripper geometry employing compression-based actuation. The compression actuation fundamentally differentiates this new design class from previous kirigami grippers, resulting in more compact robotic grippers with superior grasping capabilities. In particular, we investigate how the shapes of the internal cuts of the kirigami geometries can affect the gripper performance in terms of force exertion and grasping capabilities. A series of experiments are conducted to understand better the working principles behind this new type of kirigami grippers and experimentally validate their efficacy in the execution of complex, everyday life tasks. Further demonstrations of the gripper's capabilities include the pick-and-placing of human hair, egg yolk, and even liquids.

Soft robotic devices have been popular in handling intricate grasping and dexterous manipulation tasks, serving as an alternative to conventional, rigid end-effectors. These devices are relatively simple, lightweight, and cost-effective. Recently, kirigami based structures have been used to create low-cost and disposable soft robotic grippers and hands. These grippers undergo a complex post-contact reconfiguration and conform to an object's shape and size during grasping. In this paper, we explore this new class of soft robotic grippers by utilising them for single-grasp object classification and grasping force estimation. We install simplistic sensors on both the gripper and the actuation system to estimate the state of the kirigami gripper, and the collected data features are employed to train Random Forest models for identifying the grasped object. The classifier trained exhibits a high accuracy of 98 % in discriminating objects of various shapes. When handling food items, the classifier achieves an accuracy of 94 %, while in classifying transparent objects, the classifier obtained again a high accuracy of 97 %. Finally, object-specific force estimation models are triggered based on the classification decision of the Random Forest model to estimate the grasping force exerted by the gripper. These positive outcomes demonstrate the kirigami based robotic gripper's potential for object classification in a variety of circumstances, particularly where vision systems are not available or not reliable.

Aotearoa New Zealand has a strong and growing apple industry but struggles to access workers to complete skilled, seasonal tasks such as thinning. To ensure effective thinning and make informed decisions on a per-tree basis, it is crucial to accurately measure the crop load of individual apple trees. However, this task poses challenges due to the dense foliage that hides the fruitlets within the tree structure. In this paper, we introduce the vision system of an automated apple fruitlet thinning robot, developed to tackle the labor shortage issue. This paper presents the initial design, implementation, and evaluation specifics of the system. The platform straddles the 3.4 m tall 2D apple canopy structures to create an accurate map of the fruitlets on each tree. We show that this platform can measure the fruitlet load on an apple tree by scanning through both sides of the branch. The requirement of an overarching platform was justified since two-sided scans had a higher counting accuracy of 81.17% than one-sided scans at 73.7%. The system was also demonstrated to produce size estimates within 5.9% RMSE of their true size.

Social robots are becoming increasingly present in our daily lives and will continue to be integrated into society to help people with their daily routines. In this paper, we create a general motion dictionary for the NAO robot, to generate emotional gestures when the robot is interacting with humans. We implemented the motions in the context of a museum setting, wherein NAO interacts with visitors as a guide. We present a Motion Dictionary which integrates each gesture’s features and the corresponding emotions. By using the Choregraphe simulator to create the motions and validate them with a real robot, we intend to simplify and help with the generation of emotional gestures for human-robot interaction.

A popular solution for connecting different components in modern electronics, such as mobile phones, laptops, tablets, etc, is the use of flexible flat cables (FFC). Typically, it takes hours of repetition from a highly trained worker, or a high precision autonomous robot with specialised end effectors to reliably manage the installation of these cables. Human workers are prone to error, and cannot work endlessly without a break, while the robots often come with a significant expense, and require a substantial amount of time to program and reprogram. Additionally, the use of sophisticated sensing elements further increases the complexity of the required control system. As a result, the performance and robustness of such systems is far from sufficient, hindering their mass adoption. The manipulation of FFCs is also quite challenging. In this work, we focus on the robotic manipulation of a plethora of flexible cables, proposing a multi-modal gripper with locally-dexterous tips and active fingernails. The fingers of the gripper are equipped with: i) locally-dexterous fingertips that accommodate manipulation-capable degrees of freedom, ii) a combination of Nitinol-based active fingernails and suction cups that allow picking up and handling of cables that rest on flat surfaces, and iii) compliant finger-pads that conform to the object surface to increase grasping stability. The proposed robotic gripper is equipped with a camera and a perception system that allow for the execution of complex cable manipulation and assembly tasks in dynamic environments.

Grasping and manipulation are complex and demanding tasks, especially when executed in dynamic and unstructured environments. Typically, such tasks are executed by rigid articulated end-effectors, with a plethora of actuators that need sophisticated sensing and complex control laws to execute them efficiently. Soft robotics offers an alternative that allows for simplified execution of these demanding tasks, enabling the creation of robust, efficient, lightweight, and affordable solutions that are easy to control and operate. In this work, we introduce a new class of soft, kirigami-based robotic grippers, we study their post-contact behavior, and we investigate different cut patterns for their development. We follow an experimental approach in which several designs are proposed and employed in a series of grasping and force exertion tests to compare their capabilities and post-contact behavior. The results of such experiments indicate a clear relationship between degree of reconfiguration and grasping force, and provide key insights into the effect of the cut patterns in the performance of the designs. These findings are then used in the design process of an improved version of multi-layer, disposable kirigami grippers that are fabricated employing simple 3D printed layers and silicone rubber using the concept of Hybrid Deposition Manufacturing (HDM). A series of experimental results demonstrate that the proposed design and manufacturing methods can enable the creation of soft, kirigami-based grippers with superior grasping capabilities that can handle delicate, contaminated, and everyday life objects and can even be disposed off in an automated way (e.g., after handling hazardous materials, such as medical waste).

Social robots are often critiqued as being too ‘robotic’ and unemotional. For affective human-robot interaction (HRI), robots must detect sentiment and express emotion and empathy in return. We explored the extent to which people can detect emotions, empathy and sentiment from speech expressed by a computer system, with a focus on changes in prosody (pitch, tone, volume) and how people identify sentiment from written text, compared to a sentiment analyzer. 89 participants identified empathy, emotion and sentiment from audio and text embedded in a survey. Empathy and sentiment were best expressed in the audio, while emotions were the most difficult detect (75%, 67% and 42% respectively). We found moderate agreement (70%) between the sentiment identified by the participants and the analyzer. There is potential for computer systems to express affect by using changes in prosody, as well as analyzing text to identify sentiment. This may help to further develop affective capabilities and appropriate responses in social robots, in order to avoid ‘robotic’ interactions. Future research should explore how to better express negative sentiment and emotions, while leveraging multi-modal approaches to HRI.

Participatory design includes stakeholders in the development of products intended to solve real-life challenges. Involving end users in the design of robots is vital for developing effective, useful, acceptable and user-friendly products that meet expectations, needs, and preferences. This four-year international project developed and evaluated a home-based robot for mood stabilization and cognitive improvement in older adults with mild cognitive impairment and age-related health needs. The daily-care robot was developed in collaboration with experts, carers, relatives, and older adults, through six phases. Two phases were dedicated to cognitive stimulation games. This paper provides a summary of the participatory design and mixed-methods evaluation processes undertaken to develop, refine, and test the robot. The final robot and games were acceptable to older adults, and useful for delivering stimulating activities and providing reminders for medication, health and wellbeing checks. Personalization is required to optimize human-robot interaction, and imagery and speech should be consistent with local users. Functions should be personalizable to accommodate individual health needs and preferences. This project highlights the importance of participatory design and testing robotics in end-user environments, as technical issues associated with long-term use were uncovered. Recommendations for future development and the design of assistive health robots are made.

Flexible flat cables (FFC) are very popular for connecting different components in modern electronics (e.g., mobile phones, laptops, tablets, etc.). The manipulation of FFCs typically relies on highly trained workers that spend hours performing the same repetitive processes, or on autonomous robotic systems that are equipped with simple clamping mechanisms or pneumatically driven suction cups. Such robotic systems are difficult to program and reprogram and often rely on sophisticated sensing elements and complicated control laws. Moreover, the performance and robustness of such systems is far from sufficient, hindering their mass adoption. The manipulation of FFCs is also quite challenging. A good gripper should be able to pinch the cable steadily and execute insertion tasks of the cable connector with ease. The suction cup based solution is a good approach for holding the cable, but it makes the cable connector insertion very challenging as it can only apply limited shear forces. In this paper, we propose a locally-adaptive, pneumatic, parallel-jaw robot gripper equipped with fingertips that are able to both pinch the cable with a soft clamping mechanism and roll the cable surface on the soft fingertip structure until it reaches the desired connector. The gripper base accommodates a camera that allows for the recognition and pose estimation of the flat, flexible cables and other electronic components. The gripper is of low-cost and low-complexity and it can facilitate the efficient and robust execution of FFC grasping and assembly tasks.

Robot grasping and dexterous, in-hand manipulation allow robots to interact with their surroundings and execute a plethora of complex tasks such as pushing buttons, opening doors, and interacting with electrical appliances. In robotics, such complicated tasks are typically executed by multi-fingered end-effectors that are heavy, rigid, and expensive, employing numerous degrees of freedom and actuation. In this paper, we focus on the analysis, design, and development of a multi-grasp, reconfigurable, five fingered, anthropomorphic robot hand that can facilitate the execution of both robust grasping and dexterous manipulation tasks in service robotics and industrial automation applications. The robot hand is composed of eight actuators driving eighteen degrees of freedom with a telescoping mechanism and opposable thumb and pinky fingers to produce multiple anthropomorphic and non-anthropomorphic configurations for grasping and manipulation tasks. The reconfigurable finger base frames allow the hand to transform and utilize its degrees of actuation in an optimal manner to overcome its underactuated limitations. The underactuated robot hand is designed with a human hand structure that takes advantage of objects specifically designed for human operation (e.g., tool or handles with ergonomics for the human hand). This allows the system to better operate within a human-centered environment. The effectiveness of the proposed device is experimentally validated through three different tests: i) grasping experiments involving everyday-life objects, ii) force experiments that assess the force exertion capabilities of the hand in different finger base frame configurations, and iii) demonstration of in-hand object manipulation capabilities. The proposed hand weighs 1.28 kg and has a cost of approximately $1920 USD. The device is capable of exerting up to 14.3 N of contact force during pinch grasping and a maximum of 150.6 N power grasping.

Heterogeneous Multiple Robots(HMR) can be used in daily life for smart homes and industry. The differences in implementing different HMR can be minimized using middle-ware like Robot Operating System (ROS). However, the ROS topics, nodes, and message formats to subscribe and publish can differ from one robot to another. When a user expresses high-level instructions through the Web interface, all multiple robots must understand instructions uniformly and take the actions accordingly without considering each robot's internal software and hardware implementation. This paper represents an optimized ontology-based algorithm for HMR registration and control for high-level instructions. Autonomous robot registration was achieved using an ontology-based optimized algorithm. User-level high-level instructions are processed using an ontology-based algorithm to determine the corresponding actions for each robot. Finally, autonomous publication and subscription to different ROS topics were implemented using another optimized algorithm. The evaluation of the proposed algorithms was completed with Turtlebot, Husky and TiaGo robots using gazebo.

Cognitive stimulation games delivered on robots may be able to improve cognitive functioning and delay decline in older adults. However, little is known about older adults’ in-depth opinions of robot-delivered games, as current research primarily focuses on technical development and one-off use. This article explores the usability, acceptability, and perceptions of community-dwelling older adults towards cognitive games delivered on a robot that incorporated movable interactive blocks. Semi-structured interviews were conducted with participants at the end of a 12-week cognitive stimulation games intervention delivered entirely on robots. Participants were 10 older adults purposively sampled from two retirement villages. A framework analysis approach was used to code data to predefined themes related to technology acceptance (perceived benefits, satisfaction, and preference), and usability (effectiveness, efficiency, and satisfaction). Results indicated that cognitive games delivered on a robot may be a valuable addition to existing cognitive stimulation activities. The robot was considered easy to use and useful in improving cognitive functioning. Future developments should incorporate interactive gaming tools, the use of social anthropomorphic robots, contrasting colour schemes to accommodate macular degeneration, and cultural-specific imagery and language. This will help cater to the preferences and age-related health needs of older adults, to ultimately enhance usability and acceptability.

Task-oriented dialogue system has a vital role in Human-Robot Interaction (HRI). However, it has been developed based on conventional pipeline approach which has several drawbacks; expensive, time-consuming, and so on. Based on this approach, developers manually define a robot's behaviour such as gestures and facial expressions on the corresponding dialogue states. Recently, end-to-end learning of Recurrent Neural Networks (RNNs) is an attractive solution for the dialogue system. In this paper, we proposed a social robot system using end-to-end dialogue system in the context of hospital receptionist. We utilized Hybrid Code Network (HCN) as an end-to-end dialogue system and extended to select both response and gesture using RNN based gesture selector. We evaluate its performance with human users and compare the results with one of the conventional methods. Empirical result shows that the proposed method has benefits in terms of dialogue efficiency, which indicates how efficient users were in performing the given tasks with the help of the robot. Moreover, we achieved the same performance regarding the robot's gesture with the proposed method compared to manually defined gestures.

With the emergence of agile manufacturing in highly automated industrial environments, the demand for efficient robot adaptation to dynamic task requirements is increasing. For assembly tasks in particular, classic robot programming methods tend to be rather time intensive. Thus, effectively responding to rapid production changes requires faster and more intuitive robot teaching approaches. This work focuses on combining programming by demonstration with path optimization and local replanning methods to allow for fast and intuitive programming of assembly tasks that requires minimal user expertise. Two demonstration approaches have been developed and integrated in the framework, one that relies on human to robot motion mapping (teleoperation based approach) and a kinesthetic teaching method. The two approaches have been compared with the classic, pendant based teaching. The framework optimizes the demonstrated robot trajectories with respect to the detected obstacle space and the provided task specifications and goals. The framework has also been designed to employ a local replanning scheme that adjusts the optimized robot path based on online feedback from the camera-based perception system, ensuring collision-free navigation and the execution of critical assembly motions. The efficiency of the methods has been validated through a series of experiments involving the execution of assembly tasks. Extensive comparisons of the different demonstration methods have been performed and the approaches have been evaluated in terms of teaching time, ease of use, and path length.

A robot cannot be warm and friendly - or can it? To explore whether a robot can be a medical receptionist, we developed a robotic system for interacting with patients at a doctor's clinic, including acting friendly. We designed the robot to interact naturally with patients at the start and finish of a clinic visit. We investigated people's perceptions to the robot in a wizard-of-Oz study, where the participants interacted with the robot over four interactions. 40 participants evaluated the robot. The results indicate the participants thought the robot could be a friendly receptionist, especially after repeated interactions with the robot. However, the participants mainly thought the robot was friendly in a “professional” way, rather than a personal friend.

This paper presents a new version of robot receptionist system for healthcare facility environment. Our HealthBots consists of three subsystems: a receptionist robot system, a nurse assistant robot system, and a medical server. Our first version of receptionist robot, interacts with human at hospital reception, gives instructions to human verbally, but cannot understand what human says, so it uses a touch screen to get the response from human. In this paper, we design a receptionist robot that recognizes human face as well as speech, which enhances verbal interaction skill of robot. In addition, we design a reaction generation engine to generate appropriate reactive motions and speech. Moreover, we study which social skills are important to a hospital receptionist robot to enhance social interaction, such as friendliness and attention. We implemented perception modules, decision-making modules, and reaction modules to our HealthBots architecture, and did two case studies to find essential social skills for hospital receptionist robots.

The motivation of the work presented here is to create a component of a safety system based on 3D lidar sensors, specifically for industrial environments where some rules can be set for people who will be in close proximity to working robots. Specifically, the operating procedure that is put in place in the workplace is that all people must wear the provided high visibility clothing, which has retro-reflective strips attached. It is shown here that the retro-reflective strips provide a strong cue for pedestrian detection in the intensity data from a lidar sensor within a range of 4 metres. We present and compare multiple methods of exploiting this cue and provide a recommendation for how a safety system should be architected in order to best exploit the lidar intensity data in combination with more common approaches for detection of objects from the lidar range data. Amongst these detection methods is the use of neural networks, which present challenges for key components of standardized safety system development-in particular, for programming methodology control, interpretability of testing and diagnostic coverage. We propose methods for how to start to address these challenges and how to integrate neural networks into safety systems.

Artificial speech developed using speech synthesizers has been used as the voice for robots in Human Robot Interaction (HRI). As humans anthropomorphize robots, an empathetically interacting robot is expected to increase the level of acceptance of social robots. Here, a human perception experiment evaluates whether human subjects perceive empathy in robot speech. For this experiment, empathy is expressed only by adding appropriate emotions to the words in speech. Also, humans' preferences for a robot interacting with empathetic speech versus a standard robotic voice are also assessed. The results show that humans are able to perceive empathy and emotions in robot speech, and prefer it over the standard robotic voice. It is important for the emotions in empathetic speech to be consistent with the language content of what is being said, and with the human users' emotional state. Analyzing emotions in empathetic speech using valence-arousal model has revealed the importance of secondary emotions in developing empathetically speaking social robots.

Mobile service robots have the potential to improve the efficiency of fruit production in orchards. One of the key tasks that such robots must perform is traversing the rows. Many of the past implementations of row following in orchards have been developed for rows where the trees appear like walls on both sides. Another orchard structure that is used is the pergola, where a sparse array of trunks and posts hold up a canopy, which resembles a ceiling. Navigation in pergola structured environments has received less attention. The variations in the pergola environment- including the presence of tall weeds, hanging branches, undulating terrain and varying geometry-make following the rows a challenging problem. This paper presents solutions for finding the row centreline in pergola structured environments, in the presence of real world variability. A 3D laser scanner is used to measure the positions of posts and trunks, amongst the other features in the pergola. From the extracted features, the mode gradient of nearest neighbours is used to find the row direction and hence the centreline. The practicality of the system is demonstrated by autonomously driving a mobile robot through over 5000 meters of a kiwifruit orchard with a pergola structure, using the row detection method. This method performs favourably compared to an existing method of row detection in kiwifruit orchards.

Safety and acceptability are critical issues when people are interacting with robots in healthcare. Traditional robot arms are hard and inflexible, and may cause harm to users on impact. Soft robotic arms may be safer and more acceptable in these situations. Similarly robot fingers made of soft materials may be more acceptable and safer than fingers made of hard materials. Robot designers need to know how best to design arms for healthcare scenarios. There is limited research on the acceptability of soft robotic arms and fingers to date. This study aimed to investigate people's reactions to the touch of soft robotic arms and fingers, compared to more traditional hard forms, and to human arms and fingers. A second aim was to investigate people's perceptions of the usefulness of the arms and fingers for healthcare tasks. Thirty five community participants were blindfolded and participated in touching tasks for: 3 arms (soft robot, hard robot, and human) and four fingers (soft robot, medium robot, hard robot, and human) in a randomised order. The soft arm was rated significantly more human-like but also more fragile and less reliable than the hard arm. Participants perceived the soft arm as good for intimate tasks like washing the body, but the hard arm was perceived as better for weight-bearing tasks. The soft finger was rated significantly more creepy, fragile and unreliable than the other fingers. The medium robot finger was rated the most human-like of the robot fingers and was the favourite robot finger. These findings suggest people perceive soft robots to be more fragile than hard robots and as more appropriate for personal tasks. Overly soft fingers may be too creepy to be acceptable.

This paper presents a robot system for healthcare facility environments. Current healthcare robot systems do not address healthcare workflows well and our goal is to provide distributed, heterogeneous multiple robot systems that are capable of integrating with healthcare workflows and are easy to modify when workflow requirements change. The proposed system consists of three subsystems: a receptionist robot system, a nurse assistant robot system, and a medical server. The roles of the receptionist robot and the nurse assistant robot are to do tasks to help the human receptionist and nurse. The healthcare robots upload and download patient information through the medical server and provide data summaries to human care givers via a web interface. We developed these healthcare robot systems based on our new robotic software framework, which is designed to easily integrate different programming frameworks, and minimize the impact of framework differences and new versions. We test the functionalities of each healthcare robot system, evaluate the robot-robot collaboration, and present a case study.

Robots have been proposed to reduce the costs of the provision of healthcare in rural settings, but as yet little research has tested this. This study investigated the feasibility and cost-effectiveness of a robot measuring routine vital signs in a family medicine clinic in a rural setting. The length of patient consultations was compared before (N = 85 patients) and after a robot was deployed in the clinic (N = 48 patients). A Cafero touchscreen robot took the patient's vital signs prior to the consultation and transferred the results to the medical professional's computer. Time-savings were calculated in New Zealand dollar terms and compared to the costs of the robot and its maintenance. Results showed that consultation lengths were cut by 18% on average (3 minutes and 13 seconds). If 20% of the clinics' annual consultations were augmented with the robot this translates to a total annual savings of NZ$19075. The annual cost of the robot was calculated to be NZ$9400 overs 5 years. Present value calculations of Benefit Cost result in a Benefit Cost ratio of 2.3. These results support the cost-effectiveness of the robot in a rural medical clinic. Further research is needed to improve the services provided by the robot and test it in a larger trial.

Whilst robots are increasingly being deployed as social agents, it is still difficult to program them to interact socially. This is because current programming tools either require programmers to work at a low level or lack features needed to create certain aspects of social interaction. High level, domain specific tools with features designed specifically to meet the requirements of social interaction have the potential to ease the creation of social applications. We present a domain specific application programming interface (API) that is designed to meet the requirements of social interaction. The Cognitive Dimensions Framework was used as a design tool during the design process and the API was validated by implementing an exemplar application. The evaluation of the API showed that programmers with no robotics knowledge were positively impressed by the notation and that its organization, domain specific interfaces and object oriented nature positively affected several Cognitive Dimensions.

We describe an engaging interactive robot and the workflow design for incorporating such service robots in health care. The research is analyzing the long term usability of automated medication support for older people as they interact with a Stationary Robotic Medication Management System (StRoMMS). It delivers timely instructions and automated guidance as people take their daily medications in their independent living quarters in a retirement village. A pilot user study evaluated the hypothesized technological requirements of the robotic system and the clinical workflow requirements in the healthcare context. The novel contributions are our interactive robot and the workflow design. Following a “system of systems” design approach we determined that robots cannot work in isolation in a complex operational space such as healthcare. The value of introducing interactive robots in healthcare can be realized when the robot has interfaces with the healthcare system, which enhance the overall outcome and experience of the patient. Our research will inform the research community of the importance of the confluence of people, workflows and tools while designing healthcare robotics technology.

Service robots for personal and domestic use are increasingly gaining momentum. Easy and efficient programming of such robots is an enormous research and commercial space that is beginning to be explored. In this paper, we present RoboStudio, a Visual Programming Environment (VPE) to program the interactive behavior of personal service robots. RoboStudio lies at the intersection of VPEs which aid in authoring the robot user interface and control logic. A novel contribution of this work is that it advances the research in authoring service applications on robotic platforms, specially for researchers who do development in decentralized multidisciplinary teams and validate their research goals through field trials. Furthermore, service robot programming environments is a novel area of research, particularly when it comes to expressing what the robot does in a declarative syntax.

Many programming by demonstration methods encode demonstrations into sequences of predefined symbols and then build a generalized task structure such as a topological graph. The longest common subsequence (LCS) algorithm is one of the potential techniques to help build generalized task structures from multiple sequences. However the LCS problem is NP hard, so a couple of suboptimal LCS approaches have been adopted in the past, involving a pair-wise comparison of sequences or a search for the common symbols within a small window. This paper argues that an LCS of multiple sequences results in a better generalization than pairwise comparison, and in many practical situations it is feasible to find an LCS of multiple sequences. So a novel LCS finding algorithm is presented for applications in the programming by demonstration domain. The algorithm has been extensively tested for sequences of random symbols and its application in a path planning example is presented.

Robots are useful tools for teaching novices programming as real and immediate outcomes of programs can be seen. However robot software development has unique problems making aspects of programming difficult compared with general software development. These problems include the robot platform, the robot's environment and its interaction in three-dimensional space and the fact that events occur in real time. We describe Ruru, a novel visual language that addresses these difficulties through a principled approach to its design. It also visualizes robot inputs intuitively in real time and allows the intuitive amendment of parameters. This improves its usefulness and user friendliness as a tool for teaching novices programming.

The expression of reactive behaviour is a significant and important requirement in robotic software engineering, since robots must cope with a wide range of unpredictable events and environments. However it is important that the semantics for reactive expression can be used across different architectures and languages. The RADAR robot programming language provides architecture- and language-independent semantics for managing the reactive parts of robot software together with the deliberative parts, allowing greater interaction between the two. We evaluate the architecture-independence of RADAR, as an example, by implementing its reactive semantics using the OpenRTM-aist component-based, distributed architecture. Our goal is to evaluate what limitations the choice of implementation environment may place on the capabilities of such an architecture-independent semantics. In our implementation, we aimed to produce a standard OpenRTM-aist system using the RADAR semantics. We have found that the architecture-independent semantics concept works well in the case of RADAR, although some specific improvements are needed for full interaction between deliberative and reactive sections of robotic software.

This paper presents the first version of a mobile service robot designed for older people. Six service application modules were developed with the key objective being successful interaction between the robot and the older people. A series of trials were conducted in an independent living facility at a retirement village, with the participation of 32 residents and 21 staff. In this paper, challenges of deploying the robot and lessons learned are discussed. Results show that the robot could successfully interact with people and gain their acceptance.

This study investigated whether attitudes and emotions towards robots predicted acceptance of a healthcare robot in a retirement village population. Residents (n = 32) and staff (n = 21) at a retirement village interacted with a robot for approximately 30 minutes. Prior to meeting the robot, participants had their heart rate and blood pressure measured. The robot greeted the participants, assisted them in taking their vital signs, performed a hydration reminder, told a joke, played a music video, and asked some questions about falls and medication management. Participants were given two questionnaires; one before and one after interacting with the robot. Measures included in both questionnaires were the Robot Attitude Scale (RAS) and the Positive and Negative Affect Schedule (PANAS). After using the robot, participants rated the overall quality of the robot interaction. Both residents and staff reported more favourable attitudes (p <; .05) and decreases in negative affect (p <; .05) towards the robot after meeting it, compared with before meeting it. Pre-interaction emotions and robot attitudes, combined with post-interaction changes in emotions and robot attitudes, were highly predictive of participants' robot evaluations (R = .88, p <; .05). The results suggest both pre-interaction emotions and attitudes towards robots, as well as experience with the robot, are important areas to monitor and address in influencing acceptance of healthcare robots in retirement village residents and staff. The results support an active cognition model that incorporates a feedback loop based on re-evaluation after experience.

Mobile robots are increasingly entering the real and complex world of humans in ways that necessitate a high degree of interaction and cooperation between human and robot. Complex simulation models, expensive hardware setup, and a highly controlled environment are often required during various stages of robot development. There is a need for robot developers to have a more flexible approach for conducting experiments and to obtain a better understanding of how robots perceive the world. Mixed Reality (MR) presents a world where real and virtual elements co-exist. By merging the real and the virtual in the creation of an MR simulation environment, more insight into the robot behaviour can be gained, e.g. internal robot information can be visualised, and cheaper and safer testing scenarios can be created by making interactions between physical and virtual objects possible. Robot developers are free to introduce virtual objects in an MR simulation environment for evaluating their systems and obtain a coherent display of visual feedback and realistic simulation results. We illustrate our ideas using an MR simulation tool constructed based on the 3D robot simulator Gazebo.

This paper presents our expressive facial speech synthesis system Eface, for a social or service robot. Eface aims at enabling a robot to deliver information clearly with empathetic speech and an expressive virtual face. The empathetic speech is built on the Festival speech synthesis system and provides robots the capability to speak with different voices and emotions. Two versions of a virtual face have been implemented to display the robot's expressions. One with just over 100 polygons has a lower hardware requirement but looks less natural. The other has over 1000 polygons; it looks realistic, but costs more CPU resource and requires better video hardware. The whole system is incorporated into the popular open source robot interface Player, which makes client programs easy to write and debug. Also, it is convenient to use the same system with different robot platforms. We have implemented this system on a physical robot and tested it with a robotic nurse assistant scenario.

A novel trajectory segmentation and modeling approach is presented. Trajectory segmentation and matching is an important step in the programming by demonstration (PbD) process to extract the user's intentions from multiple trajectories. To match multiple trajectories, the segmentation and modeling approach must be consistent and robust to disparities caused by robot dynamics and human imperfections. Several curve segmentation approaches have demonstrated substantial potential in the field of image processing and gesture recognition. They emphasize reduction of the degree of mismatch between given and model curves. However they fail to reduce mismatch between models of multiple trajectories recorded to demonstrate the same intention.We propose an M-estimator for trajectory modeling and set up a new segmentation criterion to address the issue. The proposed approach is better suited for PbD of mobile robots. The approach is evaluated for real robot trajectories.

As the proportion of people in the older age groups grows, demands on care providers increase. The ability of robotic technology to meet these demands is limited by a lack of acceptance by older people. This study investigates which tasks staff and residents in a retirement village would like a robot to assist with, as well as their attitudes towards robots and preferences for their appearance. Findings show that residents are more positive about robots than staff, and participants prefer a silver robot of 1.25 m height, with wheels and a screen on the body. Residents would most like the robot to assist with detecting falls, turning on and off appliances, lifting, cleaning, medication reminding, making phone calls and monitoring location. Making robots that fit these preferences may increase the acceptance of robotic assistants by older people.

Human-robot interaction (HRI) and user acceptance become critical when service robots start to provide a variety of assistance to users on a personal level. Limited research to date has studied the influence of users' attributes (such as age and gender) on the acceptance of service robots and the implications for HRI design. This paper describes the development of a social interactive healthcare robot named Charles, capable of measuring blood pressure. Using blood pressure monitoring as the service scenario, a user study was conducted to investigate the differences between two age groups (40 to 65 years and over 65 years) in attitudes and reactions before and after their interactions with Charles. The results showed few differences between the two age groups. A significant gender effect was found, with males having a more positive attitude toward robots in healthcare than females. This study reveals the importance of considering gender issues in the design of healthcare robots for older people. Overall, the performance of the robot was rated high, however the participants expressed desires to have more interactiveness and a better voice from the robot. According to our sample, age need not be a barrier to users' acceptance of healthcare robots.

A significant challenge in programming robots by demonstration is to accurately capture the user's intentions, so that sensor differences can be managed during playback. Sensor difference can be caused by: natural sensory data variations, minor variations in the task conditions, significant changes in the task scenario, or because the task requires a new set of actions to be executed. This paper presents a design for a programming by demonstration system that focuses on the important goal of capturing the intentions of the user during the demonstration. A gesture interface for a large touch screen is used during demonstration, to capture more clearly the user's intentions for robot movements, and also during a pre- playback session to capture the user's intentions regarding sensor data.

A key part of programming a robotic system is specifying the responses to events that the robot may encounter. This is provided by a new language, RADAR. This paper proposes evaluating robot programming systems work by: a formalisation of the semantics, an evaluation in terms of criteria that determine a languagepsilas suitability for programming, and a small user study to test the readability of programs written using the semantics. The evaluation of the reactivity semantics found in the RADAR language shows clear benefits for programmers.

There has been little previous research assessing people’s emotions and cognitions in response to different types of robot behaviour. This study investigated how people think and feel during interactions with robots who behave either well or poorly. 45 participants interacted with a B21r robot in a basic task to lead the robot along a marked path. Each participant was randomly assigned to either the robot following well or the robot following poorly. The most frequently reported emotions were frustration, fear, and happiness, and people commonly reported thoughts about the robot and also about themselves in the interaction. People reported more positive emotions in response to the good robot. Independently of group assignment, positive emotions during the task were associated with more positive evaluations of the robot and negative emotions were associated with more negative evaluations. These results suggest that robot designers should seek to maximize people’s positive emotions and minimize negative emotions to maximise the quality of human-robot interactions. Results also suggest that recognising emotions may be difficult for robots, and direct questioning may be an easier strategy.

Robot programmers are faced with the challenging problem of understanding the robot's view of its world, both when creating and when debugging robot software. As a result tools are created as needed in different laboratories for different robots and different applications. We discuss the requirements for effective interaction under these conditions, and propose an augmented reality approach to visualising robot input, output and state information, including geometric data such as laser range scans, temporal data such as the past robot path, conditional data such as possible future robot paths, and statistical data such as localisation distributions. The visualisation techniques must scale appropriately as robot data and complexity increases. Our current progress in developing a robot visualisation toolkit is presented.

One of the greatest challenges when debugging a robot application is understanding what is going wrong. Robots are embodied in a complex, changing and unpredictable real world, using sensors and actuators that are different from humans'. As a result humans may find the development of robotic software to be difficult and time consuming. We present an augmented reality visualisation tool for the popular open source Player system, that enhances the developers understanding of the robots world view and thus improves the robot development process

A key part of programming a robotic system is specifying the responses to events that the robot may encounter. Existing methods of programming responses include event loops, reactive languages and hybrid architectures, none of which meet the specific needs of mobile robot programming. This work presents a design for new semantics for specifying reactivity in mobile robot programs, one that allows for effective specification of reactive behaviour within procedural robot programs. An initial evaluation version is implemented in Python. Events and responses are supported as program objects, and are connected together by new statements. Programmers specify connections between events and responses anywhere within the program code, so connections can easily be changed in response to changes in program and robot state

A distributed real– time robot application framework is developed, to improve the scalability and reusability of software modules. The design is based on real– time CORBA and structured into two layers: the infrastructure layer for basic functionality and the service layer, which includes several reusable services. The implementation of the framework is evaluated from functionality and performance perspectives to demonstrate the feasibility of the design. The required functionality is achieved and performance tests show that the real– time aspect gives dramatic improvements in the performance of high priority tasks under load.

Robot developers need to simulate and visualise robot behaviour in environments where sometimes an accurate dynamics simulation is needed. This paper presents a flexible dynamics architecture that is able to integrate a suitable dynamics engine (such as OpenDE) together with a suitable visualisation tool, to create a dynamics simulation tool most appropriate for a researcher’s specific needs.

Many generic position-estimation algorithms are vulnerable to ambiguity introduced by nonunique landmarks. Also, the available high-dimensional image data is not fully used when these techniques are extended to vision-based localization. This paper presents the landmark matching, triangulation, reconstruction, and comparison (LTRQ global localization algorithm, which is reasonably immune to ambiguous landmark matches. It extracts natural landmarks for the (rough) matching stage before generating the list of possible position estimates through triangulation. Reconstruction and comparison then rank the possible estimates. The LTRC algorithm has been implemented using an interpreted language, onto a robot equipped with a panoramic vision system. Empirical data shows remarkable improvement in accuracy when compared with the established random sample consensus method. LTRC is also robust against inaccurate map data.

Simultaneous localisation and map-building (SLAM) can be considered as a combined state and parameter estimation problem. Instead of using extended Kalman filtering, a more flexible Sequential Monte Carlo method is considered. Multiple generic particle filters are initialised to estimate the robot and obstacle positions concurrently. Simulation results based on a simple robot environment, which represents obstacles by line segments, indicate the feasibility of the proposed method.

In applications such as vacuum cleaning, painting, demining and foraging, a mobile robot must cover an unknown surface. The efficiency and completeness of coverage is improved via the construction of a map of covered regions while the robot covers the surface. Existing methods generally use grid maps, which are susceptible to odometry error and may require considerable memory and computation. This paper proposes a topological map and presents a coverage algorithm in which natural landmarks are added as nodes in a partial map. The completeness of the algorithm is argued. Simulation tests show over 99% of the surface is covered; 85% for real (Khepera) robot tests. The path length is about 10% worse than optimal in simulation tests, and about 20% worse than optimal for the real robot, which are within theoretical upper bounds for approximates solutions to traveling salesman based coverage problems. The proposed algorithm generates shorter paths and covers a wider variety of environments than topological coverage based on Morse decompositions.

A distributed mobile robot software application infrastructure is developed, improving integration and leverage between projects in a research environment. The resulting design includes a three layer CORBA based, service broker application architecture. A reference implementation and tests on B21r, LEGO Mindstorm and Khepera robots demonstrate the feasibility of the design.

A panoramic image based localisation system has been implemented on a B21r mobile robot and tested in a heavily cluttered environment. It takes vertical object edges as natural landmarks and extracts a one-dimensional token sequence from the input image before matching with reference image sequences. Matched points are then triangulated to give a set of position estimates, and the best result is taken as the current robot position. Selection criteria and methods for the reference image sites are also discussed, including the use of generalised Voronoi vertices.

An integrative framework is developed for describing concept learning techniques that makes it possible to evaluate their relevance to knowledge engineering. The framework provides a general basis for relating concept learning to knowledge acquisition and is a starting point for the development of formal design rules. First, concept learning is framed in the context of knowledge acquisition. Then the general forms of input and concept representation such as logic, functions and procedures are discussed. Next, methods of biasing the search for a suitable concept are described and illustrated including: background knowledge, conceptual bias, composition bias, and preference orderings. Finally, modes of teacher interaction are reviewed, including the nature of examples given and the method of presenting them. The framework is illustrated by applying it to the better-documented concept learning systems.>