Tactile sensors are indispensable in robotic systems because they deliver vital contact information during environmental interactions. In our work, we leverage the variable compliance of a porous material—where different interaction forces induce varying degrees of compliance—to achieve self-adapting tactile sensing. This distinctive non-linear characteristic allows its sensitivity to be automatically tuned over a range from 0.008 mT/N to 0.045 mT/N. After coating with a magnetic polymer, the porous material functions as a 3-axis magnetic sensing medium. Its length and width are set at 30 mm and 35 mm respectively to accommodate the printed circuit board. To preserve the overall measuring range, it is designed with a thickness of 15 mm. This thickness enables monitoring of the volumetric changes due to the enhanced compliance, which is suitable for three-dimensional shape recognition. In this work, we present the design, fabrication, experimental characterization, and applications of an lightweight 3-axis magnetic sponge sensor with overall dimensions of 30 mm (width) × 35 mm (length) × 17 mm (height) and a detection range of 60 N. Notably, the sensing material weighs only 2 g, thanks to its porous structure.

Traffic accidents, mostly caused by human errors, pose a significant challenge in automotive safety. This paper introduces TUN-DAS, a novel Time-series analysis and Unsupervised learning-based Driving behavior Assessment System, designed to systematically evaluate drivers’ behavior and identify potential risks. Unlike conventional in-cabin driver monitoring systems, which generally rely on rule-based or supervised methods, TUN-DAS utilizes a unique time-series clustering approach. This method combines a k-means algorithm with Dynamic Time Warping (DTW) and DTW Barycenter Averaging (DBA), enabling a more nuanced and comprehensive assessment of driving behavior. The modification of conventional DTW in our system allows for effective computation of temporal differences between driving data. TUN-DAS stands out in its ability to detect both sporadic human errors and consistent unsafe driving patterns by analyzing anomalies within individual drivers and across a broader driver dataset. Our validation with bus driver data shows TUN-DAS’s effectiveness in driving behavior assessment, significantly achieving better accuracy than traditional rule-based evaluation methods and allowing for quantitative and temporal analysis of a series of driving behavior. This system holds promise for enhancing road safety by facilitating the early identification of high-risk drivers, thereby contributing to the development of safer driving environments.

Mixed reality (MR) technologies have a high potential to enhance obstacle negotiation training beyond the capabilities of existing physical systems. Despite such potential, the feasibility of using MR for obstacle negotiation on typical training treadmill systems and its effects on obstacle negotiation performance remains largely unknown. This research bridges this gap by developing an MR obstacle negotiation training system deployed on a treadmill, and implementing two MR systems with a video see-through (VST) and an optical see-through (OST) Head Mounted Displays (HMDs). We investigated the obstacle negotiation performance with virtual and real obstacles. The main outcomes show that the VST MR system significantly changed the parameters of the leading foot in cases of Box obstacle (approximately 22 cm to 30 cm for stepping over 7cm-box), which we believe was mainly attributed to the latency difference between the HMDs. In the condition of OST MR HMD, users tended to not lift their trailing foot for virtual obstacles (approximately 30 cm to 25 cm for stepping over 7cm-box). Our findings indicate that the low-latency visual contact with the world and the user's body is a critical factor for visuo-motor integration to elicit obstacle negotiation.

Handovers play a key role in human-robot interactions. However, current research focuses on visible-hand handovers, thereby heavily relying on hand detection. Large objects in human-robot interactions present a unique challenge: they inherently block the person’s hands and arms from the robot's view. This occlusion raises the robot’s risk of unintended physical contact with the person, leading to discomfort and safety concerns. This study aims to develop a model that can determine a pose for the robot that ensures a handover that avoids physical contact with the person, especially in scenarios when hands and arms are occluded. Toward this goal, a three-branch multimodal Deep Neural Network (DNN) regression model was implemented. First, a robust human-pose keypoints detection to calculate shoulder-elbow angles is applied. Secondly, we extract the refined object’s segmented mask. Thirdly, we compute two intrinsic object properties. The concatenated outputs from these branches pass through extra dense layers, resulting in the prediction of the robot's 14 arms-joint angles. Compared to an only keypoint data processed-based model, our multimodal approach made a 17.7% accuracy improvement. The experiments highlight each pipeline step’s significance, showing important results even when hands and arms were heavily occluded, adjusting to different variations.

Tactile sensors are crucial in robotics as they enable robots to perceive and interact with their environment through touch, akin to the human sense of touch. Adjustable sensors that can adapt to various tasks by functional or structural modification have not been extensively explored. In terms of sensing adjustability of a sensor, two important aspects are the sensor’s sensitivity and compliance. This paper proposes a novel design for an adjustable compliance and sensitivity sensor composed of a silicone base, a permanent magnet elastomer (PME), and a printed circuit board (PCB) with magnetic transducers installed. Its adjustability is achieved by varying the pneumatic pressure. This paper presents the design, manufacturing process, and experimental characterization of such an adjustable compliance and sensitivity sensor. This paper thoroughly investigates how altering the pressure of the sensor influences its sensing properties. The results show that it can achieve adjustability in all three axes. For the current design, the sensitivity can be varied from 0.093 to 0.125 mT/N (34.41%), 0.089 to 0.13 mT/N (31.54%), and 0.169 to 0.45 mT/N (62.44%) in the X-, Y-, and Z-axis, respectively. The deformation it undertakes varies from 3.20 to 3.79 mm (18.44%), indicating the compliance change.

Driver inattention is a significant contributor to fatal car crashes, leading to a need for accurate driver gaze zone classification methods. However, these classifications are particularly challenging under unconstrained conditions, such as when a driver’s face is partially occluded by masks or scarves, when the environment has significant lighting differences, or when the driver’s eyeglasses have reflections. This paper presents a framework that addresses these challenges by combining computer vision techniques and different deep-learning models to robustly recognize a driver’s gaze zone under highly unconstrained conditions. The framework uses a Contrast-Limited Adaptive Histogram Equalization (CLAHE) to adjust the color space of the frame, making it easier to recognize features under varying light conditions. It then employs dense landmark detection techniques to achieve robust recognition of the face, eyes, and pupils, including the use of optical flow estimation methods for tracking pupil and eyelid movement. The framework considers two facial poses and trains individual Deep Neural Network (DNN) models for each pose. As facial structure varies among individuals, the feature vector parameters for these DNN models are based on different relations between pupil and eye landmarks proportional to the driver’s face. The method has demonstrated its outstanding performance under a dataset involving highly unconstrained driving conditions.

In both teleoperation in real space and exploration in virtual space, ‘passive’ and ‘active’ haptic feedback can help to improve the performance of the task, especially in object handover and exploring. However, the current wearable haptic devices are hard to display continuous omnidirectional motion feedback simultaneously, which makes it not yet achieved. In this study, we thus propose a cutaneous haptic device, which enables continuous omnidirectional motion feedback for exhibiting ‘active’ and ‘passive’ haptic feedback. By applying small smart actuators (i.e., piezo actuators), the device can obtain contact force and be wearable. By arranging the closed loop with a plain-woven structure, our device makes continuous omnidirectional motion feedback possible. Our 35 g device can generate 0.94 N contact force and 0.5 N shear force. The passive and active haptic evaluations also proved its haptic capability. In conclusion, our proposed device with ‘active’ and ‘passive’ haptic feedback can provide continuous omnidirectional motion making it possible to be used for precise teleoperation.

A treadmill system has a large potential to provide humans with an augmented walking experience in real-life without a spatial limitation. However, a treadmill gait is different from walking on level ground. In previous studies, the adaptive belt speed control of a treadmill was developed to achieve a self-paced walking for making the users' treadmill gait similar to their level ground gait. Such studies have focused on steady-state walking and regulating the user's position on the treadmill. A normal gait can be divided into an acceleration state after gait initiation, a steady state, and a deceleration state for stopping. The objective of this study is to develop a treadmill system with a wire tension application enabling a human to experience a similar gait to a level ground gait during the transition phase from an acceleration state to a steady state. We developed a treadmill 4 m long × 1 m wide. To allow a user to move on the treadmill during the gait acceleration phase, an insensitive zone where a user can move without the treadmill belt drive was set. In addition, the treadmill was equipped with a wire traction system to apply a traction force canceling the effect of the belt floor acceleration of the treadmill when the belt speed of the treadmill changes. Through an experiment with six participants, the proposed treadmill system allowed the users to move in an acceleration state with the same head acceleration pattern as with level ground walking and cancel the inertial effect with the wire traction, which enabled the users to transition to a steady state from an acceleration state.

Drowsy drivers cause many deadly crashes. As a result, researchers focus on using driver drowsiness classifiers to predict this condition in advance. However, they only consider constraint situations. Under highly unrestricted scenarios, this categorization remains extremely difficult. For example, several studies consider the driver’s mouth closure crucial for detecting drowsiness. However, the mouth closure cannot be seen when the driver wears a mask, which is a potential failure for these classifiers. Moreover, these works do not make experiments under unconstrained situations as environments with considerable light variation or a driver with eyeglasses reflections. As a result, this paper proposes a video-based novel pipeline that employs new parameters, computer vision and deep-learning techniques to identify drowsiness in drivers under unconstrained situations. First, we alter the Lab color space of the frame to ease strong light changes. Then, we achieve a robust recognition of the face, eyes and body-joints landmarks using dense landmark detection that includes optical flow estimation methods for 3D eyelid and facial expression movement tracking and an online optimization framework to build the association of cross-frame poses. After this, we consider three important landmarks: eyes, lower-face contour, and chest. We performed several pre-processing and combinations using these landmarks to compare the efficiency of three alternative feature vectors. Finally, we fuse spatiotemporal features using a Gated Recurrent Units (GRU) model. Results over a dataset with highly unconstrained driving conditions demonstrate that our method outperforms classifying the driver’s drowsiness correctly in various challenging situations, all under mask-wearing scenarios.

Vision plays an important role in motion planning for mobile robots which coexist with humans. Because a method predicting a pedestrian path with a camera has a trade-off relationship between the calculation speed and accuracy, such a path prediction method is not good at instantaneously detecting multiple people at a distance. In this study, we thus present a method with visual recognition and prediction of transition of human action states to assess the risk of collision for selecting the avoidance target. The proposed system calculates the risk assessment score based on recognition of human body direction, human walking patterns with an object, and face orientation as well as prediction of transition of human action states. First, we investigated the validation of each recognition model, and we confirmed that the proposed system can recognize and predict human actions with high accuracy ahead of 3 m. Then, we compared the risk assessment score with video interviews to ask a human whom a mobile robot should pay attention to, and we found that the proposed system could capture the features of human states that people pay attention to when avoiding collision with other people from vision.

The skin is an important organ which enables humans to interact with the unstructured environment around. It is perfectly soft and covers the entire body providing immediate feedback even when that part is not directly in the field of vision. With the human skin as an inspiration, in this paper, we develop a novel completely soft robot skin for tactile sensing. The skin utilizes a new type of material called as Permanent Magnet Elastomer (PME) to replace the traditionally used hard permanent magnet for hall effect based tactile sensors. PME is formed by mixing Neodymium particles in a polymer base and using strong magnetization (up to 6 T) for anisotropy and to achieve strong and complete magnetization. The 6-axis soft PME is a perfect replacement for powerful hard magnets. We also do a thorough analysis of this material by infusing it in different types of silicone and as a result the most suitable combinations are selected. Performance tests show that the sensor can detect minute forces like 0.1 N. Moreover, the hysteresis test is carried out and the hysteresis error for our skin is found to be only 1.402%. An overloading test is also performed by loading the skin up to 64 N to check the robustness. In conclusion, the skin can produce reliable Triaxial force measurements and we present two models of it for smaller and large force range measurements respectively.

Although exists several drivers' gaze direction classifiers to prevent traffic accidents caused by inattentive driving, making this classification while the driver's face is temporarily or permanently occluded remains exceptionally challenging. For example, drivers using masks, sunglasses, or scarves and daily light variations are non-ideal conditions that recurrently appear in an everyday driving scenario and are frequently overlooked by the existing classifiers. This paper presents a single camera framework gaze zone classifier that operates robustly even during non-uniform lighting, non-frontal face pose, and faces undergo temporal or permanent occlusions. The usage of a normalized dense aligned face pose vector, the classification result of a pre-processed right eye area pixels, and the classification result of a pre-processed left eye area pixels is the cornerstone of the feature vector used in our model. The key of this paper is double-folded: firstly, the usage of a normalized dense alignment for a robust face, landmark, and head-pose direction detection and secondly, the processing of the right and left eye images using computer vision and deep learning techniques for refining, modifying, and finally labeling eyes information. Experiments on a challenging dataset involving non-uniform lighting, non-frontal face pose, and faces with temporal or permanent occlusions show each feature's importance towards making a robust gaze zone classifier under unconstrained driving situations.

Transmission systems that enable speed, torque, and direction conversion with high responsiveness and back-drivability are strongly required for higher performance robotic systems. Engagement states can be changed by clutches, but traditional clutches cannot change the direction and provide sufficient backdrivability. On the other hand, direction conversion can be realized by gear box such as bevel gears, but its backdrivability is poor. Thus, we newly develop a prototype transmission system with backdrivablity and responsiveness integrating clutch and direction converter using magnetorheological fluid (MRF). MRF is a functional fluid consisted of magnetic particles and carrier fluids which can change its viscosity rapidly and continuously according to the strength of magnetic field. MRF clutch consists of driving and driven shaft connected by vanes and coils for controlling MRF. MRF direction converter consists of bevel gears, brake, and MRF for reversing the output direction. In addition to traditional functions, such as speed and torque conversion, the proposed power transmission system provides five working modes: forward direction, reverse direction, and three kinds of free, according to states of the MRF clutch, MRF direction converter, and brake. Preliminary experiments revealed that the proposed transmission system could adequately realize implemented functions.

Distracted driving is one of the most common causes of traffic accidents around the world. Recognizing the driver's gaze direction during a maneuver could be an essential step for avoiding the matter mentioned above. Thus, we propose a gaze zone classification system that serves as a base of supporting systems for driver's situation awareness. However, the challenge is to estimate the driver's gaze inside not ideal scenarios, specifically in this work, scenarios where may occur self-occlusions or non-aligned head and eyes direction of the driver. Firstly, towards solving miss classifications during self-occlusions scenarios, we designed a novel protocol where a 3D full facial geometry reconstruction of the driver from a single 2D image is made using the state-of-the-art method PRNet. To solve the miss classification when the driver's head and eyes direction are not aligned, eyes and head information are extracted. After this, based on a mix of different data pre-processing and deep learning methods, we achieved a robust classifier in situations where self-occlusions or non-aligned head and eyes direction of the driver occur. Our results from the experiments explicitly measure and show that the proposed method can make an accurate classification for the two before-mentioned problems. Moreover, we demonstrate that our model generalizes new drivers while being a portable and extensible system, making it easy-adaptable for various automobiles.

Highly-automated vehicles operating in level 3 issue a takeover request (TOR) to transfer the control authority from the automated driving (AD) system to a human driver when they encounter system limitations. In such `unscheduled' situations, the driver is required to immediately re-engage in the driving task both physically and cognitively, and perform suitable action, e.g. lane change. Thus, evaluating driver engagement by the AD system would lead to safe takeover. Physical engagement is easily estimated but there are few studies on evaluating cognitive engagement. In this study, we thus developed a driver situational awareness estimation system based on glance information. We first defined seven standard glance areas and driver glance classification model using a convolutional neural network. We then obtained a large amount of glance data when both safe and dangerous takeover situations (lane change) by using a driving simulator, and we derived the standard glance model including the glance area and time, in order to estimate whether driver gained enough cognitive re-engagement in real-time. To evaluate the effectiveness of the proposed model, we created a situational awareness assist system to visually indicate regions with insufficient glance. As a result, we found that the assist system drastically improved driving performance and reduced the number of accidents during takeover.

Some electric-powered wheelchairs are recently redefined as personal mobility devices. Their users are not only elderly or handicapped people, but also passengers with large baggage or pedestrians going from station to destination, i.e., last-mile transport. Consequently, people with different operation skills and expectations on personal mobility would become new users of this kind of devices. Safe and comfort travel in human co-existing environment such as sidewalks and airports is a social expectation for personal mobility. In order to realize this, understanding the operation skill of each user by a practical and simple method is essential. This paper thus introduced a skill level classification method by machine learning using only joystick data as input. In order to determine the number of skill level clusters, basic 26 features of joystick operation data are used for unsupervised clustering (single-linkage). We then made evaluation indexes by using speed, speed control, and direction control. For a five-level classification by using gradient boosting as supervised learning, we achieved a 67% accuracy (tolerance: 0) and a 98% accuracy (tolerance: 1). Further analysis of the feature importance of gradient boosting revealed key features to a good operation. Results also show that skill level differed among people with different driving experiences.

Gas pipeline requires to be inspected regularly for leakages caused by natural disaster. Robots are widely used for pipeline inspection since they are more convenient than manual inspection. Several problems, however, exist due to the restriction by complex pipe networks. The most significant one is limited inspection range caused by restriction of cable length or wireless signal attenuation. In this paper, we proposed a concept of wireless relay communication to assist robot to extend the inspection range, and we newly developed a tracked robot chain system. In this system, each robot serves as a relay communication node. Leakage information of pipes are transmitted via these relay nodes. To ensure the stability of relay communication between adjacent robots, we adopted RSSI (received signal strength indication)-based evaluation method for cooperative and coordinated movement of robot chain system. Moreover, wireless application layer communication protocol (WALCP) was used to increase the stable performance of wireless relay communication. Each robot can self-navigate based on distance measurement module, which enables robots to pass through an elbow junction. Multiple experiments to evaluate relay transmission efficiency, RSSI-based cooperative movement, and comprehensive performance were conducted. Results revealed that our proposed system could realize relatively accurate relay transmission and RSSI-based coordinated movement.

Human sensing enables intelligent vehicles to provide driver-adaptive support by classifying perceived workload into multiple levels. Objective of this study is to classify driver workload associated with traffic complexity into five levels. We conducted driving experiments in systematically varied traffic complexity levels in a simulator. We recorded driver physiological signals including electrocardiography, electrodermal activity, and electroencephalography. In addition, we integrated driver performance and subjective workload measures. Deep learning based models outperform statistical machine learning methods when dealing with dynamic time-series data with variable sequence lengths. We show that our long short-term memory based recurrent neural network model can classify driver perceived-workload into five classes with an accuracy of 74.5%. Since perceived workload differ between individual drivers for the same traffic situation, our results further highlight the significance of including driver characteristics such as driving style and workload sensitivity to achieve higher classification accuracy.

Smooth and efficient robot navigation among humans is a crucial requirement for successful integration of robots in human society. Towards this end, an indispensable characteristic of robot action is legibility while communicating its intention. However, unlike humans, present robots cannot convey its intention through human-like non-verbal communication. This paper explores the use of projection indicators for communicating directional intent of a robot across three different `crossing scenarios' as a means of overcoming the shortcomings of the robot's non-verbal communication abilities. The results of the study show statistically significant improvement in perceived feelings of the measured attributes when using the auxiliary communication method. The studied method also improves cooperation from the participants. Nevertheless, the improvement in perceived feeling does not necessarily replicate in terms of smoothness across all the scenarios.

Efficient navigation in a congested environment is immensely difficult for the current state-of-the-art mobile robots owing to the challenges related to both sensing and actuation. Consequently, conventional approaches in mobile robotics research prioritize safety by slowing down or stopping robot's movement under challenging circumstances. This paper considers an alternative approach in which the robot utilizes forearm contact to create space for itself and to act as a safety buffer, during very close or congested navigation interactions. First, two categories of contact methods are defined based on different navigation scenarios. These contact methods are analyzed through different contact positions and force directions in a set of comparative experiments. The results of the comparative experiments are then incorporated into a contact-based framework that outputs the necessity, and the appropriate form of forearm contact. Finally, a set of evaluation experiments are performed to test the usability and effectiveness of the constructed framework. The results indicate that the proposed framework effectively outputs appropriate contact according to the situation, and aids the robot while navigating through space-constrained scenarios.

Disaster response crawler robot OCTOPUS has four arms and four flippers for better adaptability to disaster environments. To further improve the robot mobility and terrain adaptability in unstructured terrain, we propose a new locomotion control method called compound motion pattern (CMP) for multi-limb robots like OCTOPUS. This hybrid locomotion by cooperating the arms and flippers would be effective to adapt to the unstructured terrain due to combining the advantages of crawling and walking. As a preliminary study on CMP, we proposed a fundamental and conceptual CMP while clarifying problems in constructing CMP, and developed a semi-autonomous control system for realizing the CMP. Electrically-driven OCTOPUS was used to verify the reliability and correctness of CMP. Results of experiments on climbing a step indicate that the proposed control system could obtain relatively accurate terrain information and the CMP enabled the robot to climb the step. We thus confirmed that the proposed CMP would be effective to increase terrain adaptability of robot in unstructured environment, and it would be a useful locomotion method for advanced disaster response robots.

Intelligent vehicles operating in different levels of automation require the driver to fully or partially conduct the dynamic driving task (DDT) and to conduct fallback performance of the DDT, during a trip. Such vehicles create the need for novel human-machine interfaces (HMIs) designed to conduct high-level vehicle control tasks. Multimodal interfaces (MMIs) have advantages such as improved recognition, faster interaction, and situation-adaptability, over unimodal interfaces. In this study, we developed and evaluated a MMI system with three input modalities; touchscreen, hand-gesture, and haptic to input tactical-level control commands (e.g. lane-changing, overtaking, and parking). We conducted driving experiments in a driving simulator to evaluate the effectiveness of the MMI system. The results show that multimodal HMI significantly reduced the driver workload, improved the efficiency of interaction, and minimized input errors compared with unimodal interfaces. Moreover, we discovered relationships between input types and modalities: location-based inputs-touchscreen interface, time-critical inputs-haptic interface. The results proved the functional advantages and effectiveness of multimodal interface system over its unimodal components for conducting tactical-level driving tasks.

Human's ability to perceive intent plays a crucial role in achieving smooth and efficient navigation. At the present state, even with the state-of-the-art anthropomorphic robots, displaying human-like non-verbal communication (kinesics) is a challenging task. This poses a significant difficulty in performing legible navigation behavior for robots. In this paper, we look into light (turn indicator) and screen (arrow indicator) indicators as a means of overcoming the shortcomings of the robot's non-verbal communication abilities. Our results show a statistically significant improvement in perceived comfort, predictability, and performance with the use of light indicators.

We developed a four-arm four-crawler advanced disaster response robot called OCTOPUS. Disaster response robots are expected to be capable of both mobility, e.g., entering narrow spaces over very rough unstable ground, and workability, e.g., conducting complex debris-demolition work. However, conventional disaster response robots are specialized in either mobility or workability. Moreover, strategies to independently enhance the capability of crawlers for mobility and arms for workability will increase the robot size and weight. To balance environmental applicability with the mobility and workability, OCTOPUS is equipped with a mutual complementary strategy between its arms and crawlers. The four arms conduct complex tasks while ensuring stabilization when climbing steps. The four crawlers translate rough terrain while avoiding toppling over when conducting demolition work. OCTOPUS is hydraulic driven and teleoperated by two operators. To evaluate the performance of OCTOPUS, we conducted preliminary experiments involving climbing high steps and removing attached objects by using the four arms. The results showed that OCTOPUS completed the two tasks by adequately coordinating its four arms and four crawlers and improvement in operability needs.

The advantages of mechanical compliance have driven the development of devices using new smart materials. A new kind of magnetorheological piston based on a toroidal array of magnetorheological valves, has been previously tested to prove its feasibility. However, being an initial prototype its potential was still limited by its complex design, and low output force. This study presents the revisions done to the design with several improvements targeting key performance parameters. An improved annular piston design is also introduced as comparison with conventional devices. The toroidal and annular piston head prototypes are built and tested, and their force performance compared with the previous iteration. The experimental results show an overall performance improvement of the toroidal assembly. However, the force model used in the study still fails to accurately predict the magnetic flux at the gaps of the piston head. This deviation is later verify and corrected using a FEM analysis. The force performance of the new toroidal assembly is on par with the commonplace annular design. It also displays a more linear behaviour, at the expense of lower energy efficiency. Finally, it also shows potential for a greater degree of customisation to meet different system requirements.

Autonomous vehicles would make the future roads safer by keeping the human driver out of the loop. However, reduced degree of human-control could result in loss of the feeling of driving for some drivers. Therefore, in this study we proposed a method of interaction between the driver and autonomous vehicle by allowing the driver to control the vehicle's lateral and longitudinal motions. We adopted hand gestures as input modality because it can reduce driver's visual and cognitive demands. We first derived seven fundamental vehicle maneuvers to improve driver experience, and related them to seven independent hand gestures. We then created a hand gesture interface to control an autonomous vehicle, using Leap Motion as the gesture recognition platform. We conducted driving experiments involving twenty drivers in a virtual reality driving simulator to investigate the effectiveness of this interface for vehicle control. We evaluated the driving experience and drivers' opinions regarding the gestural interface. The results proved that semi-autonomous controlling using the hand gesture interface significantly reduced drivers' perceived workload.

In unmanned construction, work efficiency is lower than that in manned construction due to lack of visual information. Thus, we previously developed an autonomous camera control system to provide various visual information suited to work states through multiple displays. However, that system increased the cognitive load on operators, and required them to have much experience to choose appropriate views for various situations. Next, we should investigate the degree of effectiveness for each view in a certain state. Thus, in this study, we analyzed gaze patterns to clarify which are the displays that operators often watch in work states, i.e., moving, grasping, transport, and releasing. We then derived which gaze patterns have higher work performance, including time efficiency and safeness. We clustered gaze patterns using Ward's method, which is a criterion applied in hierarchical clustering. To evaluate the objective of this study, we conducted experiments involving debris transport tasks, using a virtual reality simulator. The results indicated that gaze patterns differed in operators and we found that better time efficiency related to specific gaze patterns for each work state.

A new design of a magnetorheological piston prototype intended for passive or active force control in robotic applications for human robot interaction is introduced. It is based in a novel toroidal array of valves, contained within the piston head, which are used to control the output force of the actuator in order to achieve a high degree of reliability, size efficiency, and safety, by exploiting the material properties of magnetorheological fluids and permalloy metals. This paper describes the main points in the development of the magnetorheological piston prototype, the mathematical modelling of the magnetic circuit, and the results of the experiments conducted using a universal testing machine to evaluate the passive performance of the prototype. Results show the feasibility and performance of the new toroidal magnetic circuit of the magnetorheological hydraulic piston prototype. Improvements in order to be able to test the active performance of the design together with a pump setup are proposed.

Unmanned construction machines are used after disasters. Compared with manned construction, time efficiency is lower because of incomplete visual information, communication delay, and lack of tactile experience. We have developed an autonomous camera control system to supply appropriate visual information. In order to inform operators the potential data in the views, we introduced several augmented reality (AR) elements to induce visual attention to suitable images in a scenario. The purpose of this study is to develop an attention inducement system using AR technique and evaluate it under different conditions. We first improve our autonomous camera control system and then implement several kinds of AR items suitable for each work situation. The experimental results conducted using our virtual reality simulator confirm that AR items can supply a better positional relationship between machine and objects in the environment which makes operator handle the conditions as experienced. The results from eye-tracker data also indicate that AR items can induce visual attention to images suitable for situations.

A method to autonomously control multiple environmental cameras, which are currently fixed, for providing more adaptive visual information suited to the work situation for advanced unmanned construction is proposed. Situations in which the yaw, pitch, and zoom of cameras should be controlled were analyzed and imaging objects including the machine, manipulator, and end-point and imaging modes including tracking, zoom, posture, and trajectory modes were defined. To control each camera simply and effectively, four practical camera roles combined with the imaging objects and modes were defined as the overview-machine, enlarge-end-point, posture-manipulator, and trajectory-manipulator. A role assignment system was then developed to assign the four camera roles to four out of six cameras suitable for the work situation, e.g., reaching, grasping, transport, and releasing, on the basis of the assignment priority rules, in the real time. Debris removal tasks were performed by using a VR simulator to compare fixed camera, manual control, and autonomous systems. Results showed that the autonomous system was the best of the three at decreasing the number of grasping misses and error contacts and increasing the subjective usability while improving the time efficiency.

A method to tune a basic input/output gain (BIOG) according to usage conditions for human-machine systems is proposed for improving work performance. Adapting a BIOG is effective in terms of improving operability and work efficiency, but frequent changes using a gain scheduling strategy degrade the operability by confusing the machine dynamics. The proposed tuning system adjusts a BIOG at long intervals on the basis of comprehensive features in operator and work content obtained from the histogram of control lever input. The target value is set to the normal distribution, meaning that all ranges of the control lever are evenly used in a spring-type lever, as leveling the histogram independent of an operator and work content provides the consistent operational feeling, which leads to comfortable operability. To ensure practicality and effectiveness, a BIOG curve is set to a polygonal line involving a break point and a saturation point that are tuned by equivalent transform of area differences between the obtained histogram and normal distribution curves. Two types of experimental task were performed using a hydraulic arm system. Results showed that the proposed BIOG tuning system improves time efficiency by reducing the area difference while increasing the subjective usability compared with a conventional fixed BIOG system.

This paper proposes a framework for visualizing comprehensive work tendency using a frequency plot map of the end-point of a manipulator as a fundamental study of work analysis using long-term data for human-operated work machines. A visualization system requires high generality and commonality to enable to extract various characteristics to be extracted on an arbitrary time scale independently of the work machine specifications, work contents and environment, and machine operator. The proposed framework meets this requirement by first creating a two-dimensional end-point plot map with its origin fixed at the yaw joint of the manipulator to deal with arbitrary usage conditions. It then extracts arbitrary characteristics by using hierarchical feature extraction filters, including binary, quantity, and advanced filters, defined by three kinds of essential data: operation, movement, and load. Finally, it quantifies the filtered map by gridding and normalization to visually grasp its frequency distribution. Two experiments in which the work environments and completion time differed were conducted using an instrumented hydraulic arm. Results indicated that the comprehensive work tendency revealed by using the proposed framework corresponds to the actual work results independently of the various conditions.

This paper proposes a practical framework to estimate whether or not a grapple installed in demolition machines is in a grasp state. Object grasp is a highly difficult task that requires safe and precise operations, so identifying a grasp or non-grasp state is important for assisting an operator. These types of outdoor machines lack visual and tactile sensors, so the proposed framework adopts practically available lever operation and cylinder pressure sensors. The grasp is formed by a grasp motion, which is operations to make the grapple pinch an object, and the grasp state, where the grapple holds the object in any manipulator movements. Thus, the framework determinately confirms the grasp motion through the requisite conditions defined by using sequential changes of binarized operation and pressure data for the grapple and the manipulator, and stochastically confirms the grasp state through the enhancement conditions defined by using force and movement vectors including vertical downward force, movement in the longer direction, and horizontal reciprocating movement. The results of experiments conducted to transport objects using an instrumented hydraulic arm indicated that the proposed framework is effective for identifying grasp/non-grasp with high accuracy, independently of various operators and environments.

This paper proposes a quantification method for a comprehensive work flow in construction work for describing work states in more detail on the basis of analyzing state transitions of primitive static states (PSS), which consist of 16 symbolic work states defined by using on-off state of the lever operations and joint loads for the manipulator and end-effector. On the basis of the state transition rules derived from a transition-condition analysis, practical state transitions (PST), which are common and frequent transitions in arbitrary construction work, are defined. PST can be classified into essential (EST) or nonessential state transitions (NST). EST extracts common phases of work progress and estimates positional relations between a manipulator and an object. NST reveals wasted movements that degrade the efficiency and quality of work. To evaluate comprehensive work flows modeled by combing EST and NST, work-analysis experiments using our instrumented setup were conducted. Results indicate that all the PSS definitely changes on the basis of PST under various work conditions, and work analysis using EST and NST easily reveals work characteristics and untrained tasks related to wasted movements.

This paper proposes a practical framework for detecting (identifying the on-off state of) the external force applied to a construction manipulator (front load) by using a hydraulic sensor. Such a detection system requires high accuracy and robustness considering the uncertainty in pressure-based force measurement. Our framework is thus organized into (i) identifying the dominant error force component (self-weight and driving force) using theoretical and experimental estimation and binarizing the analog external cylinder force, (ii) evaluating detection conditions to address indeterminate conditions such as stroke-end, singular posture, and impulsive or oscillatory force and redefining three-valued outputs such as on, off, or not de terminate (ND), and (iii) outputting the front load decision by combining all the cylinder decisions to improve robustness through priority analysis. Experiments were conducted using an instrumented hydraulic arm. Results indicate that our frame work adequately detects on, off, and ND outputs of the front load in various detection conditions without misidentification.

This paper proposes a practical framework for measuring the external force applied to a construction manipulator (front load vector) by using a hydraulic sensor. Such a force measurement system requires high accuracy and robustness considering the uncertainty in the construction machinery field, but it inevitably includes measurement errors owing to difficult-to-reduce modeling errors. Our framework thus adopts a relative accuracy improvement strategy without correcting the models for the practicality. It comprises (i) quantifying the internal error range (IER) by using the sum of the maximal measurement errors of static and dynamic friction forces, which change in postural and kinematic conditions, (ii) calculating the error force vector by using IER to select cylinders (sensors) that have less error, and (iii) outputting the front load vector using the cylinders whose error force vector is minimum. Experiments were conducted using an instrumented hydraulic arm. The results indicate that our framework can enhance the relative accuracy of external force measurement independently of various postural and kinematic conditions.

This paper proposes a state identification framework to support the complicated dual-arm operations in construction work. The operational support in construction machinery filed requires the compatibility with different types of support and the commonality among various operator skill levels. The proposed framework is therefore organized into two functions: real-time task phase identification and time-series attentional condition identification. The task phase is defined by utilizing the joint load applied according to the environment constraint condition. The attentional condition is defined as one of the internal work-state condition classified by the necessity level of operational support, and is dependent on the vectorial or time-series date selected by the identified task phase. Experiments are conducted using the hydraulic dual arm system to perform transporting and removing tasks. Results show that the number of error contacts, internal force applied, and mental workload is decreased without time-consumption increase. The result confirmed that the proposed framework greatly contribute to improving each operator's work performance.

Advanced operated-work machines, which have been designed for complicated tasks and which have complicated operating systems, requires intelligent systems that can provide the quantitative work analysis needed to determine effective work procedures and that can provide operational and cognitive support for operators. Construction work environments are extremely complicated, however, and this makes state identification, which is a key technology for an intelligent system, difficult. We therefore defined primitive static states (PSS) that are determined using on-off information for the lever inputs and manipulator loads for each part of the grapple and front and that are completely independent of the various environmental conditions and variation in operator skill level that can cause an incorrect work state identification. To confirm the usefulness of PSS, we performed experiments with a demolition task by using our virtual reality simulator. We confirmed that PSS could robustly and accurately identify the work states and that untrained skills could be easily inferred from the results of PSS-based work analysis. We also confirmed in skill-training experiments that advice information based on PSS-based skill analysis greatly improved operator's work performance. We thus confirmed that PSS can adequately identify work states and are useful for work analysis and skill improvement.