We present NeuralLabeling, a labeling approach and toolset for annotating 3D scenes using either bounding boxes or meshes and generating segmentation masks, affordance maps, 2D bounding boxes, 3D bounding boxes, 6DOF object poses, depth maps, and object meshes. NeuralLabeling uses Neural Radiance Fields (NeRF) as a renderer, allowing labeling to be performed using 3D spatial tools while incorporating geometric clues such as occlusions, relying only on images captured from multiple viewpoints as input. To demonstrate the applicability of NeuralLabeling to a practical problem in robotics, we added ground truth depth maps to 30000 frames of transparent object RGB and noisy depth maps of glasses placed in a dishwasher captured using an RGBD sensor, yielding the Dishwasher30k dataset. We show that training a simple deep neural network with supervision using the annotated depth maps yields a higher reconstruction performance than training with the previously applied weakly supervised approach. We also show how instance segmentation and depth completion datasets generated using NeuralLabeling can be incorporated into a robot application for grasping transparent objects placed in a dishwasher with an accuracy of 83.3%, compared to 16.3% without depth completion. Supplementary URI: https://florise.github.io/neural_labeling_web/.

We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DoF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. PEGASUS allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that capture images from both the object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.

We propose a technique for depth completion of transparent objects using augmented data captured directly from real environments with complicated geometry. Using cyclic adversarial learning we train translators to convert between painted versions of the objects and their real transparent counterpart. The translators are trained on unpaired data, hence datasets can be created rapidly and without any manual labeling. Our technique does not make any assumptions about the geometry of the environment, unlike SOTA systems that assume easily observable occlusion and contact edges, such as ClearGrasp. We show how our technique outperforms ClearGrasp in a dishwasher environment, in which occlusion and contact edges are difficult to observe. We also show how the technique can be used to create an object manipulation application with a humanoid robot. Supplementary URI: https://ftorise.github.io/faking_depth_web/.

Robot technologies, such as robot technology middleware (RTM) that is a component-oriented platform, are popular. However, RTM does not ensure stable real-time processing in common object request broker architecture. In this paper, a collaboration framework of RTM and TOPPERS embedded component system (TECS) is proposed to address this problem. TECS, a system that satisfies real-time processing requirements, is employed to enhance real-time processing in the proposed framework. To implement the collaboration of RTM and TECS, we have adopted remote procedure call and one-way communication. In addition, extending a generator enables the generation of robot technology components from TECS components. We have evaluated the processor cycle counts of the proposed framework in comparison with those of a conventional method. In addition, we evaluated the execution time of serial communication and a motor application using the proposed framework. The evaluation results show that the proposed framework is functionally employed in a hard real-time system. Furthermore, we evaluated the amount of code generated by the proposed framework. The evaluation results reveal that the code generated by the proposed framework is reusable and can enhance productivity.

The 4C's of software, computation, communication, configuration and coordination, indicate that the deployment and configuration of software components is an important part of using a component-based software system. This applies as much to robotics as anywhere else, where how to easily deploy the complex web of software components that make up a complete robot system is a well-known problem. We have developed an experimental tool for the deployment and configuration of components using RT-Middleware. It is based on the supervision tree concept. Components are organised into a hierarchy of sub-systems and managed in these groups. The use of a supervision tree ensures that dependencies between components are managed when errors occur, and that errors will propagate up the tree to a supervisor that is capable of handling them.

This paper presents the development of Robot Technology Components (RTCs) using RT-Middleware (RTM) for designing assistive robotic arms and human-machine interfaces (HMIs) that can be used by the disabled. RTM is an open software platform for a component-oriented robot system. An RTC is the basic functional unit for constructing a robot system using RTM. The interface definition for the basic functionality of RTCs is standardized by the Object Management Group (OMG). Therefore, software modules written as RTCs can be shared and reused in other robot systems. One feature of an assistive robotic arm is that the arm is controlled by manual operation. Various HMIs for manually operating the robotic arms have been developed according to the abilities of each user and the requirements regarding the actions that the robot can perform. These HMIs can be interchangeably connected to the assistive robotic arms by implementing the software modules for the robotic arm and the HMI as RTCs with a common software interface.

We are developing a manipulator system in order to support disabled people with less muscle strength such as muscular dystrophy patients. Such a manipulator should have an easy interface for the users to control it. In this paper, we report on the construction of the user interface of the system using RT-middleware. RT-middleware is an open software platform for robot systems. Therefore other interface components or robot components which are adapted to other symptoms can be replaced with the interface without any change of the contents. A single switch is introduced as the input device for the manual control of the robot arm in our interface. The scanning menu panel is designed to perform various actions of the robot arm with the single switch. A manipulator simulation system was constructed to evaluate the input performance. Two muscular dystrophy patients tried of our interface to control the robot simulator and made comments.

In this paper, we propose RT-middleware for robot system integration. "RT" means "robot technology", which is applied not only to industrial field but also to nonindustrial field such as human daily life support systems. RT-middleware which is proposed in this paper is a software platform for RT systems. We have studied modularization of RT elements and have developed RT-middleware, which promotes application of RT in various fields. Robotic system development methodology and our RT-middleware concepts are discussed. The RT-component, which is a basic software unit of RT-middleware based system integration, is derived from this discussion. A methodology of system development by using RT-components, and a framework for component development are proposed. Evaluations of some RT-component based systems are performed. Finally, conclusions and future work is described.

This paper proposes human-robot cooperative tele-micromanipulation system with novel single-master (PHANToM haptic device) multi-slaves (6 D.O.F parallel manipulator). The different kinematical configuration and the different D.O.F between the master and the slave introduces a mapping problem which can be serious for some cooperative manipulations. The position/force virtual mapping method is implemented in the single-master dual-slave tele-micromanipulation system to realize the human-robot cooperative internal force regulation while grasping task. The generated reference trajectories of both manipulators by the virtual mapping method are shared with the cooperative impedance control of multi-slaves to regulate the internal force while grasping an object. Lastly, an improved internal force regulation using the SMMS (single-master multi-slave) tele-micromanipulation system while grasping an object is verified by the pick-and-place experimental results.

The concept of "SmallConnection (abbr. SC)" is creating easy to operate tangible media for communication over networks. Focusing on the scenario where two intimate people live in distant places, we developed communication media that can be handled like tools, and can convey faint information such as light, wind and touch through the use of a robot technology. The goal of this project is to propose and prototype new media design for communication between two people. Through working 3 products derived from SC, we hope to widely propose a?future communication design using multimedia.

The Intelligent Space is a space where we can easily interact with computers and robots, and get useful service from them. To achieve such a space, a distributed intelligent networked device (DIND) has been proposed. Many DINDs are installed in a space and they cooperate each other to make the space to become an Intelligent Space. In this paper, an optimal DIND placement, self-calibration of DIND and handover protocol for cooperation among DINDs, are described.

In this paper, control schemes for the master haptic interface are mainly discussed. We proposed the tele-micromanipulation systems, which enables human operators to operate micro-tasks, such as assembly or manufacturing, without feeling the stress. The paper focuses on the haptic interface, which gives the operators the feeling of presence. The mechanism applied in the human interface device often has a reasonable immanent friction. This friction must be compensated in a way that the operator cannot feel this friction force, but only the force from the manipulated environment. The main contribution of this paper is a direct model based chattering free sliding mode friction estimator and a compensator for the human interface device. The experimental results obtained are presented.

In this paper, control schemes for haptic interface for micromanipulation systems is mainly discussed. We have proposed a tele-micromanipulation system, which enables human operators to operate micro tasks, such as assembly or manufacturing, without feeling stress. This paper focuses on the haptic sense, which gives the operators the feeling of presence. The mechanical instrument applied in the human interface device usually has a reasonable immanent friction. This friction must be compensated in a way that the operator should not feel this friction force but the force from the manipulated environment. The main contribution of this paper is a direct model based chattering free sliding mode friction estimator and compensator for a mechanical instrument. Experimental results are presented.

A dexterous tele-micromanipulation system using haptic interfaces has been developed for the application in assembling electrical/optical parts for electrical devices, micro-machine, etc. The paper describes the development of visual operation support systems for tele-micromanipulation systems. Because of its better manipulation positioning accuracy, stiffness and speed characteristics, a parallel mechanism micromanipulator was chosen for our system. While the parallel manipulator has the above-mentioned merit, there is the disadvantage that the parallel mechanism's workspace is narrow and the translation motion workspace changes depends on the posture of end-effector. Providing workspace information in real time to the human operator, he can recognize the workspace and helps planning the micromanipulation process. A parallel manipulator motion is restricted by three factors: mechanical limits of the passive joints, collision between the links and actuators limitations. Results of numerical workspace analysis considering the above factors are shown. Based on the workspace analysis, a VR simulator and workspace display systems for micromanipulator are proposed. The feasibility of the use of a workspace display system for micromanipulation is then discussed.

We propose a method of guiding mobile robots in a networked space. To watch human and robots, distributed sensor devices with processors are located around the networked space. In this space, robots as well as the human are supported informatively and physically. The distributed sensor devices guide mobile robots in this space and navigation with high adaptability is realized. The simulation and experimental results including the camera arrangement and robot guidance with distributed sensor devices are shown.

In this paper, we discuss about micro teleoperation with haptic interfaces. We developed the micro teleoperation systems for micro tasks, such as assembly or manufacturing. We show about structure of master/slave manipulators, its control and experimental results of teleoperation with the manipulators in this paper. We introduce the haptic interface that provides operators the sense as if he/she touches the expanded micro objects with his/her fingers. This micro tele-manipulation system is a new tool, which enables human operators to touch and feel the expanded micro objects with their fingers.

Usually localization in mobile robots is interested in only the geographical position of the robot in space. However, to utilize mobile robots in general environments such as a hospital, an office, a school etc., we need more than conventional localization. We propose a new localization method that watches the entire space as well as the mobile robot. Some experimental results are shown to show the performance and the merits of proposed localization method.