Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agravante, Michiaki Tatsubori, Asim Munawar, Alexander Gray. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Deep neural networks have recently led to promising results for the task of multiple sound source localization. Yet, they require a lot of training data to cover a variety of acoustic conditions and micro-phone array layouts. One can leverage acoustic simulators to inexpensively generate labeled training data. However, models trained on synthetic data tend to perform poorly with real-world recordings due to the domain mismatch. Moreover, learning for different microphone array layouts makes the task more complicated due to the infinite number of possible layouts. We propose to use adversarial learning methods to close the gap between synthetic and real do-mains. Our novel ensemble-discrimination method significantly improves the localization performance without requiring any label from the real data. Furthermore, we propose a novel explicit transformation layer to be embedded in the localization architecture. It enables the model to be trained with data from specific microphone array layouts while generalizing well to unseen layouts during inference.

In this paper, we propose novel deep learning based algorithms for multiple sound source localization. Specifically, we aim to find the 2D Cartesian coordinates of multiple sound sources in an enclosed environment by using multiple microphone arrays. To this end, we use an encoding-decoding architecture and propose two improvements on it to accomplish the task. In addition, we also propose two novel localization representations which increase the accuracy. Lastly, new metrics are developed relying on resolution-based multiple source association which enables us to evaluate and compare different localization approaches. We tested our method on both synthetic and real world data. The results show that our method improves upon the previous baseline approach for this problem.

This paper contributes to the field of physical human-robot collaboration. We present a complete control framework, which aims at making humanoid robots capable of carrying objects together with humans. First, we design a template identifying the primitive subtasks necessary for collaborative carrying. Then, these subtasks are formulated as constrained optimization problems for controlling the whole-body motion of a humanoid robot. The subtasks include two walking pattern generators that account for physical collaboration, as well as posture and grasping controllers. Finally, we validate our framework in a variety of collaborative carrying experiments, using the HRP-4 humanoid robot.

This paper is a contribution towards interpretability of the deep learning models in different applications of time-series. We propose a temporal attention layer that is capable of selecting the relevant information to perform various tasks, including data completion, key-frame detection and classification. The method uses the whole input sequence to calculate an attention value for each time step. This results in more focused attention values and more plausible visualisation than previous methods. We apply the proposed method to three different tasks. Experimental results show that the proposed network produces comparable results to a state of the art. In addition, the network provides better interpretability of the decision, that is, it generates more significant attention weight to related frames compared to similar techniques attempted in the past.

In active vision, the camera motion is controlled in order to improve a certain visual sensing strategy. In this paper, we formulate an active vision task function to improve pose estimation. This is done by defining an optimality metric on the Fisher Information Matrix. This task is then incorporated into a weighted multi-objective optimization framework. To test this approach, we apply it on the three image point visual servoing problem which has a degenerate configuration - a singularity cylinder. The simulation results show that the singular configurations of pose estimation are avoided during visual servoing. We then discuss the potential of active vision to be integrated into more complex multi-task frameworks.

This paper is about the design of humanoid walking pattern generators to be used for physical collaboration. A particular use case is a humanoid robot helping a human to carry large and/or heavy objects. To do this, we construct a reduced model which takes into account physical interaction. This is used in a model predictive control framework to generate separate behaviors for being a follower or a leader. The approach is then validated both on simulation and on the HRP-4 humanoid robot.

We propose a framework for combining vision and haptic information in human-robot joint actions. It consists of a hybrid controller that uses both visual servoing and impedance controllers. This can be applied to tasks that cannot be done with vision or haptic information alone. In this framework, the state of the task can be obtained from visual information while haptic information is crucial for safe physical interaction with the human partner. The approach is validated on the task of jointly carrying a flat surface (e.g. a table) and then preventing an object (e.g. a ball) on top from falling off. The results show that this task can be successfully achieved. Furthermore, the framework presented allows for a more collaborative setup, by imparting task knowledge to the robot as opposed to a passive follower.

In this paper, a framework for visual servo control of a humanoid robot's upper body is presented. The framework is then implemented and tested on the REEM humanoid robot. The implementation is composed of 2 controllers - a head gaze control and a hand position control. The main application is precise manipulation tasks using the hand. For this, the hand controller takes top priority. The head controller is designed to keep both the hand and object in the eye field of view. For robustness, a secondary task of joint limit avoidance is implemented using the redundancy framework and a large projection operator proposed recently. For safety, joint velocity scaling is implemented. The implementation on REEM is done using the ROS and ViSP middleware. The results presented show simulations on Gazebo and experiments on the real robot. Furthermore, results with the real robot show how visual servoing is able to overcome some deficiency in REEM's kinematic calibration.

In this paper, a first step is taken towards using vision in human-humanoid haptic joint actions. Haptic joint actions are characterized by physical interaction throughout the execution of a common goal. Because of this, most of the focus is on the use of force/torque-based control. However, force/torque information is not rich enough for some tasks. Here, a particular case is shown: height stabilization during table carrying. To achieve this, a visual servoing controller is used to generate a reference trajectory for the impedance controller. The control law design is fully described along with important considerations for the vision algorithm and a framework to make pose estimation robust during the table carrying task of the humanoid robot. We then demonstrate all this by an experiment where a human and the HRP-2 humanoid jointly transport a beam using combined force and vision data to adjust the interaction impedance while at the same time keeping the inclination of the beam horizontal.