Teleoperation of robots can be challenging, especially for novice users with little to no experience at such tasks. The difficulty is largely due to the numerous degrees of freedom users must control and their limited perception bandwidth. To help mitigate these challenges, we propose in this paper a solution which relies on artificial intelligence to understand user intended motion and then on mixed reality to communicate the estimated trajectories to the users in an intuitive manner. User intended motion is estimated using a deep learning network trained on a dataset of motion primitives. During teleoperation, the estimated motions are augmented onto a first-person live video feed from the robot. Finally, if a suggested motion is accepted by the user, the robot is driven along that trajectory in an autonomous manner. We validate our proposed mixed reality teleoperation scheme with simulation experiments on a drone and demonstrate, through subjective and objective evaluation, its advantages over other teleoperation methods.

Controlling and manning robots from a remote location is difficult because of the limitations one faces in perception and available degrees of actuation. Although humans can become skilled teleoperators, the amount of training time required to acquire such skills is typically very high. In this paper, we propose a novel solution (named Autocomplete) to aid novice teleoperators in manning robots adroitly. At the input side, Autocomplete relies on machine learning to detect and categorize human inputs as one from a group of motion primitives. Once a desired motion is recognized, at the actuation side an automated command replaces the human input in performing the desired action. So far, Autocomplete can recognize and synthesize lines, arcs, full circles, 3-D helices, and sine trajectories. Autocomplete was tested in simulation on the teleoperation of an unmanned aerial vehicle, and results demonstrate the advantages of the proposed solution versus manual steering.

Have you ever wondered how a song might sound if performed by a different artist? In this work, we propose SCM-GAN, an end-to-end non-parallel song conversion system powered by generative adversarial and transfer learning, which allows users to listen to a selected target singer singing any song. SCM-GAN first separates songs into vocals and instrumental music using a U-Net network, then converts the vocal segments to the target singer using advanced CycleGAN-VC, before merging the converted vocals with their corresponding background music. SCM-GAN is first initialized with feature representations learned from a state-of-the-art voice-to-voice conversion and then trained on a dataset of non-parallel songs. After that, SCM-GAN is evaluated against a set of metrics including global variance GV and modulation spectra MS on the 24 Mel-cepstral coefficients (MCEPs). Transfer learning improves the GV by 35% and the MS by 13% on average. A subjective comparison is conducted to test the output's similarity to the target singer and its naturalness. Results show that the SCM-GAN's similarity between its output and the target reaches 69%, and its naturalness reaches 54%.

In this work, we are proposing an intuitive Augmented SLAM method (A-SLAM) that allows the user to interact, in real-time, with a robot running SLAM to correct for pose and map errors. We built an AR application that works on HoloLens and allows the operator to view the robot's map superposed on the physical environment and edit it. Through map editing, the operator can account for errors affecting real environment's representation by adding navigation-forbidden areas to the map in addition to the ability to correct errors affecting the localization. The proposed system allows the operator to edit the robot's pose (based on SLAM request) and can be extended to sending navigation goals to the robot, viewing the planned path to evaluate it before execution, and teleoperating the robot. The proposed solution could be applied on any 2D-based SLAM algorithm and can easily be extended to 3D SLAM techniques. We validated our system through experimentation on pose correction and map editing. Experiments demonstrated that through A-SLAM, SLAM runtime is cut to half, post-processing of maps is totally eliminated, and high quality occupancy grid maps could be achieved with minimal added computational and hardware costs.