Real-time SLAM with dense 3D mapping is computationally challenging, especially on resource-limited devices. The recent development of 3D Gaussian Splatting (3DGS) offers a promising approach for real-time dense 3D reconstruction. However, existing 3DGS-based SLAM systems struggle to balance hardware simplicity, speed, and map quality. Most systems excel in one or two of the aforementioned aspects but rarely achieve all. A key issue is the difficulty of initializing 3D Gaussians while concurrently conducting SLAM. To address these challenges, we present Monocular GSO (MGSO), a novel real-time SLAM system that integrates photometric SLAM with 3DGS. Photometric SLAM provides dense structured point clouds for 3DGS initialization, accelerating optimization and producing more efficient maps with fewer Gaussians. As a result, experiments show that our system generates reconstructions with a balance of quality, memory efficiency, and speed that outperforms the state-of-the-art. Furthermore, our system achieves all results using RGB inputs. We evaluate the Replica, TUM-RGBD, and EuRoC datasets against current live dense reconstruction systems. Not only do we surpass contemporary systems, but experiments also show that we maintain our performance on laptop hardware, making it a practical solution for robotics,$A / R$, and other real-time applications.

In this paper, we propose a collaborative centralized 3D mapping and localization framework that harnesses the capabilities of both SLAM (Simultaneous Localization And Mapping) and XR (eXtended Reality). On one hand, our framework allows for integrating local maps generated by a multitude of heterogeneous agents (e.g. robots) into a unified map. On the other hand, it allows human intervention at multiple levels: first, humans can inspect and intervene in the mapping process in situ to produce 3D maps, overlay virtual assets, and add annotations, all of which can contribute towards enhanced autonomy and navigation. Second, beyond the mapping aspect, a human can also intervene in the localization task of any collaborating robot by inspecting and correcting its generated paths, and, if necessary, enforcing a desired trajectory. Experiments inside two real settings demonstrated the superiority of the proposed system.

Manually teleoperating a flying robot can be a demanding task, especially for users with limited levels of experience. This is primarily due to the non-linear properties of such robots in addition to the difficulty of controlling various degrees of freedom at the same time. 3D Autocomplete helps mitigate such limitations by assisting the users in teleoperation. It aids in teleoperating 3D motions, such as helical motions, which are more challenging to the users. The proposed framework uses Artificial Intelligence (AI) to predict just-in-time the user’s intended motion and then, if the user accepts, completes it autonomously in 3D. The AI component of 3D Autocomplete was presented in our previous work, where we introduced a deep learning model and an algorithm to predict as early as possible the user’s desired motion. Moving forward in this work, we focus on synthesizing and completing the user-intended motion autonomously. Also, we introduce a Mixed Reality (MR) user interface for better human-robot interaction. Finally, we evaluate our system subjectively and objectively through human-subject experiments. Autocomplete outperformed traditional method on all criteria with at least 30% improvement in all objective measures.

Simultaneous Localization and Mapping (SLAM) has emerged as a prime autonomous mobile agent localization algorithm. Despite the global research effort to improve SLAM, its mapping component remains limited and serves little more than to satisfy the coupled localization problem. We present a collaborative 3D SLAM approach leveraging the power of augmented reality (AR). The system introduces a trio of diverse agents, each with its unique capability to become an active member in the mapping process: mobile robots, human operators, and AR head-mounted display (AR-HMD). A 3D complementary mapping pipeline is developed to utilize the built-in SLAM capabilities of the AR-HMD as shareable data. Our system aligns and merges the AR-HMD and the robot’s local map automatically, triggered by a human-dictated initial guess. The created merged map proves advantageous in scenarios where the robot is restricted from navigating in certain areas. To correct map imperfections resulting from problematic objects such as transparent or reflective surfaces, the fused map is overlayed onto the environment, and hand gestures are used to add or delete 3D map features in real-time. Our system is implemented in both a lab and a real industrial warehouse setup. The results show a significant improvement in the map quality and mapping duration.

This paper presents an integrated approach to Visual SLAM, merging online sequential photometric calibration within a Hybrid direct-indirect visual SLAM (H-SLAM). Photometric calibration helps normalize pixel intensity values under different lighting conditions, and thereby improves the direct component of our H-SLAM. A tangential benefit also results to the indirect component of H-SLAM given that the detected features are more stable across variable lighting conditions. Our proposed photometrically calibrated H-SLAM is tested on several datasets, including the TUM monoVO as well as on a dataset we created. Calibrated H-SLAM outperforms other state of the art direct, indirect, and hybrid Visual SLAM systems in all the experiments. Furthermore, in online SLAM tested at our site, it also significantly outperformed the other SLAM Systems.

Tele-operating aerial vehicles without any automated assistance is challenging due to various limitations, especially for inexperienced users. Autocomplete addresses this problem by automatically identifying and completing the user's intended motion. Such a framework uses machine learning to recognize and classify human inputs as one of a set of motion primitives, and then, if the human operator accepts, synthesizes the motion in order to complete the desired motion. This has been shown to improve the performance of the system and reduce operator workload. Previous Autocomplete systems focused on different 2D motions (line, arc, sine,..). However, since most UAVs tasks are in a 3D world, this paper introduces 3D Autocomplete for 3D motions. Moreover, the proposed framework presents just-in-time prediction of the 3D motions by proposing a change point detection technique, which allows the framework to autonomously identify when to conduct a prediction. Also, it deals with variable motion sizes. Real time simulation results show that the proposed framework is capable of predicting the user intentions after change point detection.

In Simultaneous Localization and Mapping (SLAM), quality maps play an important role beyond the implicit benefits they have in localization. For example, good quality SLAM maps could be used as ’pre-maps’ for generating higher semantic meaning and geometric structure, such as in 2D CAD drawings in which doors, windows, and other entities carry meaning. Unfortunately, in its raw form, the quality of a SLAM map is limited by several software and hardware related factors. This paper proposes to address this problem by converting SLAM maps to CAD maps based on a generative adversarial network (GAN). The paper also investigates the inverse problem, that of generating SLAM maps from 2D CAD drawings, when synthetic SLAM maps are needed. We do so by proposing a method based on an analytical model and another on GANs. Experiments demonstrate both qualitatively and quantitatively the success of the proposed approaches.

Remote controlling robots without any automated help is difficult due to various limitations. Autocomplete mitigates this difficulty by automatically detecting and completing the intended motions on robots from the input of the user. Such an approach can improve the system performance and reduce the load on the operator. Usually, recognizing intended motions is achieved using pre-trained Deep Learning (DL) models. In this paper, we introduce personalization to the autocomplete teleoperation framework when new operators take over by customizing the autocomplete DL model using incremental learning. Also, we tackle the problem of concept drift that arises in real-life applications; the data distribution of already learned classes may change in unforeseen ways as new observations of these classes come sequentially over time. We create and update an exemplar set using new observations of the classes online so that the model can be trained to adapt to the new observations. Several scenarios have been evaluated to balance the speed of learning with the accuracy of the model, and results demonstrate the effectiveness of the proposed models and their advantage in adapting to the specific operator versus our previous framework: personalization using transfer learning with full feedback.

Teleoperation of robots can be challenging, especially for novice users with little to no experience at such tasks. The difficulty is largely due to the numerous degrees of freedom users must control and their limited perception bandwidth. To help mitigate these challenges, we propose in this paper a solution which relies on artificial intelligence to understand user intended motion and then on mixed reality to communicate the estimated trajectories to the users in an intuitive manner. User intended motion is estimated using a deep learning network trained on a dataset of motion primitives. During teleoperation, the estimated motions are augmented onto a first-person live video feed from the robot. Finally, if a suggested motion is accepted by the user, the robot is driven along that trajectory in an autonomous manner. We validate our proposed mixed reality teleoperation scheme with simulation experiments on a drone and demonstrate, through subjective and objective evaluation, its advantages over other teleoperation methods.

Controlling and manning robots from a remote location is difficult because of the limitations one faces in perception and available degrees of actuation. Although humans can become skilled teleoperators, the amount of training time required to acquire such skills is typically very high. In this paper, we propose a novel solution (named Autocomplete) to aid novice teleoperators in manning robots adroitly. At the input side, Autocomplete relies on machine learning to detect and categorize human inputs as one from a group of motion primitives. Once a desired motion is recognized, at the actuation side an automated command replaces the human input in performing the desired action. So far, Autocomplete can recognize and synthesize lines, arcs, full circles, 3-D helices, and sine trajectories. Autocomplete was tested in simulation on the teleoperation of an unmanned aerial vehicle, and results demonstrate the advantages of the proposed solution versus manual steering.

In this paper, we study the value of using synthetically produced videos as training data for neural networks used for action categorization. Motivated by the fact that texture and background of a video play little to no significant roles in optical flow, we generated simplified textureless and background-less videos and utilized the synthetic data to train a Temporal Segment Network (TSN). The results demonstrated that augmenting TSN with simplified synthetic data improved the original network accuracy (68.5%), achieving 71.8% on HMDB-51 when adding 4,000 videos and 72.4% when adding 8,000 videos. Also, training using simplified synthetic videos alone on 25 classes of UCF-101 achieved 30.71% when trained on 2500 videos and 52.7% when trained on 5000 videos. Finally, results showed that when reducing the number of real videos of UCF-25 to 10% and combining them with synthetic videos, the accuracy drops to only 85.41%, compared to a drop to 77.4% when no synthetic data is added.

Have you ever wondered how a song might sound if performed by a different artist? In this work, we propose SCM-GAN, an end-to-end non-parallel song conversion system powered by generative adversarial and transfer learning, which allows users to listen to a selected target singer singing any song. SCM-GAN first separates songs into vocals and instrumental music using a U-Net network, then converts the vocal segments to the target singer using advanced CycleGAN-VC, before merging the converted vocals with their corresponding background music. SCM-GAN is first initialized with feature representations learned from a state-of-the-art voice-to-voice conversion and then trained on a dataset of non-parallel songs. After that, SCM-GAN is evaluated against a set of metrics including global variance GV and modulation spectra MS on the 24 Mel-cepstral coefficients (MCEPs). Transfer learning improves the GV by 35% and the MS by 13% on average. A subjective comparison is conducted to test the output's similarity to the target singer and its naturalness. Results show that the SCM-GAN's similarity between its output and the target reaches 69%, and its naturalness reaches 54%.

Whether for localization, path planning, or scene manipulation, complete and accurate scene reconstruction is an essential component of robotic operation. Due to their low cost and versatility, vision-based scene reconstruction methods have been the subject of research for decades. However, a major disadvantage of vision-based methods is that they require the scene to be populated with distinctive features that can be unambiguously matched across different images. In the absence of these features, such as in planar homogeneously painted surfaces, the scene reconstruction fails. This paper proposes a novel idea, where the user can virtually texturize planar surfaces at run-time to be used for the scene reconstruction. To do so, the corners of planes are tracked across the images and used to warp virtual texture patches to the correct perspective. Two methods are then proposed, one that actively tracks the corners as long as they are in view, and one that requires the camera poses to augment planes once their corners are no longer visible. The conducted experiments demonstrate the effectiveness of our approach as it increases the number of points in scene reconstructions. The end result is a denser scene reconstruction where textureless planes, typically not recovered in traditional methods, are reconstructed.

The inverted pendulum is by nature a dynamically unstable system and may be subjected to severe disturbances due to its environmental or loading conditions. This paper formulates a design for a nonlinear controller to balance a two-wheeled mobile robot (TWMR) based on Model Reference Adaptive Control. The proposed solution overcomes the limitations of control systems that rely on fixed parameter controllers. Given the nonlinear single-input multi-output (SIMO) nature of the TWMR platform, the proposed adaptive controller can handle non-linearities without the need for linearization, and inherently dealing with SIMO systems. By studying the influence that hidden dynamic effects can cause, we show the preference of the proposed controller over other designs. Simulation results demonstrate the applicability and efficiency of our proposed design, and experimental results validate the effectiveness of the proposed scheme in guaranteeing asymptotic output tracking, even in the presence of unknown disturbances.

In this work, we are proposing an intuitive Augmented SLAM method (A-SLAM) that allows the user to interact, in real-time, with a robot running SLAM to correct for pose and map errors. We built an AR application that works on HoloLens and allows the operator to view the robot's map superposed on the physical environment and edit it. Through map editing, the operator can account for errors affecting real environment's representation by adding navigation-forbidden areas to the map in addition to the ability to correct errors affecting the localization. The proposed system allows the operator to edit the robot's pose (based on SLAM request) and can be extended to sending navigation goals to the robot, viewing the planned path to evaluate it before execution, and teleoperating the robot. The proposed solution could be applied on any 2D-based SLAM algorithm and can easily be extended to 3D SLAM techniques. We validated our system through experimentation on pose correction and map editing. Experiments demonstrated that through A-SLAM, SLAM runtime is cut to half, post-processing of maps is totally eliminated, and high quality occupancy grid maps could be achieved with minimal added computational and hardware costs.

In this paper, we are proposing a collaborative SLAM system between a team of three heterogeneous agents: a robot, a human operator, and an augmented reality head mounted display (ARHMD). The system allows for online editing of a map produced by a robot running SLAM. Through hand gestures, the user can edit, in real time, the robot map that is augmented on top of the physical environment. Moreover, the proposed system leverages the built-in SLAM capabilities of the AR-HMD to correct the robot's map and map areas that are not yet discovered by the robot. Our method aims to combine the unique and complementary capabilities of each of the three different agents to produce the maximum possible mapping accuracy in the minimum amount of time. The proposed system is implemented on ROS and Unity. Experiments performed demonstrate the considerably superior SLAM outputs in terms of reducing mapping time, eliminating maps post-processing, and increasing mapping accuracy.

Monocular Odometry systems can be broadly categorized as being either Direct, Indirect, or a hybrid of both. While Indirect systems process an alternative image representation to compute geometric residuals, Direct methods process the image pixels directly to generate photometric residuals. Both paradigms have distinct but often complementary properties. This paper presents a Unified Formulation for Visual Odometry, referred to as UFVO, with the following key contributions: (1) a tight coupling of photometric (Direct) and geometric (Indirect) measurements using a joint multi-objective optimization, (2) the use of a utility function as a decision maker that incorporates prior knowledge on both paradigms, (3) descriptor sharing, where a feature can have more than one type of descriptor and its different descriptors are used for tracking and mapping, (4) the depth estimation of both corner features and pixel features within the same map using an inverse depth parametrization, and (5) a corner and pixel selection strategy that extracts both types of information, while promoting a uniform distribution over the image domain. Experiments show that our proposed system can handle large inter-frame motions, inherits the sub-pixel accuracy of direct methods, can run efficiently in real-time, can generate an Indirect map representation at a marginal computational cost when compared to traditional Indirect systems, all while outperforming state of the art in Direct, Indirect and hybrid systems.

In this paper we develop a real-time human augmented mapping system. This approach replaces the traditional offline post processing of maps by a user-friendly system allowing for online editing capabilities. A wide number of applications that acquire accurate mapping of the environment could benefit from such a solution. The proposed framework consists of two main parts: 2D map building using LIDAR, encoders, and IMU; and a user interface for human map augmentation. The first part is built over Gmapping ROS package, while the second is developed in Unity software. Realworld experiments validated the ability of our system to correct for sensor noise and various mapping errors, thus increasing the accuracy of the obtained maps without additional computational costs.

This paper proposes a novel technique to extract training data from free space in a scene using a stereo camera. The proposed technique exploits the projection of planes in the v-disparity image paired with Bayesian linear regression to reliably identify training image pixels belonging to free space in a scene. Unlike other methods in the literature, the algorithm does not require any prior training, has only one free parameter, and is shown to provide consistent results over a variety of terrains without the need for any manual tuning. The proposed method is compared to two other data extraction methods from the literature. Results of Support Vector classifiers using training data extracted by the proposed technique are superior in terms of quality and consistency of free space estimation. Furthermore, the computation time required by the proposed technique is shown to be smaller and more consistent than that of other training data extraction methods.

This paper proposes a novel technique for segmenting the ground plane and at the same time estimating the occupancy probability of each point in a scene. Using a stereo camera rig, our system first calculates a disparity map and transforms it to a v-disparity map, which is then filtered and processed to generate a corresponding probability field. The probability field generated is then used for precise segmentation of ground planes as well as for the generation of occupancy grids. Unlike what is proposed in the prior art, our system requires minimal initialization and is independent of the stereo sensor characteristics as well as the parameters of the disparity algorithm. More importantly, our technique does not require any prior assumption about the terrain visual characteristics. Experimental results using sequences of images from two different data sets are presented to validate the proposed methods.

This paper presents a novel visual servoing system that is simple, robust, and relatively fast. Our system utilizes depth information available from an RGB-D Microsoft Kinect, rather than a stereo camera that is known to fail in poorly textured environments. The technique is based on a two-phase controller. The first phase provides a coarse image plane alignment with a target image using image saliencies for registration. The second phase relies on a depth-map-error-minimization to further reduce the position error and precisely situate the robot. Based on the magnitude of error between actual and desired position the system activates one or both phases sequentially to successfully servo the robot to its destination. Experiments demonstrate the success of the proposed system for both large and small initial position errors.

This paper is concerned with gait generation for the simplest underactuated multi-link robot. This robot is realized by a two-link planar system that locomotes in a viscous environment due to the existence of friction pads along its body. The proposed solution of the motion planning problem yields both a control gait for the inter-link angle as well as the location of the friction pad to produce a desired motion of the system.

If we are to one day rely on robots as assistive devices they should be capable of mitigating the impact of random disturbances and avoid falling. Humans are surprisingly apt at remaining on their feet when pushed; they rely on reflexes such as bending the ankles and/or the hips, or by taking a step if the magnitude of the disturbance is relatively large. This paper presents a fall avoidance scheme that is capable of applying both ankle and hip strategies on a humanoid robot. While both strategies serve the same purpose, the hip strategy can absorb larger disturbances but has a higher energy overhead and should be avoided when it is not necessary. Our system is capable of detecting at the onset of a disturbance if an ankle or hip strategy is more appropriate. The decision is taken based on a 'decision surface' that is delimited by threshold values of the robot's state variables. The control is based on the intuitive Virtual Model Control (VMC) approach. The system is tested on a simulated robot developed under Gazebo. Results show successful fall avoidance with an ability to choose the optimum fall avoidance strategy.

SLAM in an outdoor environment using natural landmarks stands as the holy grail of SLAM algorithms. Segmenting landmarks from background clutter in such environments is difficult and vision, rather than laser, has a higher potential to perform such tasks due to the higher bandwidth of information it carries. There is a need to establish a benchmark upon which emerging vision SLAM algorithms can be assessed and compared. Towards this objective, this paper proposes the infrastructure for such a benchmark and discusses the issues involved in compiling it. Ego-motion information is extracted via a strap-down inertial measurement unit (IMU). Synchronized Global Positioning System (GPS), IMU, and surrounding images of an outdoor park environment are compiled into a database. IMU data in tested on an inertial navigation system (INS) dead-reckoning algorithm. The adequacy of the stereo image database is validated by extracting disparity maps of each stereo image in the database. IMU simulations show the necessity for visual SLAM to improve pose estimation. The complete data set, including GPS, IMU, and stereo images is available for downloading purposes

A robot perceives space in order to navigate, map, search and investigate its surroundings. One application area where humans interact with robots is search and rescue. The robot may have unique capabilities such as seeing outside the visible spectrum and being able to navigate in tight and hazardous spaces. In such an operation, the human may also be an effective search agent. Thus, it would be beneficial if the robot's spatial perception was conveyed to the human via a secondary peripheral modality. We have worked on developing a visual to tactile substitution device for people who are visually impaired or blind. We propose to use this similar technique to the remote robot in order that it convey its spatial perception to the human operator or co-worker.