The application of autonomous robotics to close-contact healthcare tasks, such as the nasopharyngeal (NP) swab test, has clear potential for reducing infection risks to staff and improving efficiency. We propose a control system that performs the NP swab test with a collaborative manipulator arm, guided by an instrumented end-effector that measures force and visual information. We assume a scenario where the patient is unrestrained, with hardware general enough for other types of close contact tasks. The system employs visual servo control to align the swab with the nostrils. A compliant joint velocity controller inserts the swab into the nasal cavity, following a planned trajectory adjusted with force-feedback. Fuzzy logic systems are designed to detect when the swab reaches the nasopharynx and enforce safety criteria. We validate the system using a second robotic arm that holds a nasal cavity phantom and simulates natural head motions. Extensive experiments identify controller configurations capable of effectively performing the NP swab test even with significant head motion.

Real-time SLAM with dense 3D mapping is computationally challenging, especially on resource-limited devices. The recent development of 3D Gaussian Splatting (3DGS) offers a promising approach for real-time dense 3D reconstruction. However, existing 3DGS-based SLAM systems struggle to balance hardware simplicity, speed, and map quality. Most systems excel in one or two of the aforementioned aspects but rarely achieve all. A key issue is the difficulty of initializing 3D Gaussians while concurrently conducting SLAM. To address these challenges, we present Monocular GSO (MGSO), a novel real-time SLAM system that integrates photometric SLAM with 3DGS. Photometric SLAM provides dense structured point clouds for 3DGS initialization, accelerating optimization and producing more efficient maps with fewer Gaussians. As a result, experiments show that our system generates reconstructions with a balance of quality, memory efficiency, and speed that outperforms the state-of-the-art. Furthermore, our system achieves all results using RGB inputs. We evaluate the Replica, TUM-RGBD, and EuRoC datasets against current live dense reconstruction systems. Not only do we surpass contemporary systems, but experiments also show that we maintain our performance on laptop hardware, making it a practical solution for robotics,$A / R$, and other real-time applications.

Simultaneous Localization and Mapping (SLAM) is a crucial task in various domains, including intelligent robotics, computer vision, and indoor navigation. Accurate and robust trajectory estimation is especially challenging in indoor environments due to the presence of feature-poor or repetitive scenes, limited visibility, and dynamic objects. Obtaining highly accurate machine platform odometry is also an important basis for solving the “last mile” problem in intelligent transportation. This paper proposes a novel algorithm that combines LiDAR-based SLAM, total station measurements, and graph optimization to optimize the robot’s trajectory in indoor environments. By integrating highly accurate positional data from total station measurements as additional constraints, the proposed method enhances the performance of indoor LiDAR SLAM, effectively addressing the challenges of drift and trajectory offsets. Moreover, the proposed algorithm can provide trajectory optimization even in the absence of loop closure detection, making it more robust and suitable for a broader range of indoor environments. Experimental results validated the effectiveness of the proposed approach in reducing drift and improving trajectory estimation for low-cost indoor LiDAR devices, demonstrating its potential in various applications such as autonomous navigation, facility management, and augmented reality. It provides targeted ground truth for autonomous driving of machine platforms in intelligent transportation scenarios.

This paper presents an integrated approach to Visual SLAM, merging online sequential photometric calibration within a Hybrid direct-indirect visual SLAM (H-SLAM). Photometric calibration helps normalize pixel intensity values under different lighting conditions, and thereby improves the direct component of our H-SLAM. A tangential benefit also results to the indirect component of H-SLAM given that the detected features are more stable across variable lighting conditions. Our proposed photometrically calibrated H-SLAM is tested on several datasets, including the TUM monoVO as well as on a dataset we created. Calibrated H-SLAM outperforms other state of the art direct, indirect, and hybrid Visual SLAM systems in all the experiments. Furthermore, in online SLAM tested at our site, it also significantly outperformed the other SLAM Systems.

Accurate estimation of human pose and the pose of interacting objects, like a hockey stick, is crucial for action recognition and performance analysis, particularly in sports. Existing methods capture the object along with the human in the bounding boxes, assuming all keypoints are visible within the bounding box. This necessitates larger bounding boxes to capture the object, introducing unnecessary visual features and hindering performance in real-world cluttered environments. We propose a simple image and text-based multimodal solution TokenCLIPose that addresses this limitation. Our approach focuses solely on human keypoints within the bounding box, treating objects as unseen. TokenCLIPose leverages the rich semantic representations endowed by language for inducing keypoint-specific context, even for occluded keypoints. We evaluate the performance of TokenCLIPose on a real-world Ice-Hockey dataset, and demonstrate its generalizability through zero-shot transfer to a smaller Lacrosse dataset. Additionally, we showcase its flexibility on CrowdPose, a popular occlusion benchmark with keypoints within the bounding box. Our method significantly improves over state-of-the-art approaches on all three datasets, with gains of 4.36\%, 2.35\%, and 3.8\%, respectively.

Effective road marking classification and segmentation play a pivotal role in advancing vehicle-to-everything (V2X) applications and refining road inventory databases. However, the irregular data formats and unordered permutation modes of 3D point clouds, along with the limited availability of large-scale datasets with point-level annotations, remain significant obstacles to designing deep learning-based networks with superior performance. To address these challenges, this paper proposes a novel multi-level feature optimization network structure, named MFPNet, and introduces two point cloud benchmarks, RdmkNet and Toronto-Rdmk, for road marking classification and segmentation in intricate urban environments. MFPNet is composed of three integral modules. First, the M-transformer module, consisting of three transformers obtained from different channels, fully captures rich point cloud background information and long-distance dependencies between objects. Then, the feature pooling aggregation module uses parallel structured pooling attention mechanisms to aggregate features captured by the M-transformer module, while the prediction refinement module further enhances the acquisition of semantic features. Comparative studies indicate that MFPNet can be embedded into general deep learning networks without changing their original network structures, significantly improving the accuracy of multiple baseline networks. Furthermore, extensive experiments demonstrate that the two newly-developed point cloud datasets are meaningful for road marking classification and segmentation tasks, contributing to the development of autonomous driving.

Recent text detection frameworks require several handcrafted components such as anchor generation, non-maximum suppression (NMS), or multiple processing stages (e.g. label generation) to detect arbitrarily shaped text images. In contrast, we propose an end-to-end trainable architecture based on Detection using Transformers (DETR), that outperforms previous state-of-the-art methods in arbitrary-shaped text detection.At its core, our proposed method leverages a bounding box loss function that accurately measures the arbitrary detected text regions’ changes in scale and aspect ratio. This is possible due to a hybrid shape representation made from Bezier curves, that are further split into piece-wise polygons. The proposed loss function is then a combination of a generalized-split-intersection-over-union loss defined over the piece-wise polygons, and regularized by a Smooth-ln regression over the Bezier curve’s control points.We evaluate our proposed model using Total-Text and CTW-1500 datasets for curved text, and MSRA-TD500 and ICDAR15 datasets for multi-oriented text, and show that the proposed method outperforms the previous state-of-the-art methods in arbitrary-shape text detection tasks.

Presented in this paper is a novel method for the mapping and semantic modeling of an underground parking lot using 3D point clouds collected by a low-cost Backpack Laser Scanning (BLS) or LiDAR system. Our method consists of two parts: a Simultaneous Localization and Mapping (SLAM) algorithm based on Sparse Point Clouds (SPC) and a semantic modeling algorithm based on a modified PointNet model. The main contributions of this paper are as follows: (1) a probability frontend framework for the alignment of point clouds using the local point cloud surface variance as the weight of registration, which modifies registration failure caused by the lack of features in sparse point clouds, (2) a robust submap-based strategy for loop closure detection and back-end optimization under sparse point clouds, and (3) a modified PointNet model for classifying the point clouds of underground parking lots into four categories: ceiling, floor, wall, others. Experimental results show that our SPC-SLAM algorithm achieves centimeter-level accuracy (0.09% trajectory error rate) after closed loop processing in a Global Navigation Satellite System (GNSS)-denied underground parking lot, and precision of 84.8% in semantic segmentation.

The backpack LiDAR system is a highly efficient device for indoor positioning and navigation. With the use of multi-sensor interactions inside of the backpack, it can solve the problem of undetermined trajectories or locations in a GNSS-denied environment. This paper presents a multi-sensor-based backpack LiDAR system for mapping in GNSS-denied environments. With this device, we solve the problem of inaccurate positioning caused by the inability to receive GNSS signals in a small-sized lidar portable device, also reduces the cumulative error caused by the IMU module through the analysis of the vibration pattern. Through the demonstration of the results, the method is of great significance for 3D reconstruction of the indoor environment and object detection.

Monocular Odometry systems can be broadly categorized as being either Direct, Indirect, or a hybrid of both. While Indirect systems process an alternative image representation to compute geometric residuals, Direct methods process the image pixels directly to generate photometric residuals. Both paradigms have distinct but often complementary properties. This paper presents a Unified Formulation for Visual Odometry, referred to as UFVO, with the following key contributions: (1) a tight coupling of photometric (Direct) and geometric (Indirect) measurements using a joint multi-objective optimization, (2) the use of a utility function as a decision maker that incorporates prior knowledge on both paradigms, (3) descriptor sharing, where a feature can have more than one type of descriptor and its different descriptors are used for tracking and mapping, (4) the depth estimation of both corner features and pixel features within the same map using an inverse depth parametrization, and (5) a corner and pixel selection strategy that extracts both types of information, while promoting a uniform distribution over the image domain. Experiments show that our proposed system can handle large inter-frame motions, inherits the sub-pixel accuracy of direct methods, can run efficiently in real-time, can generate an Indirect map representation at a marginal computational cost when compared to traditional Indirect systems, all while outperforming state of the art in Direct, Indirect and hybrid systems.

The reconstruction of high dynamic range (HDR) images via conventional camera systems and low dynamic range (LDR) images is a growing field of research in image acquisition. The radiance map associated with the HDR image of a scene is typically computed using multiple images of the same scene captured at different exposures (i.e., bracketed LDR imzages). This approach, though inexpensive, is sensitive to noise under high camera ISO. Each bracketed image is associated with a different level of noise due to the change in exposure time, and the noise is further amplified when tone-mapping the HDR image for display. A new framework is proposed to address the associated noise in the context of random fields. The estimation of the HDR image from a set of LDR images is formulated as a stochastically fully connected conditional random field where the spatial information is incorporated to compute the HDR value in combination with the LDR image values. Experimental results show that the proposed framework compensated the non-stationary ISO noise while preserving the boundaries in the estimated HDR images.

This paper presents a novel visual servoing system that is simple, robust, and relatively fast. Our system utilizes depth information available from an RGB-D Microsoft Kinect, rather than a stereo camera that is known to fail in poorly textured environments. The technique is based on a two-phase controller. The first phase provides a coarse image plane alignment with a target image using image saliencies for registration. The second phase relies on a depth-map-error-minimization to further reduce the position error and precisely situate the robot. Based on the magnitude of error between actual and desired position the system activates one or both phases sequentially to successfully servo the robot to its destination. Experiments demonstrate the success of the proposed system for both large and small initial position errors.

A novel statistical textural distinctiveness approach for robustly detecting salient regions in natural images is proposed. Rotational-invariant neighborhood-based textural representations are extracted and used to learn a set of representative texture atoms for defining a sparse texture model for the image. Based on the learnt sparse texture model, a weighted graphical model is constructed to characterize the statistical textural distinctiveness between all representative texture atom pairs. Finally, the saliency of each pixel in the image is computed based on the probability of occurrence of the representative texture atoms, their respective statistical textural distinctiveness based on the constructed graphical model, and general visual attentive constraints. Experimental results using a public natural image dataset and a variety of performance evaluation metrics show that the proposed approach provides interesting and promising results when compared to existing saliency detection methods.

The paper focuses on 3D structure and motion factorization from uncalibrated image sequences. A rank-4 affine factorization algorithm and a robust structure and motion factorization scheme are proposed to handle outlying and missing data. The novelty and main contribution of the paper are as follows: (i) The rank-4 factorization algorithm is a new addition to previous affine factorization family using rank-3 constraint; (ii) the outliers and image uncertainty are estimated directly from the image reprojection residuals; and (iii) the robust factorization scheme is proved empirically to be more efficient and accurate than other robust algorithms. Extensive experiments on synthetic data and real images validate the proposed approach.

This paper focuses on the problem of structure and motion recovery from uncalibrated image sequences. It has been empirically proven that image measurement uncertainties can be modeled spatially and temporally by virtue of reprojection residuals. Consequently, a spatial-and-temporal-weighted factorization (STWF) algorithm is proposed to handle significant noise contained in the tracking data. This paper presents three novelties and contributions. First, the image reprojection residual of a feature point is demonstrated to be generally proportional to the error magnitude associated with the image point. Second, the error distributions are estimated from a different perspective, that of the reprojection residuals. The image errors are modeled both spatially and temporally to cope with different kinds of uncertainties. Previous studies have considered only the spatial information. Third, based on the estimated error distributions, an STWF algorithm is proposed to improve the overall accuracy and robustness of traditional approaches. Unlike existing approaches, the proposed technique does not require prior information of image measurement and is easy to implement. Extensive experiments on synthetic data and real images validate the proposed method.

Spatio-temporal salient features can localize the local motion events and are used to represent video sequences for many computer vision tasks such as action recognition. The robust detection of these features under geometric variations such as affine transformation and view/scale changes is however an open problem. Existing methods use the same filter for both time and space and hence, perform an isotropic temporal filtering. A novel anisotropic temporal filter for better spatio-temporal feature detection is developed. The effect of symmetry and causality of the video filtering is investigated. Based on the positive results of precision and reproducibility tests, we propose the use of temporally asymmetric filtering for robust motion feature detection and action recognition.

This paper suggests using discrete feature displacements and optical flow simultaneously to determine the camera motion and its velocity. This is advantageous when the number of feature correspondences is low or when the feature correspondences are noisy. The reason is that usually the available optical flow data largely outnumbers the available feature correspondences data. It is also advantageous from the perspective of the instantaneous motion estimation because it gives better estimates for the camera velocity than those obtained from optical flow by itself. We propose a probabilistic framework capitalizing on the this idea. Monte-Carlo filtering is employed due to the non-linearities involved in the problem and to the non-Gaussianity of the measurements¿ probability distributions.

We represent face images by a set of triangular labeled graphs, each containing information on the appearance and geometry of a 3-tuple of face feature points. Our method automatically learns a model set and builds a graph space for each individual. A two-stage method for fast matching is developed, where in the first stage a maximum a posterior solution based on PCA factorization is used to efficiently prune the search space and select very few candidate model sets, and in the second stage a nearest neighborhood classifier is used to find the closest model graphs to the query image graphs. Finally, the probe image is assigned to the trained individual with the maximum number of references. Our proposed technique achieves perfect results on the ORL face set and an accuracy rate of 97.7% on the FERET face set, which shows the superiority of the proposed technique over all considered state-of-the-art methods (elastic bunch graph matching, LDA+PCA, Bayesian intra/extra-personal classifier, boosted Haar classifier)

Discerning depth from IR stereopsis is difficult because in general IR imagery does not contain sufficient features for left-right correspondence. We investigate the production of sparse disparity maps from uncalibrated infrared stereo images and argue that a dense depth field may not be attained directly from IR stereo images, but perhaps a sparse depth field may be obtained that can be interpolated to produce a dense depth field. In our proposed technique the sparse disparity map is produced by matching the stable features, extracted from the phase congruency model. A set of Log-Gabor wavelet coefficients is used to analyze and describe the extracted features for matching. The resulted sparse disparity map is then refined by triangular and epipolar geometrical constraints. In densifying the sparse disparity map, a watershed transformation is performed on the discontinuity map of the reference image to divide the image into several segments and finally the surface of each segment is reconstructed independently by fitting a thin-plate spline to its known disparities. Experiments on a set of IR stereo pairs lend credibility to the robustness of our IR stereo matching and surface reconstruction technique

SLAM in an outdoor environment using natural landmarks stands as the holy grail of SLAM algorithms. Segmenting landmarks from background clutter in such environments is difficult and vision, rather than laser, has a higher potential to perform such tasks due to the higher bandwidth of information it carries. There is a need to establish a benchmark upon which emerging vision SLAM algorithms can be assessed and compared. Towards this objective, this paper proposes the infrastructure for such a benchmark and discusses the issues involved in compiling it. Ego-motion information is extracted via a strap-down inertial measurement unit (IMU). Synchronized Global Positioning System (GPS), IMU, and surrounding images of an outdoor park environment are compiled into a database. IMU data in tested on an inertial navigation system (INS) dead-reckoning algorithm. The adequacy of the stereo image database is validated by extracting disparity maps of each stereo image in the database. IMU simulations show the necessity for visual SLAM to improve pose estimation. The complete data set, including GPS, IMU, and stereo images is available for downloading purposes

Most haptic educational papers focus on using haptic devices as an aid to teaching. We introduced the use of vibro-tactile communication as well as tactile and kinesthetic sensing and their combination to high school students in order to introduce them to the field of haptics and its applications. The effectiveness of our approach for introducing the concepts as well as the technical issues associated with haptics to this age group was relatively successful. Initially, various physiological response properties of the human tactile system, including the kinesthetic component as well as the tactual (force, texture) properties were presented. The students were then exposed to theory, hardware (sensors, actuators) interfacing as well as the software to close the loop between sensing and haptic feedback after initially investigating open loop configurations. Vibro-tactile (i.e., pager motors) were used in the labs as chiefly communication devices to convey either pan-tilt from an accelerometer (kinesthetic) or the intensity of force exerted on a force pressure sensor. The students enjoyed the experience, especially the hands-on nature of the project and their only major complaint was that there was too much programming. Our future intent is to abstract the building blocks, simplify the software interfacing so as to allow students of various levels and capabilities to effectively be creative and imaginative in developing various haptic communication applications and tools

A robot perceives space in order to navigate, map, search and investigate its surroundings. One application area where humans interact with robots is search and rescue. The robot may have unique capabilities such as seeing outside the visible spectrum and being able to navigate in tight and hazardous spaces. In such an operation, the human may also be an effective search agent. Thus, it would be beneficial if the robot's spatial perception was conveyed to the human via a secondary peripheral modality. We have worked on developing a visual to tactile substitution device for people who are visually impaired or blind. We propose to use this similar technique to the remote robot in order that it convey its spatial perception to the human operator or co-worker.

Dynamic planning involves continuously updating a map by sensing changes in the environment and planning appropriate actions, with all tasks sharing common computational resources. We use harmonic functions for dynamic planning. Analogous representations of harmonic functions as Markov chains and resistor networks are used to develop the notion of escape probability and energy dissipation. These measures are used to indicate convergence (event that permits resources to be devote to non-planning tasks) more robustly than monitoring maximum or average field changes between iterations. The convergence of the harmonic function is related quadratically to the number of grid elements. An example of an irregular sampling strategy - quad tree - is developed for harmonic functions, which is complete yet imprecise. Quad trees are not a sufficient sampling strategy for addressing the exponential growth of multiple dimensions and therefore current investigations include other sampling strategies or dimensional parallelization.

One of our major research focus areas is real-time visual tracking and monitoring of moving and static objects in a video sequence. In particular we are interested in (1) object localization (also referred to as the focus of attention) which involves identifying the object of interest, (2) tracking the object using a model of the object which was initiated in step 1, and (3) understanding the accumulation of movements of the object over time (i.e., behavior). The objects of interiest for the purpose of this proposal are pigs. Automatically monitoring pigs via a non-invasively placed camera in their pens is interesting because the pigs are monitored in their natural habitat. Visual tracking involves modelling the object of interest and keeping track of its position and orientation through time. Issues include tracker recovery from error and preventing the tracker from jumping to other pigs. we have been able to demonstrate tracking pigs at about 10-15 Hz, however, the tracker tends to drift off the target eventually. We have only experimented with a single pig but our initial tests indicate that we can probably track at least 10 pigs simultaneously. Some unknowns include determining how quickly the pigs move and the type of motions, including quick jerky movements. Our preliminary investigations revealed that a blob tracker is insufficient for producing accurate traces.

Sensor-based discovery (i.e. dynamic) path planning is problematic because the path needs to be continually recomputed as new information is discovered. A process-based client-server approach is presented that permits concurrent sensor-based map and localization-correction updates as well as concurrent path computation and execution. A potential function is created by solving Laplace's equation (i.e. a harmonic function) using an iteration kernal convolved with an occupancy-grid representation of the current free space. The path produced (i.e. by steepest gradient descent on the harmonic function) is optimal in the sense of globally minimizing the distance to the goal as well as locally minimizing a hitting probability. This helps alleviate the influence of uncertainty on path planning. On a regular grid, the computation of the harmonic function is linear in the total number of grid elements, thus constraining this planner to be local. A global planner-provided that an a priori CAD map of the fixed objects exists-provides information to the local planner about the effects of a global goal. An algorithm for the three typical local scenarios found in indoor office-like environments is presented showing how a global goal is projected into the local context for each case. Since objects are sensed on-the-fly it is possible to discover that passage through a hallway or room may be blocked. A zero gradient vector in the local potential function is used to signal a blocked passage and subsequently initiate global replanning. The technique has been used to control a Nomadics 200 robot.

Sensor-based discovery path planning is problematic because the path needs to be continually recomputed as new information is discovered. A process-based client-server approach is presented that permits concurrent sensor-based map updates, robot localization corrections, as well as concurrent path computation and execution. A harmonic function is constantly recomputed using an iteration kernel on an occupancy-grid representation of what is known about the world at any current time. The path produced (i.e., by steepest gradient descent on the harmonic function) is optimal in the sense of minimizing the distance to the goal as well as minimizing the hitting probability. This helps alleviate the influence of uncertainty on path planning. In addition, the computation time for generating the path is insignificant provided that the harmonic function has converged. On a regular grid, the computation of the harmonic function is linear in the total number of grid elements. A quad-tree representation is proposed to help minimize the computation time by reducing the number of grid elements and minimally representing large spaces void of obstacles and goals. The computation time is found to be approximately reduced by a factor of n/sup 2/, where n is the ratio of grid elements in the regular grid divided by the number in the reduced quad representation.

Interaction between robots and humans should be at a level which is accessible and natural for human operators. A lexicon template is proposed for specifying commands for 2D mobile robot navigational tasks. The language lexicon is a minimal spanning semantic set for human 2D navigational tasks. The task command lexicon consists of a verb, destination, direction and a speed. The destination is a location in the environment defined by a geometric model positioned at a particular spatial location in a globally-referenced Cartesian coordinate space. The task command lexicon has been used as a language template for specifying commands for the SPOTT mobile robot control architecture. It is not the intent that the template serve as the only commands that the robot recognizes, but rather as an internal language that gets mapped onto planning and control constructs. A speech recognition system maps spoken commands onto the proposed internal natural language command template.