This study addresses the challenge of performing visual localization in demanding conditions such as night-time scenarios, adverse weather, and seasonal changes. While many prior studies have focused on improving image matching performance to facilitate reliable dense keypoint matching between images, existing methods often heavily rely on predefined feature points on a reconstructed 3D model. Consequently, they tend to overlook unobserved keypoints during the matching process. Therefore, dense keypoint matches are not fully exploited, leading to a no-table reduction in accuracy, particularly in noisy scenes. To tackle this issue, we propose a novel localization method that extracts reliable semi-dense 2D-3D matching points based on dense keypoint matches. This approach involves regressing semi-dense 2D keypoints into 3D scene coordinates using a point inference network. The network utilizes both geometric and visual cues to effectively in-fer 3D coordinates for unobserved keypoints from the observed ones. The abundance of matching information significantly enhances the accuracy of camera pose estimation, even in scenarios involving noisy or sparse 3D models. Comprehensive evaluations demonstrate that the proposed method outperforms other methods in challenging scenes and achieves competitive results in large-scale visual localization benchmarks. The code will be available at https://github.com/TruongKhang/DeViLoc.

Wearable technology has transformed sports analytics, offering new dimensions in enhancing player experience. Yet, many solutions involve cumbersome setups that inhibit natural motion. In tennis, existing products require sensors on the racket or dominant arm, causing distractions and discomfort. We propose Silent Impact, a novel and user-friendly system that analyzes tennis shots using a sensor placed on the passive arm. Collecting Inertial Measurement Unit sensor data from 20 recreational tennis players, we developed neural networks that exclusively utilize passive arm data to detect and classify six shots, achieving a classification accuracy of 88.2% and a detection F1 score of 86.0%, comparable to the dominant arm. These models were then incorporated into an end-to-end prototype, which records passive arm motion through a smartwatch and displays a summary of shots on a mobile app. User study (N=10) showed that participants felt less burdened physically and mentally using Silent Impact on the passive arm. Overall, our research establishes the passive arm as an effective, comfortable alternative for tennis shot analysis, advancing user-friendly sports analytics.

For stroke survivors, diminished hand functions limit their ability to perform activities of daily living (ADLs). Recently, soft-robotic gloves have assisted stroke survivors in active rehabilitation by facilitating their finger movements based on intentions expressed through biosignals, such as electromyogram and electroencephalogram. In this regard, helping stroke survivors actively train multiple hand postures can improve hand functions required for ADLs. However, detecting intentions regarding multiple hand postures remains challenging, often resulting in low online classification performance. To address this, we propose a hand rehabilitation system comprising a vision-based intention detection framework and 8-degree-of-freedom soft-robotic glove. Our proposed framework, depth enhanced hand posture intention network, analyzes images and depths data observing users' arm behavior and hand-object interactions to predict intentions for multiple hand postures. The 8-degrees-of-freedom soft-robotic glove facilitates flexion and extension of individual fingers to help users perform desired hand postures. To support active rehabilitation, we operate our glove to facilitate user's finger movements when the user exerts effort to generate desired hand postures. We test our system on a real-time pick and place task involving five hand postures most commonly utilized in ADLs. Our vision-based system could predict and facilitate the desired hand postures for five healthy individuals and three stroke survivors with average accuracy of 90.4 ± 3.6% and 80.3 ± 4.6%, respectively, outperforming methods reported in previous studies.

This study tackles image matching in difficult scenarios, such as scenes with significant variations or limited texture, with a strong emphasis on computational efficiency. Previous studies have attempted to address this challenge by encoding global scene contexts using Transformers. However, these approaches have high computational costs and may not capture sufficient high-level contextual information, such as spatial structures or semantic shapes. To overcome these limitations, we propose a novel image-matching method that leverages a topic-modeling strategy to capture high-level contexts in images. Our method represents each image as a multinomial distribution over topics, where each topic represents semantic structures. By incorporating these topics, we can effectively capture comprehensive context information and obtain discriminative and high-quality features. Notably, our coarse-level matching network enhances efficiency by employing attention layers only to fixed-sized topics and small-sized features. Finally, we design a dynamic feature refinement network for precise results at a finer matching stage. Through extensive experiments, we have demonstrated the superiority of our method in challenging scenarios. Specifically, our method ranks in the top 9% in the Image Matching Challenge 2023 without using ensemble techniques. Additionally, we achieve an approximately 50% reduction in computational costs compared to other Transformer-based methods. Code is available at https://github.com/TruongKhang/TopicFM.

Generating actionable algorithmic recourse requires understanding each user’s preferences. Users provide their relevant information, and the system uses it to generate recourse that can be easily followed by individual users. To gain insight into users’ perceptions and experiences of this novel form of interaction, we developed a prototype that enables users to provide the required information for algorithmic recourse customization. With the prototype, we conducted a user study where participants customized the recourse. Through both quantitative and qualitative analysis, we found that: (1) repetitive user-AI interaction not only enables users to customize the recourse but also explore other possibilities, (2) users prefer recourse customization method that offers high controllability and understandability, and (3) degree of customization users want depends on various factors. With these findings, we discuss the implications for systems that aim to provide actionable algorithmic recourse in real-life situations.

This study addresses an image-matching problem in challenging cases, such as large scene variations or textureless scenes. To gain robustness to such situations, most previous studies have attempted to encode the global contexts of a scene via graph neural networks or transformers. However, these contexts do not explicitly represent high-level contextual information, such as structural shapes or semantic instances; therefore, the encoded features are still not sufficiently discriminative in challenging scenes. We propose a novel image-matching method that applies a topic-modeling strategy to encode high-level contexts in images. The proposed method trains latent semantic instances called topics. It explicitly models an image as a multinomial distribution of topics, and then performs probabilistic feature matching. This approach improves the robustness of matching by focusing on the same semantic areas between the images. In addition, the inferred topics provide interpretability for matching the results, making our method explainable. Extensive experiments on outdoor and indoor datasets show that our method outperforms other state-of-the-art methods, particularly in challenging cases.

Human emotions and behaviors are reciprocal components that shape each other in everyday life. While the past research on each element has made use of various physiological sensors in many ways, their interactive relationship in the context of daily life has not yet been explored. In this work, we present a wearable affective life-log system (ALIS) that is robust as well as easy to use in daily life to accurately detect emotional changes and determine the cause-and-effect relationship between emotions and emotional situations in users' lives. The proposed system records how a user feels in certain situations during long-term activities using physiological sensors. Based on the long-term monitoring, the system analyzes how the contexts of the user's life affect his/her emotional changes and builds causal structures between emotions and observable behaviors in daily situations. Furthermore, we demonstrate that the proposed system enables us to build causal structures to find individual sources of mental relief suited to negative situations in school life.

Learning from human feedback using event-related electroencephalography (EEG) signals has attracted extensive attention recently owing to their intuitive communication ability by decoding user intentions. However, this approach requires users to perform specified tasks and their success or failure. In addition, the amount of attention needed for decision-making increases with the task difficulty, decreasing human feedback quality over time because of fatigue. Consequently, this can reduce the interaction quality and can even cause interaction breakdowns. To overcome these limitations and enable the interaction of robots with higher complexity tasks, we propose a closed-loop control system that learns affective responses to robot behaviors and provides natural feedback to optimize robot parameters for smoothing the next action. Experimental results demonstrate our affect-driven closed-loop control system yielded better affective outcomes and task performance than an open-loop system with correlated neuroscientific characteristics of EEG signals, thus enhancing the quality of human-robot interaction.

Vision aided dynamic exploration on bipedal robots poses an integrated challenge for perception and control. Rapid walking motions as well as the vibrations caused by the landing-foot contact-force introduce critical uncertainty in the visual-inertial system, which can cause the robot to misplace its feet placing on complex terrains and even fall over. In this paper, we present a streamlined integration of an efficient geometric footstep planner and the corresponding walking controller for a humanoid robot to dynamically walk across rough terrain at speeds up to 0.3 m/s. To handle perception uncertainty that arises during dynamic locomotion, we present a geometric safety scoring method in our footstep planner to optimally select feasible path candidates. In addition, the real-time performance of the perception pipeline allows for reactive locomotion such as generating a new corresponding swing leg trajectory in mid-gait if a sudden change in the terrain is detected. The proposed perception-control pipeline is evaluated and demonstrated with real experiments using a full-scale humanoid to traverse across various terrains.

We propose a rank-based metric learning method by leveraging a concept of the Riemannian Potato for better separating non-linear data. By exploring the geometric properties of Riemannian manifolds, the proposed loss function optimizes the measure of dispersion using the distribution of Riemannian distances between a reference sample and neighbors and builds a ranked list according to the similarities. We show the proposed function can learn a hypersphere for each class, preserving the similarity structure inside it on Riemannian manifold. As a result, compared with Euclidean distance-based metric, our method can further jointly reduce the intra-class distances and enlarge the inter-class distances for learned features, consistently outperforming state-of-the-art methods on three widely used non- linear datasets.

Multi-view stereo (MVS) algorithms have been commonly used to model large-scale structures. When processing MVS, image acquisition is an important issue because its reconstruction quality depends heavily on the acquired images. Recently, an explore-then-exploit strategy has been used to acquire images for MVS. This method first constructs a coarse model by exploring an entire scene using a pre-allocated camera trajectory. Then, it rescans the unreconstructed regions from the coarse model. However, this strategy is inefficient because of the frequent overlap of the initial and rescanning trajectories. Furthermore, given the complete coverage of images, MVS algorithms do not guarantee an accurate reconstruction result.In this study, we propose a novel view path-planning method based on an online MVS system. This method aims to incrementally construct the target three-dimensional (3D) model in real time. View paths are continually planned based on online feedbacks from the partially constructed model. The obtained paths fully cover low-quality surfaces while maximizing the reconstruction performance of MVS. Experimental results demonstrate that the proposed method can construct high quality 3D models with one exploration trial, without any rescanning trial as in the explore-then-exploit method.

Here we present a robust physiological model for the recognition of human emotions, called Deep Physiological Affect Network. This model is based on a convolutional long short-term memory (ConvLSTM) network and a new temporal margin-based loss function. Formulating the emotion recognition problem as a spectral-temporal sequence classification problem of bipolar EEG signals underlying brain lateralization and photoplethysmogram signals, the proposed model improves the performance of emotion recognition. Specifically, the new loss function allows the model to be more confident as it observes more of specific feelings while training ConvLSTM models. The function is designed to result in penalties for the violation of such confidence. Our experiments on a public dataset show that our deep physiological learning technology significantly increases the recognition rate of state-of-the-art techniques by 15.96 percent increase in accuracy. An extensive analysis of the relationship between participants' emotion ratings and physiological changes in brain lateralization function during the experiment is also presented.

This paper proposes a two-factor authentication system that utilizes the knowledge factor: the knowledge of client's acquaintances as the key and inherence factor: P300 ERP responses to the visual stimuli as the medium. The system works by presenting a sequence of human photographs consisting of random people photographs mixed with a few of client's acquaintances photographs that trigger P300 responses. The system then verifies the client by considering the correctness of P300 responses to the client's acquaintances photographs. The proposed system achieves an error rate of nearly zero outperforming other brainsignal-based systems and has advantages over other conventional systems in the situations where the key is exposed to the imposer.

This paper determines the weight decay parameter value of a deep convolutional neural network (CNN) that yields a good generalization. To obtain such a CNN in practice, numerical trials with different weight decay values are needed. However, the larger the CNN architecture is, the higher is the computational cost of the trials. To address this problem, this paper formulates an analytical solution for the decay parameter through a proposed objective function in conjunction with Bayesian probability distributions. For computational efficiency, a novel method to approximate this solution is suggested. This method uses a small amount of information in the Hessian matrix. Theoretically, the approximate solution is guaranteed by a provable bound and is obtained by a proposed algorithm, where its time complexity is linear in terms of both the depth and width of the CNN. The bound provides a consistent result for the proposed learning scheme. By reducing the computational cost of determining the decay value, the approximation allows for the fast investigation of a deep CNN (DCNN) which yields a small generalization error. Experimental results show that our assumption verified with different DCNNs is suitable for real-world image data sets. In addition, the proposed method significantly reduces the time cost of learning with setting the weight decay parameter while achieving good classification performances.

In this study, we addressed a path planning problem of a mobile robot to construct highly accurate 3D models of an unknown environment. Most studies have focused on exploration approaches, which find the most informative viewpoint or trajectories by analyzing a volumetric map. However, the completion of a volumetric map does not necessarily describe the completion of a 3D model. A highly complicated structure sometimes cannot be represented as a volumetric model. We propose a novel exploration algorithm that considers not only a volumetric map but also reconstructed surfaces. Unlike previous approaches, we evaluate the model completeness according to the quality of the reconstructed surfaces and extract low-confidence surfaces. The surface information is used to guide the computation of the exploration path. Experimental results showed that the proposed algorithm performed better than other state-of-the-art exploration methods and especially improved the completeness and confidence of the 3D models.

Most user interfaces require motion; the motor-impaired therefore do not have many options to choose from. Electroencephalography (EEG) and the analysis of eye movement are two of the commonly proposed methods for enhancing the user experiences of the motor-impaired. In this paper, we propose an electrooculography (EOG)/EEG-based hybrid BCI that combines the strengths of EOG and EEG by using them simultaneously. We have shown the effectiveness of EOG/EEG-based hybrid BCIs by implementing the proposed interface and applying it to a chess game. Through this paper, we hope to provide improved interface systems for patients who have physical disabilities.

Positioning is recognized as an important feature in cellular networks due to regulatory requirements on emergency calls, but also to its potential for massive commercial applications, such as intelligent transportation, entertainment, industry automation, robotics, remote operation, healthcare, smart parking, etc. The standardization of the fifth generation (5G) has started, and one fundamental feature is beamforming, where it is possible to support both connected and idle mode users via fixed or flexible beams. 5G positioning is yet to be defined, and in this paper we show positioning results based on data from beam tracking and uplink timing alignment in a 5G radio access testbed. This specific testbed is operating at a carrier frequency of 28 GHz and is deployed with multiple transmission points at a race track to show 5G performance at different vehicular speeds. In this paper, positioning is based on estimated angle of arrival (AoA) using the MUltiple SIgnal Classification (MUSIC) algorithm and estimated range using the uplink time alignment procedure. The position estimation is evaluated with Global Positioning System (GPS) data as reference. The evaluation studies show that horizontal position error at 70 percentile is 1 meter for 10 km/h tested speed and 3 meters for 120 km/h tested speed.

In this paper, we propose a novel algorithm for planning exploration paths to generate 3D models of unknown environments by using a micro-aerial vehicle (MAV). Our algorithm initially determines a next-best-view (NBV) that maximizes information gain and plans a collision-free path to reach the NBV. Along the path, the MAV explores the greatest unknown area although it sometimes misses minor unreconstructed region, such as a hole or a sparse surface. To cover such a region, we propose an online inspection algorithm that consistently provides an optimal coverage path toward the NBV in real time. The algorithm iteratively refines an inspection path according to the acquired information until the modeling of a specific local area is complete. We evaluated the proposed algorithm by comparing it with other state-of-the-art approaches through simulated experiments. The results show that our algorithm outperforms the other approaches in both exploration and 3D modeling scenarios.

The coming 5G standard aims to support high mobility use cases where the user travels at high speeds in a car or a train. In this paper, measurement results are presented from high speed vehicle tests using a 5G radio access test system with beam tracking operating at a carrier frequency of 28 GHz. The tests were performed in a multi-transmission point deployment, where transmission point (TP) switching, beam tracking and operation at high vehicle speeds were successfully demonstrated. Beam tracking was demonstrated at a vehicle speed of 170 km/h while providing a downlink (DL) throughput of 3.6 Gbps. Furthermore, successful TP switching was demonstrated at vehicle speeds exceeding 160 km/h while maintaining a downlink bitrate of over 1 Gbps. Different user speeds show only a small impact in user throughput as long as the same beam is used.

This work suggests a supervised hierarchical Bayesian model for surface electromyography (sEMG)-based motion classification and its strategy analysis. The proposed model unifies the optimal feature extraction and classification through probabilistic inference and learning by identifying the latent neural states (LNSs) that govern a collection of sEMG signals. In addition, the inference step provides an approach to identify distinct muscle activation strategies according to sEMG patterns based on LNSs. To validate the model, nine-class classification using four sEMG sensors on the limb motions is tested. The model performance is evaluated with relatively high and low activation levels, generalized classification across subjects and online classification. The model, based on LNSs to capture various motions, is assessed with respect to activation levels, individual subjects and transition during online classification. Our approach cannot only classify sEMG patterns, but also provide the interpretation of sEMG strategic patterns. This work supports the potential of the proposed model for sEMG control-based applications.

This paper presents a new approach to the incremental online learning of behaviors by a robot from multiple kinesthetic teaching trials. The approach enables a robot to refine and reproduce a specific behavior every time a new teaching trial is provided and to decide autonomously whether to accept or reject each trial. The robot neglects bad teaching trials and learns a behavior based on adequate teaching trials. The framework of this approach consists of the projection of motion data to a latent space and the description of motion data in a Gaussian mixture model (GMM). To realize the incremental online learning, the latent space and the GMM are refined incrementally after each proper teaching trial. The trial data are discarded after being used. The number of Gaussian components in the GMM is not initially fixed but is autonomously selected by the robot over the trials. The proposed method is more suitable for practical human-robot interaction. The experiments with a humanoid robot show the feasibility of the approach. We demonstrate that the robot can incrementally refine and reproduce learned behaviors that accurately represent the essential characteristics of the teaching trials through our learning algorithm and that it can reject erroneous teaching trials to improve learning performance.

The brain-computer interface (BCI) technique is a novel control interface to translate human intentions into appropriate motion commands for robotic systems. The aim of this study is to apply an asynchronous direct-control system for humanoid robot navigation using an electroencephalograph (EEG), based active BCI. The experimental procedures consist of offline training, online feedback testing, and real-time control sessions. The amplitude features from EEGs are extracted using power spectral analysis, while informative feature components are selected based on the Fisher ratio. The two classifiers are hierarchically structured to identify human intentions and trained to build an asynchronous BCI system. For the performance test, five healthy subjects controlled a humanoid robot navigation to reach a target goal in an indoor maze by using their EEGs based on real-time images obtained from a camera on the head of the robot. The experimental results showed that the subjects successfully controlled the humanoid robot in the indoor maze and reached the goal by using the proposed asynchronous EEG-based active BCI system.

This study proposes an asynchronous noninvasive Brain Computer Interface (BCI) -based navigation system for a humanoid robot, which can behave similarly to a human. In the experimental procedure, each subject is asked to undertake three different sessions: offline training, an online feedback test, and real-time control of a humanoid robot in an indoor maze. During the offline training session, amplitude features from the EEG are extracted using auto-regressive frequency analysis with a Laplacian filter. The optimal feature components are selected by using the Fisher ratio and the linear discriminant analysis (LDA) distance metric. Two classifiers are hierarchically set to build the asynchronous BCI system. During the online test session, the trained BCI system translates a subject's ongoing EEG into four mental states: rest, left-hand imagery, right-hand imagery, and foot imagery. Event-by-event analysis is applied to evaluate the performance of the BCI system. If the test performance is consistently satisfactory, the subject executes the real-time control experiments. During the navigation experiments, the subject controls the robot in an indoor maze using the BCI system while surveying the environment through visual feedback. The results show that BCI control was comparable to manual control with a performance ratio of 81%. The evaluation of the results validates the feasibility and power of the proposed system.

This paper presents a design of a thermal SMA based simple small-sized and low-weight crawling robot, mimicking the crawling motion mechanism of Caenorhabditis elegans (C. elegans). Properties of the thermal SMA are similar to those of C. elegans muscle, which enables us to generate biologically relevant undulating motions. Each of 12 body segments composed of a pair of actuators is designed to be serially connected via a link that includes a motion control unit. Microcontroller is used to implement a simple sequential mode-based motion control scheme. Computer simulation and experimental results with a four segment prototype demonstrate the feasibility of the proposed robot design and control mechanism.

Most of the previous work on non-invasive brain-computer interfaces (BCIs) has been focused on feature extraction and classification algorithms to achieve high performance for the communication between the brain and the computer. While significant progress has been made in the lower layer of the BCI system, the issues in the higher layer have not been sufficiently addressed. Existing P300-based BCI systems, for example the P300 speller, use a random order of stimulus sequence for eliciting P300 signal for identifying users' intentions. This paper is about computing an optimal sequence of stimulus in order to minimize the number of stimuli, hence improving the performance. To accomplish this, we model the problem as a partially observable Markov decision process (POMDP), which is a model for planning in partially observable stochastic environments. Through simulation and human subject experiments, we show that our approach achieves a significant performance improvement in terms of the success rate and the bit rate.

This letter studies the properties of the random neural networks (RNNs) with state-dependent firing neurons. It is assumed that the times between successive signal emissions of a neuron are dependent on the neuron potential. Under certain conditions, the networks keep the simple product form of stationary solutions and exhibit enhanced capacity of adjusting the probability distribution of the neuron states. It is demonstrated that desired associative memory states can be stored in the networks.