We present a novel 3D reconstruction-based SLAM (Simultaneous Localization and Mapping) approach for robots that leverage multimodal sensory input data, including a camera and a 2D lidar. By integrating these inputs with the gaussian splatting technique, our method significantly enhances performance over traditional SLAM approaches. Traditional SLAM techniques often struggle with the limitations of monocular vision and fail to accurately map and locate objects in dynamic and cluttered environments. Purely relying on camera to localize the robot and map creation is challenging in the presence of dynamic obstacles in the scene. To address this, we proposed a multimodal sensor fusion-based 3D reconstruction. Our approach employs lidar-based localization to achieve precise positioning of both the camera and the robot, while utilizing the gaussian splatting technique for robust environmental mapping and 3D reconstruction. This approach is robust to dynamic obstacles in the scene. We have conducted extensive experiments in various real-world and simulated environments, demonstrating that our method not only outperforms traditional monocular SLAM approaches but also achieves higher accuracy in terms of localization and constructed map. Our results demonstrate substantial improvements in 3D reconstruction for mobile robots, achieving reduced computational load, higher FPS and enhanced scaling accuracy.

Accurate 3d reconstruction capturing the fine details of an object's shape is essential for tasks such as automated assembly, inspection, and quality control. While monocular cameras provide broad visual structure but often miss critical surface details and depth accuracy in underexposed or occluded environments. Tactile sensors offer precise, localized depth information, capturing fine textures, yet exploring varied curvature surfaces with only tactile input remains challenging. To address this, the paper proposes a blind surface exploration method for convex objects using a set of sequential controllers to efficiently guide the manipulator's interaction with surfaces featuring sharp edge changes. This approach ensures precise tactile exploration, leading to highly detailed surface reconstruction. With the controller employed, the algorithm was able to move along the surface while maintaining contact along normal and reconstruct the object with IoU as high as$\mathbf{9 1 \%}$for objects with sharp edges.

Individuals with Severe Speech and Motor Impairment (SSMI) struggle to interact with their surroundings due to physical and communicative limitations. To address these challenges, this paper presents a gaze-controlled robotic system that helps SSMI users perform stamp printing tasks. The system includes gaze-controlled interfaces and a robotic arm with a gripper, designed specifically for SSMI users to enhance accessibility and interaction. User studies with gazecontrolled interfaces such as video see-through (VST), video pass-through (VPT), and optical see-through (OST) displays demonstrated the system's effectiveness. Results showed that VST had the average stamping time of$28.45 s(SD = 15.44s)$and the average stamp count$7.36(SD = 3.83)$, outperforming VPT and OST.

Rapid aiming movement involves quick, accurate, pre-programmed motions used in the context of human-computer and human-robot interaction. It incorporates target forecasting to minimize the duration of tasks requiring rapid aiming. Applications include predicting target icons in UI design, driver intent in automotive technology, and human intent during human-robot collaboration. Conventional approaches often fail to capture human preferences accurately, leading to low prediction accuracy. This work explores an Inverse Reinforcement Learning (IRL)-based system for forecasting human hand movements and intended targets during rapid aiming. Sampling-based Maximum Entropy IRL (SMEIRL) with a sampler and Maximum Entropy Deep IRL (MEDIRL) algorithms were evaluated for prediction accuracy. The proposed sampler efficiently generates sample trajectories for rapid aiming tasks. User studies were conducted to assess target prediction during two tasks involving rapid aiming movement: (1) Pointing in Virtual Reality (VR) and Mixed Reality (MR), and (2) Human-robot handovers. A multimodal target prediction algorithm was analyzed for swift and accurate anticipation of the intended target, considering both hand and eye gaze. Results demonstrate that the proposed approach achieves a prediction accuracy of 98% in MR and 96% in VR for the pointing task using SMEIRL. During human-robot handover task, prediction accuracy using MEDIRL reached 99.9% when less than 20% and 40% of the task was left using only hand motion or both hand and eye gaze, respectively, surpassing state-of-the-art methods using Path Integral-IRL (PI-IRL), Recurrent Neural Network-Inverse Kinematics-Modified Kalman Filtering (RNNIK-MKF), Bayesian Predictor for Human Motion Trajectory (BP-HMT), and Classical kinematics of motion ( \(CM_{k=5}\) ).

In shared autonomy, human-robot handover for object delivery is crucial. Accurate robot predictions of human hand motion and intentions enhance collaboration efficiency. However, low prediction accuracy increases mental and physical demands on the user. In this work, we propose a system for predicting hand motion and intended target during human-robot handover using Inverse Reinforcement Learning (IRL). A set of feature functions were designed to explicitly capture users’ preferences during the task. The proposed approach was experimentally validated through user studies. Results indicate that the proposed method outperformed other state-of-the-art methods (PI-IRL, BP-HMT, RNNIK-MKF and CMk=5) with users feeling comfortable reaching upto 60% of the total distance to the target for handover with 90% target prediction accuracy. The target prediction accuracy reaches 99.9% when less than 20% of the task remains.

Scheduling tasks for multiple heterogeneous robots is challenging, especially with human supervision in demand-aware environments. This research aims to understand human decision-making and its impact on scheduling through two studies, the first is a mixed reality based user study to explore how human perception of the scheduling environment influences task scheduling and facilitates personalized resource allocation. Our findings indicate that human task schedulers exhibit enhanced performance when assisted by autonomous agents, compared to scenarios with limited autonomy in robotic systems. To explore the impact of robot planning on human decision-making and scheduling, we conducted the second study which employed a mixed reality-based warehouse environment, where two users controlled different robots with shared objectives. Results showed that visual aids like collision cones improved collision-aware scheduling without compromising demand-aware capabilities.

Healthcare robotics has emerged as an innovative system with vast potential across various domains. This paper explores the development and application of collaborative robots for teleoperation, which facilitate safe and remote medical general examinations of patients. The study aims to help medical staff to provide vital sign check-ups remotely using legacy medical devices. We proposed an immersive media tool that was integrated with augmented reality (AR) and virtual reality (VR) interfaces to enhance the reach of medical professionals for vital signs check-ups. The medical cobot was teleoperated through AR and VR interfaces. A qualitative user study was used to compare the AR and VR interface systems within the loop of healthcare professionals.

This paper addresses the problem of multi-robot task scheduling by estimating the demand for the tasks in a real-world scenario. Scheduling tasks for multiple robots becomes complex when a human is involved in allocating limited resources. We propose a stochastic multi-agent multi-armed bandit based task scheduler which prioritizes the tasks based on the estimated demand for the tasks. To gain insight into the varying priorities of a human task allocator in a multi-armed bandit scenario, we conducted a user study in a Mixed Reality environment which can be used to customize the resource allocation process. We observe that the users consistently made sub-optimal choices due to their preference to minimize other parameters (such as cumulative distance travelled) of the real world scenario rather than strictly adhering to the optimal strategy. Our proposed method uses the Thompson sampling bandit algorithm with ϵ-greedy approach to solve the multi-agent multi-armed bandit problem. The approach outperformed other methods such as first-come-first-serve, rate monotonic scheduling, and heuristic-based Min-interference approaches in terms of the demand aware performance index.

Estimating the cognitive load of aircraft pilots is essential to monitor them constantly to identify and overcome unfavorable situations. Presently, the cognitive load of pilots is estimated using manual filling up of forms, and there is a lack of a system that can estimate workload automatically. In this paper, we used eye-tracking technology for cognitive load estimation and developed a Virtual Reality dashboard that visualizes cognitive and ocular data. We undertook a flight simulation study to observe users’ workload during primary and secondary task execution while flying the aircraft. We also undertook an eye-tracking study to identify appropriate 3D graph properties for developing graphs of the cognitive dashboard. We found a significant interaction between users’ primary and secondary tasks. We also observed that primitives like size and color were easier to encode numerical and nominal information. Finally, we developed a dashboard leveraging the results of the 3D graph study for estimating the pilot’s cognitive load. The VR dashboard enabled visualization of the cognitive load parameters derived from the ocular data in real time. The aim of the 3D graph study was to identify the optimal information to be displayed to the participants/pilots. Apart from estimating cognitive load using ocular data, the dashboard can also visualize ocular data collected in a Virtual Reality environment.

Social distancing is an effective way to reduce the spread of the SARS-CoV-2 virus. Many students and researchers have already attempted to use computer vision technology to automatically detect human beings in the field of view of a camera and help enforce social distancing. However, because of the present lockdown measures in several countries, the validation of computer vision systems using large-scale datasets is a challenge. In this paper, a new method is proposed for generating customized datasets and validating deep-learning-based computer vision models using virtual reality (VR) technology. Using VR, we modeled a digital twin (DT) of an existing office space and used it to create a dataset of individuals in different postures, dresses, and locations. To test the proposed solution, we implemented a convolutional neural network (CNN) model for detecting people in a limited-sized dataset of real humans and a simulated dataset of humanoid figures. We detected the number of persons in both the real and synthetic datasets with more than 90% accuracy, and the actual and measured distances were significantly correlated (r=0.99). Finally, we used intermittent-layer- and heatmap-based data visualization techniques to explain the failure modes of a CNN. A new application of DTs is proposed to enhance workplace safety by measuring the social distance between individuals. The use of our proposed pipeline along with a DT of the shared space for visualizing both environmental and human behavior aspects preserves the privacy of individuals and improves the latency of such monitoring systems because only the extracted information is streamed.

The Covid-19 pandemic resulted in a catastrophic loss to global economies, and social distancing was consistently found to be an effective means to curb the virus's spread. However, it is only as effective when every individual partakes in it with equal alacrity. Past literature outlined scenarios where computer vision was used to detect people and to enforce social distancing automatically. We have created a Digital Twin (DT) of an existing laboratory space for remote monitoring of room occupancy and automatically detecting violation of social distancing. To evaluate the proposed solution, we have implemented a Convolutional Neural Network (CNN) model for detecting people, both in a limited-sized dataset of real humans, and a synthetic dataset of humanoid figures. Our proposed computer vision models are validated for both real and synthetic data in terms of accurately detecting persons, posture, and intermediate distances among people.

Recent advancements in the field of robotics offers new promises for people with different range of abilities although making a human robot interface for people with severe disabilities is challenging. This paper describes the design and development of an eye gaze controlled interface for users with severe speech and motor impairment to manipulate a robotic arm. Two user studies were reported on pick and drop and reachability studies involving users with severe speech and motor impairment. Using the eye gaze controlled interface users could undertake representative pick and drop task at an average duration less than 15 secs and reach a randomly designated target within 60 secs.

Multi-Function Displays (MFD) are essential part of glass cockpit of modern military and civilian aircrafts. They can display more information in less space than traditional analog displays. Interacting with MFDs is still now limited to a joystick system attached to throttle (called Target Des-ignation System, TDS). This paper explored using gaze con-trolled interface for MFD and proposed two algorithms based on hotspots and adaptable zooming for improving response times in a gaze controlled interface. Three user studies confirmed gaze controlled MFD can significantly reduce response times for big peripheral buttons compared to touchscreen in a head down configuration and com-pared to existing TDS in a head up configuration.

In this paper a novel cloud-based Emergency Warning System (EWS) is described, using India as a primary case study. India is a country with nearly two dozen officially recognized languages and divergent communication technologies spread across a large geographical area. As of yet, this diversity has made the deployment of a single nationwide EWS near impossible. To address this deficiency, we have developed an EWS with adaptation as a central design tenet. This adaptation occurs on three levels: Dissemination adaptation addresses civilians' heterogeneous communication technologies; information adaptation morphs warnings to best reflect the capabilities of these communication technologies and users; while presentation adaptation is used to render information to the user in the most appropriate manner. We have developed a full cloud-based prototype, including the full EWS infrastructure and a civilian Android app.

Eye-gaze tracking is traditionally used to analyze ocular parameters for investigating visual psychology, marketing study, behavior analysis, and so on. Currently, eye-gaze trackers are also being used to control electronic interfaces in assistive technology, automobile control, and even consumer electronic products like smartphones and tablets. However, there are not many attempts to combine these two streams of research on active and passive uses of eye-gaze trackers. This article compares a few ocular parameters to estimate users’ cognitive load in eye-gaze-controlled interfaces. It was found that average velocity of a particular type of microsaccadic eye movement called Saccadic Intrusion is most indicative of users’ cognitive load compared to pupil dilation and eye-blink-based parameters. Results from the study can be used to develop new metrics of cognitive load measurement, as well as to design intelligent gaze-controlled interfaces that respond to users’ cognitive load.

This article presents a series of user studies to develop a new eye-gaze tracking–based pointing system. We developed a new target prediction model that works for different input modalities and combined the eye-gaze tracking–based pointing with a joystick controller that can reduce pointing and selection times. The system finds important applications in cockpit of combat aircraft and for computer novice users. User studies confirmed that users can perform significantly faster using this new eye-gaze tracking–based system for both military and everyday computing tasks compared to existing input devices. As part of the study it was also found that the amplitude of maximum power component obtained through Fourier Transform of pupil signal significantly correlates with selection times and perceived cognitive load of users in terms of Task Load Index scores.

This workshop aims to gap the bridge between mainstream research on intelligent systems and accessibility researchers by presenting papers and demonstrations on developing adaptable multimodal systems for elderly and disabled users. The workshop is organized in the context of EU GUIDE project and focus on Web and Digital TV applications. However the research and applications are relevant for different platforms like computers, tablet and ubiquitous devices. The workshop consists of a keynote speech on standardization of developing intelligent and accessible system followed by five paper and demonstration presentations. A set of papers from this workshop will later appear at the International Journal of Digital Television.

Recent research on interactive electronic systems like computer, digital TV, smartphones can improve the quality of life of many disabled and elderly people by helping them to engage more fully to the world. Previously, a simulator was developed that reflects the effect of impairment on interaction with electronic devices and thus helps designers in developing accessible systems. In this article, the scope of the simulator has been extended to multiple pointing devices. The way that hand strength affects pointing performance of people with and without mobility impairment in graphical user interfaces was investigated for four different input modalities, and a set of linear equations to predict pointing time and average number of submovements for different devices was developed. These models were used to develop an adaptation algorithm to facilitate pointing in electronic interfaces by users with motor impairment using different pointing devices. The algorithm attracts a pointer when it is near a target and thus helps to reduce random movement during homing and clicking. The algorithm was optimized using the simulator and then tested on a real-life application with multiple distractors involving three different pointing devices. The algorithm significantly reduces pointing time for different input modalities.

Elderly and disabled people can be hugely benefited through the advancement of modern electronic devices, as those can help them to engage more fully with the world. However, existing design practices often isolate elderly or disabled users by considering them as users with special needs. This article presents a simulator that can reflect problems faced by elderly and disabled users while they use computer, television, and similar electronic devices. The simulator embodies both the internal state of an application and the perceptual, cognitive, and motor processes of its user. It can help interface designers to understand, visualize, and measure the effect of impairment on interaction with an interface. Initially a brief survey of different user modeling techniques is presented, and then the existing models are classified into different categories. In the context of existing modeling approaches the work on user modeling is presented for people with a wide range of abilities. A few applications of the simulator, which shows the predictions are accurate enough to make design choices and point out the implication and limitations of the work, are also discussed.

We have investigated how hand strength affects pointing performance of people with and without mobility impair-ment in graphical user interfaces for four different input modalities. We have found that grip strength and active range of motion of wrist are most indicative of the point-ing performance. We have used the study to develop a set of linear equations to predict pointing time for different devices.

The development of TV applications nowadays excludes users with certain impairments from interacting with and accessing the same type of contents as other users do. Developers are also not interested in developing new or different versions of applications targeting different user characteristics. In this paper we describe a novel adaptive accessibility approach on how to develop accessible TV applications, without requiring too much additional effort from the developers. Integrating multimodal interaction, adaptation techniques and the use of simulators in the design process, we show how to adapt User Interfaces to the individual needs and limitations of elderly users. For this, we rely on the identification of the most relevant impairment configurations among users in practical user-trials, and we draw a relation with user specific characteristics. We provide guidelines for more accessible and centered TV application development.

We have developed a simulator to help with the design and evaluation of assistive interfaces. The simulator can predict possible interaction patterns when undertaking a task using a variety of input devices, and estimate the time to complete the task in the presence of different dis-abilities. In this paper, we have presented a study to evaluate the simulator by considering a representative application being used by able-bodied, visually impaired and mobility impaired people. The simulator predicted task completion times for all three groups with statistically significant accuracy. The simulator also predicted the effects of different interface designs on task completion time accurately.

Computers offer valuable assistance to people with physical disabilities. However designing human-computer interfaces for these users is complicated. The range of abilities is more diverse than for able-bodied users, which makes analytical modelling harder. Practical user trials are also difficult and time consuming. We are developing a simulator to help with the evaluation of assistive interfaces. It can predict the likely interaction patterns when undertaking a task using a variety of input devices, and estimate the time to complete the task in the presence of different disabilities and for different levels of skill. In this paper we describe the different components of the simulator in detail and present a prototype of its implementation.

pb400 @ cam.ac.uk Computers offer valuable assistance to people with physical disabilities. However designing human-computer interfaces for these users is complicated. The range of abilities is more diverse than for able-bodied users, which makes analytical modelling harder. Practical user trials are also difficult and time consuming. We have developed a simulator to help with the evaluation of assistive interfaces. It can predict the likely interaction patterns when undertaking a task using a variety of input devices, and estimate the time to complete the task in the presence of different disabilities and for different levels of skill.