There is significant interest in enabling robots to learn to perform tasks directly from interactions with non-expert users. Typically, a human serves as a teacher whose only task is to provide feedback to a robot learner. However, in real-world human-robot collaborations, the human often assists with the task while also offering feedback. Our key insight is that we can extract additional, implicit feedback from the human’s actions during the collaboration to augment the robot learning process. Under the assumption of fixed-role assignments, we first propose to formalize human preferences over a human-robot collaboration as a shared set of parameters encoding alignment between two reward functions: one that drives human behavior, and another that should direct robot behavior. This allows us to extract implicit feedback from an interaction by reasoning about the human’s actions in the task as actions that reveal the human’s preferences. Then, we combine this implicit feedback with traditional explicit human feedback to facilitate estimating the human’s preferences. We evaluated our proposed approach for Preference learning from Implicit and Explicit feedback (PIE) in simulations and with real users in a cooking scenario. Our simulation results indicate that combining multiple modalities of human feedback improves a robot’s ability to estimate human preferences over the collaboration, with a similar trend observed in real-world evaluations. These findings highlight a promising direction for enabling robots to adapt to a user’s preference model more quickly, thereby reducing the amount of time a person must spend teaching a robot.

The deployment of robot assistants in large indoor spaces has seen significant growth, with escorting tasks becoming a key application. However, most current escorting robots primarily rely on navigation-focused strategies, assuming that the person being escorted will follow without issue. In crowded environments, this assumption often falls short, as individuals may struggle to keep pace, become obstructed, get distracted, or need to stop unexpectedly. As a result, conventional robotic systems are often unable to provide effective escorting services due to their limited understanding of human movement dynamics. To address these challenges, an effective escorting robot must continuously detect and interpret human actions during the escorting process and adjust its movement accordingly. However, there is currently no existing dataset designed specifically for human action detection in the context of escorting. Given that escorting often occurs in crowded environments, where other individuals may enter the robot’s camera view, the robot also needs to identify the specific human it is escorting (the subject) before predicting their actions. Since no existing model performs both person re-identification and action prediction in real-time, we propose a novel neural network architecture that can accomplish both tasks. This enables the robot to adjust its speed dynamically based on the escortee’s movements and seamlessly resume escorting after any disruption. In comparative evaluations against strong baselines, our system demonstrates superior efficiency and effectiveness, showcasing its potential to significantly improve robotic escorting services in complex, real-world scenarios.

Navigating through a new indoor environment can be stressful. Recently, many places have deployed robots to assist visitors. One of the features of such robots is escorting the visitors to their desired destination within the environment, but this is neither scalable nor necessary for every visitor. Instead, a robot assistant could be deployed at a strategic location to provide wayfinding instructions. This not only increases the user experience but can be helpful in many time-critical scenarios e.g., escorting someone to their boarding gate at an airport. However, delivering route descriptions verbally poses a challenge. If the description is too verbose, people may struggle to recall all the information, while overly brief descriptions may be simply unhelpful. This article focuses on studying the optimal length of verbal route descriptions that are effective for reaching the destination and easy for people to recall. This work proposes a theoretical framework that links route segments to chunks in working memory. Based on this framework, an experiment is designed and conducted to examine the effects of route descriptions of different lengths on navigational performance. The results revealed intriguing patterns suggesting an ideal length of four route segments. This study lays a foundation for future research exploring the relationship between route description lengths, working memory capacity, and navigational performance in indoor environments.

Social robots in our daily surroundings, like personal guides, waiter robots, home helpers, assistive robots, and telepresence/teleoperation robots, are increasing day by day. Their usability and acceptability largely depend on their explicit and implicit interaction capability with fellow human beings. As a result, social behavior is one of the most sought-after qualities that a robot can possess. However, there is no specific aspect and/or feature that defines socially acceptable behavior, and it largely depends on the situation, application, and society. In this article, we investigate one such social behavior for collocated robots. Imagine a group of people is interacting with each other, and we want to join the group. We as human beings do it in a socially acceptable manner, i.e., within the group, we do position ourselves in such a way that we can participate in the group activity without disturbing/obstructing anybody. To possess such a quality, first, a robot needs to determine the formation of the group and then determine a position for itself, which we humans do implicitly. There are many theories which study group formations and proxemics; one such theory is f-formation which could be utilized for this purpose. As the types of formations can be very diverse, detecting the social groups is not a trivial task. In this article, we provide a comprehensive survey of the existing work on social interaction and group detection using f-formation for robotics and other applications. We also put forward a novel holistic survey framework combining some of the possibly more important concerns and modules relevant to this problem. We define taxonomies based on methods, camera views, datasets, detection capabilities and scale, evaluation approaches, and application areas. We discuss certain open challenges and limitations in the current literature along with possible future research directions based on this framework. In particular, we discuss the existing methods/techniques and their relative merits and demerits, applications, and provide a set of unsolved but relevant problems in this domain. The official website for this work is available at: https://github.com/HrishavBakulBarua/Social-Robots-F-formation

An efficient communication mechanism forms the backbone for any multi-robot system to achieve fruitful collaboration and coordination. Limitation in the existing asynchronous transmission based strategies in fast dissemination and aggregation compels the designers to prune down such requirements as much as possible. This also restricts the possible application areas of mobile multi-robot systems. In this work, we introduce concurrent transmission based strategy as an alternative. Despite the commonly found difficulties in concurrent transmission such as microsecond level time synchronization, hardware heterogeneity, etc., we demonstrate how it can be exploited for multi-robot systems. We propose a split architecture where the two major activities - communication and computation are carried out independently and coordinate through periodic interactions. The proposed split architecture is applied on a custom-built, fully networked control system consisting of five two-wheel differential drive mobile robots having heterogeneous architecture. We use the proposed design in a leader-follower setting for coordinated dynamic speed variation as well as the independent formation of various shapes. Experiments show a centimeter-level spatial and millisecond-level temporal accuracy while spending very low radio duty-cycling over multi-hop communication under a wide testing area.

The usage of automatic speech recognition (ASR) systems are becoming omnipresent ranging from personal assistant to chatbots, home, and industrial automation systems, etc. Modern robots are also equipped with ASR capabilities for interacting with humans as speech is the most natural interaction modality.However, ASR in robots faces additional challenges as compared to a personal assistant.Being an embodied agent, a robot must recognize the physical entities around it and therefore reliably recognize the speech containing the description of such entities.However, current ASR systems are often unable to do so due to limitations in ASR training, such as generic datasets and open-vocabulary modeling.Also, adverse conditions during inference, such as noise, accented, and far-field speech makes the transcription inaccurate.In this work, we present a method to incorporate a robot's visual information into an ASR system and improve the recognition of a spoken utterance containing a visible entity.Specifically, we propose a new decoder biasing technique to incorporate the visual context while ensuring the ASR output does not degrade for incorrect context.We achieve a 59% relative reduction in WER from an unmodified ASR system.

This paper presents a novel architecture to detect social groups in real-time from a continuous image stream of an ego-vision camera. F-formation defines social orientations in space where two or more person tends to communicate in a social place. Thus, essentially, we detect F-formations in social gatherings such as meetings, discussions, etc. and predict the robot's approach angle if it wants to join the social group. Additionally, we also detect outliers, i.e., the persons who are not part of the group under consideration. Our proposed pipeline consists of - a) a skeletal key points estimator (a total of 17) for the detected human in the scene, b) a learning model (using a feature vector based on the skeletal points) using CRF to detect groups of people and outlier person in a scene, and c) a separate learning model using a multi-class Support Vector Machine (SVM) to predict the exact F-formation of the group of people in the current scene and the angle of approach for the viewing robot. The system is evaluated using two data-sets. The results show that the group and outlier detection in a scene using our method establishes an accuracy of 91%. We have made rigorous comparisons of our systems with a state-of-the-art F-formation detection system and found that it outperforms the state-of-the-art by 29% for formation detection and 55% for combined detection of the formation and approach angle.

Many Internet of Things smart-* applications are being powered solely through ambient energy-harvested energy. These applications require periodic data collection with low latency and high reliability. Since the energy is harvested in small amounts from ambient sources and is stochastic in nature, it is extremely challenging to achieve low latency and high reliability for such applications. To this end, we propose a distributed, energy-management module called ReNEW, for Constructive Interference (CI) based protocols that utilizes the available energy effectively in order to achieve our target of increased reliability in EH-WSN, especially in the low harvesting regimes. We choose CI-based protocols to leverage the low latency guarantees. Specifically, we propose a Markov decision model to maximize the energy utility in the infinite horizon by allocating energy optimally. To this end, we also propose a threshold optimal policy. As we find that just a energy scheduler cannot achieve the goal, we also propose distributed techniques to conserve energy on the redundant nodes in the network, and dynamically activate them based on feedback. We also improve the performance of CI by adapting the transmit powers on nodes. We implement and evaluate ReNEW on Indriya testbed for real-world scenarios. We show that in a network of 20 source nodes out of the 30 nodes in the network can perform periodic data collection with an improvement of 2.5 times higher packet reception ratio as compared to LWB. This is one of the worst case scenarios as the harvested energy is as low as 50uJ/s and packets of size 100 B is sent every 30 s. Furthermore, in this scenario, ReNEW saves around 25% higher residual energy on the average as compared to the standard LWB. In a nutshell, by integrating ReNEW with CI based protocols, we enable guaranteed latency and increased reliability for the batteryless EH-WSNs.

Multi-robot systems are deployed in a warehouse to automate the process of storing and retrieving objects in and out of the warehouse. The efficiency of the system largely depends on how the tasks are allocated to the robots. Though there exists a number of techniques that can perform multi-robot task allocation quite efficiently, they hardly consider deadline for task completion while assigning tasks to the robots. A careful allocation is of paramount importance when there is an associated penalty with each of the tasks if it is not completed within a stipulated time. In this work, we develop an algorithm, called Minimum Penalty Scheduling (MPS) that allocates tasks among a group of robots with the goal that the overall penalty of executing all the tasks can be minimized. Our algorithm provides a robust, scalable, and near-optimal real-time task schedule. By comparing with the state-of-the-art algorithm, we show that MPS attracts up to 62.5% less penalty when a significant number of tasks are bound to miss the deadline. Additionally, MPS is also suitable for real-time multi-processor scheduling since it schedules a higher number of tasks within their deadline.

Warehouse management system assigns a preferred completion time for every order based on the customer profile and the good(s) that is/are ordered. Even though employing multi-robot systems to manage goods movement bring operational efficiency in a warehouse, it is difficult to meet these soft deadlines of tasks in the peak hours/seasons. This can impact the respective businesses significantly as the lateness of task completion incurs a direct/indirect penalty. In this work, we develop an online task scheduling algorithm for such a multi-robot system, called Online Minimum Penalty Scheduling (OMPS). Though there exists a large number of multi-robot task scheduling algorithms, they are not suitable (or less efficient) for a system where each task has a soft deadline and accumulates penalty if it is executed beyond its deadline. Moreover, the lack of knowledge of future tasks (online scheduling) makes task allocation a much more difficult job. OMPS provides a robust, scalable, and near-optimal online task schedule. By comparing with the state-of-the-art algorithm, we show that OMPS attracts up to 78% less penalty when a significant number of tasks are bound to miss the deadline. Additionally, it achieves a competitive ratio of up to 1 when compared with a state-of-the-art offline task scheduling algorithm.

A robot as a coworker or a cohabitant is becoming mainstream day-by-day with the development of low-cost sophisticated hardware. However, an accompanying software stack that can aid the usability of the robotic hardware remains the bottleneck of the process, especially if the robot is not dedicated to a single job. Programming a multi-purpose robot requires an on the fly mission scheduling capability that involves task identification and plan generation. The problem dimension increases if the robot accepts tasks from a human in natural language. Though recent advances in NLP and planner development can solve a variety of complex problems, their amalgamation for a dynamic robotic task handler is used in a limited scope. Specifically, the problem of formulating a planning problem from natural language instructions is not studied in details. In this work, we provide a non-trivial method to combine an NLP engine and a planner such that a robot can successfully identify tasks and all the relevant parameters and generate an accurate plan for the task. Additionally, some mechanism is required to resolve the ambiguity or missing pieces of information in natural language instruction. Thus, we also develop a dialogue strategy that aims to gather additional information with minimal question-answer iterations and only when it is necessary. This work makes a significant stride towards enabling a human-like task understanding capability in a robot.

The number of robots deployed in our daily surroundings is ever-increasing. Even in the industrial setup, the use of coworker robots is increasing rapidly. These cohabitant robots perform various tasks as instructed by co-located human beings. Thus, a natural interaction mechanism plays a big role in the usability and acceptability of the robot, especially by a non-expert user. The recent development in natural language processing (NLP) has paved the way for chatbots to generate an automatic response for users’ query. A robot can be equipped with such a dialogue system. However, the goal of human-robot interaction is not focused on generating a response to queries, but it often involves performing some tasks in the physical world. Thus, a system is required that can detect user intended task from the natural instruction along with the set of pre- and post-conditions. In this work, we develop a dialogue engine for a robot that can classify and map a task instruction to the robot’s capability. If there is some ambiguity in the instructions or some required information is missing, which is often the case in natural conversation, it asks an appropriate question(s) to resolve it. The goal is to generate minimal and pin-pointed queries for the user to resolve an ambiguity. We evaluate our system for a telepresence scenario where a remote user instructs the robot for various tasks. Our study based on 12 individuals shows that the proposed dialogue strategy can help a novice user to effectively interact with a robot, leading to satisfactory user experience.

The constructive interference (CI) phenomenon has been exploited by a number of protocols for providing energy-efficient, low-latency, and reliable data collection and dissemination services in wireless sensor networks. These protocols consider CI to provide highly reliable packet delivery. This has attracted attention to understand the working of CI; however, the existing works present inconsistent views. Furthermore, these works do not study in the real-world settings where the physical conditions of deployment and unreliable wireless channels also impact the performance of CI. Therefore, we study the phenomenon of CI, considering a receiver's viewpoint and analyze the parameters that affect CI. We validate our arguments with results from extensive and rigorous experimentation in real-world settings. This paper presents comprehensive insights into the CI phenomenon. With the understanding, we develop the destructive interference-based power adaptation (DIPA), an energy-efficient and distributed algorithm, that adapts transmission power to improve the performance of CI. Since CI-based protocols cannot have an explicit acknowledgment packet, we make use of destructive interference on a designated byte to provide a feedback. We leverage this feedback to adapt transmission powers. We compared CI with and without DIPA in two real-life testbeds. On one testbed, we achieve around 25% lower packet losses while using only half of its transmission power for 64-B packets. On the other testbed, we achieve 25% lower packet losses while consuming only 47% of its transmission power for 128-B packets. Existing CI-based protocols can easily incorporate DIPA into them to achieve lower packet losses and higher energy efficiencies.

Buildings can achieve energy-efficiency by using solar passive design, energy-efficient structures and materials, or by optimizing their operational energy use. In each of these areas, efficiency can be improved if the physical properties of the building along with its dynamic behavior can be captured using low-cost embedded sensor devices. This opens up a new challenge of installing and maintaining the sensor devices for different types of buildings. In this article, we propose BuildSense, a sensing framework for fine-grained, long-term monitoring of buildings using a mix of physical and virtual sensors. It not only reduces the deployment and management cost of sensors but can also guarantee accurate and fault-tolerant data coverage for long-term use. We evaluate BuildSense using sensor measurements from two rammed-earth houses that were custom-designed for a challenging hot-arid climate so almost no artificial heating or cooling is required. We demonstrate that BuildSense can significantly reduce the cost of permanent physical sensors while still achieving fit-for-purpose accuracy, fault-tolerance, and stability. Overall, we were able to reduce the cost of a building sensor network by 60% to 80% by replacing physical sensors with virtual ones while still maintaining accuracy of ≤1.0°C and fault-tolerance of two or more predictors per virtual sensor.

In modern warehouses, robots are being deployed to perform complex tasks such as fetching a set of objects from various locations in a warehouse to a docking station. This requires a careful task allocation along with route planning such that the total distance traveled (cost) is minimized. The number of tasks that can be performed by a robot on a single route depends on the maximum capacity of the robot and the combined weight of the objects it picks on the route. This task allocation problem is an instance of the Capacity-constrained Vehicle Routing Problem (CVRP), which is known to be NP-hard. Although, there exist a number of heuristics that provide near-optimal solutions to a CVRP instance, they do not scale well with the task size (number of nodes). In this paper, we present a heuristic, called nearest-neighbor based Clustering And Routing$(nCAR)$, which has better execution time compared to the state-of-the-art heuristics. Also, our heuristic reduces cost of the solutions when there are a large number of nodes. We compare the performance of$nCAR$with the Google OR-Tools and found a speedup of 6 in runtime when task size is 2000. Though OR-Tools provides a low-cost solution for small number of tasks, it's execution time and number of routes is 1.5 times that of nCAR.

A robot as a companion or co-worker is not an emerging concept anymore, but a reality. However, one of the major barriers to this realization is the seamless interaction with the robots that includes both explicit and implicit interaction. In this work, we assume a use-case where a human and a robot together carry a heavy object in a co-habitat (home or workplace/factory). Two human beings while doing such a work understands each other without explicit (vocal) interaction. To realize such behavior, the robot must understand the fatigue state of the human co-worker to enable seamless work experience and ensure safety. In this article, we present DeFatigue, a non-intrusive fatigue state detection mechanism. We assume that the robot's hand is equipped with a force sensor. Based on the change of force from the human side while carrying the object, DeFatigue is able to determine the fatigue state without instrumenting the human being with an additional sensor (internally or externally). Moreover, it detects the fatigues state on-the-fly (online) as well as it does not require any (user-specific) training. Based on our experiments with 18 test subjects, fatigue state detection by DeFatigue overlaps with the ground truth for 85.18% of the cases whereas it deviates 4.09 s (on average) for the remaining cases.

Constructive Interference (CI) phenomenon has been exploited by Glossy, a mechanism for low-latency and reliable network flooding and time synchronization for wireless sensor networks. Recently, CI has also been used for other applications such as data collection and multicasting in static and mobile WSNs. These applications base their working on the high reliability promised by Glossy regardless of the physical conditions of deployment, number of nodes in the network, and unreliable wireless channels that may be detrimental for CI. There are several works that study the working of CI, but they present inconsistent views. We study CI from a receiver's viewpoint, list factors that affect CI and also specify how and why they affect. We validate our arguments with results from extensive and rigorous experimentation in real-world settings. This paper presents comprehensive insights into CI phenomenon. With this understanding, we improve the performance of CI through an energy-efficient and distributed algorithm. We cause destructive interference on a designated byte to provide negative feedback. We leverage this to adapt transmission powers. Compared to Glossy, we achieve 25% lesser packet losses while using only half of its transmission power.

Typical Wireless Sensor Networks (WSN) deployments use more nodes than needed to accurately sense the phenomena of interest. This redundancy can be leveraged by switching-on only a subset of nodes at any time instant (node-scheduling) and putting the remaining nodes sleep. This effectively extends the network lifetime. In addition to sensing coverage, node-scheduling schemes must also ensure that (i) the network stays connected, and (ii) the time needed to wake-up the complete protocol stack after sleeping is minimized. We present Sleeping Beauty, a highly-efficient data collection protocol that aids node-scheduling schemes in both aspects. Sleeping Beauty uses a slotted and tightly synchronized communication primitive, where a node keeps its radio off for most of the time, except in the slots when it needs to participate for successful communication. Further, an efficient neighbor-discovery mechanism is included that provides partial, but sufficient topology information (potential parents) to avoid network partitions. Furthermore, Sleeping Beauty employs a novel, yet simple clock-offset estimation technique that maintains highly-accurate time synchronization over long radio-off periods (i.e., less than 500 us deviation even after 45, min of sleeping). This minimizes time wasted in resynchronizing the network in between data collection rounds. Through experiments on two different testbeds, we verified that Sleeping Beauty decreases the duty cycle up to a factor of 3 compared to state-of-the-art techniques, while achieving similar delivery ratios.

The advent of Internet of Things (IoT) has boosted the growth in number of devices around us and kindled the possibility of umpteen number of applications. One of the major challenges in the realization of IoT applications is interoperability among various IoT devices and deployments. Thus, the need for a new architecture-comprising smart control and actuation-has been identified by many researchers. In this paper, we propose a Distributed Internet-like Architecture for Things (DIAT), which will overcome most of the obstacles in the process of large-scale expansion of IoT. It specifically addresses heterogeneity of IoT devices, and enables seamless addition of new devices across applications. In addition, we propose an usage control policy model to support security and privacy in a distributed environment. We propose a layered architecture that provides various levels of abstraction to tackle the issues such as scalability, heterogeneity, security, and interoperability. The proposed architecture is coupled with cognitive capabilities that helps in intelligent decision-making and enables automated service creation. Using a comprehensive use-case, comprising elements from multiple-application domains, we illustrate the usability of the proposed architecture.

In data gathering wireless sensor network applications, data correlation among the sensor nodes have been utilized to extend network lifetimes. It has been shown that the data correlation also exists between nodes that are far away, contrary to the assumption that correlation decreases as a function of distance. Therefore, it is possible to group the nodes based on the correlation among their data regardless of their location. Given that data from one active node per group is sufficient to reconstruct the sensed data for the remaining sensor nodes, most of the nodes can be kept in low-power sleep mode. However, only few active nodes will usually create a disconnected network, and hence failing the purpose of the deployment. In this paper we formalize this problem, referred to as Sleep-route, of selecting the minimum number of connected active nodes that are sufficient to predict the sensed data for remaining sleeping nodes with high accuracy. We prove that the problem is NP-hard. Thus, we develop a greedy algorithm, Sleep-route heuristic that provides near-optimal solutions. Using Contiki-based simulations, we show that our scheme can extend network lifetime up to 42% as compared to the state-of-the-art solutions.