ACM UMAP June 16-19 2025, New York, USA

This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing preference-based and ranking-based reinforcement learning paradigms, based on human relative preferences over sample pairs, the proposed rating-based reinforcement learning approach is based on human evaluation of individual trajectories without relative comparisons between sample pairs. The rating-based reinforcement learning approach builds on a new prediction model for human ratings and a novel multi-class loss function. We conduct several experimental studies based on synthetic ratings and real human ratings to evaluate the effectiveness and benefits of the new rating-based reinforcement learning approach.

Electroencephalography (EEG) has found many applications cutting across many domains, such as digital health, affective computing, and human-machine interfaces. However, its widespread adoption in practice has been primarily inhibited by its susceptibility to noise artifacts and the low spatial resolution of electrodes on commercial EEG sensors. While several prior works have investigated techniques for detecting and extracting noise, our understanding of performance degradation due to electrode sparsity remains limited. In this work, we explore the feasibility of using Functional Connectivity (FC) for improving the EEG sensing-based accuracy using two exemplars downstream, working memory-related tasks: (a) cognitive task load (CTL) assessment and (b) high attentional event-evoked potential (EEP) episodes detection. This paper proposes an integrated approach, EEGAmp+ , that first utilizes channel-wise functional connectivity modules using independent component analysis (ICA) coupled with cosine distance for EEG signal reconstruction for cognitive task load assessment tasks. This is then coupled with a sliding window change point detection technique paired with continuous wavelet transformation (CWT) to extract high attentional EEP episodes. Our empirical results indicate that using independent component analysis (ICA) coupled with FC to improve spatial resolution increased cognitive load assessment accuracy by [5.6% ± 1.13] across four machine learning algorithms. Furthermore, after signal reconstruction, we introduce sliding window CPD coupled with CWT, which allows us to extract EEP segments legibly through decomposing the signals and the ability to capture both time and frequency representation from the reconstructed signal boosting detection accuracy by [11.1% ±1.31].

The recent rapid advancement of machine learning has been driven by increasingly powerful models with the growing availability of training data and computational resources. However, real-time decision-making tasks with limited time and sparse learning signals remain challenging. One way of improving the learning speed and performance of these agents is to leverage human guidance. In this work, we introduce GUIDE, a framework for real-time human-guided reinforcement learning by enabling continuous human feedback and grounding such feedback into dense rewards to accelerate policy learning. Additionally, our method features a simulated feedback module that learns and replicates human feedback patterns in an online fashion, effectively reducing the need for human input while allowing continual training. We demonstrate the performance of our framework on challenging tasks with sparse rewards and visual observations. Our human study involving 50 subjects offers strong quantitative and qualitative evidence of the effectiveness of our approach. With only 10 minutes of human feedback, our algorithm achieves up to 30\% increase in success rate compared to its RL baseline.

Spatial reasoning tasks in multi-agent environments such as event prediction, agent type identification, or missing data imputation are important for multiple applications (e.g., autonomous surveillance over sensor networks and subtasks for reinforcement learning (RL)). StarCraft II game replays encode intelligent (and adversarial) multiagent behavior and could provide a testbed for these tasks; however, extracting simple and standardized representations for prototyping these tasks is laborious and hinders reproducibility. In contrast, MNIST and CIFAR10, despite their extreme simplicity, have enabled rapid prototyping and reproducibility of ML methods. Following the simplicity of these datasets, we construct a benchmark spatial reasoning dataset based on StarCraft II replays that exhibit complex multi-agent behaviors, while still being as easy to use as MNIST and CIFAR10. Specifically, we carefully summarize a window of 255 consecutive game states to create 3.6 million summary images from 60,000 replays, including all relevant metadata such as game outcome and player races. We develop three formats of decreasing complexity: Hyperspectral images that include one channel for every unit type (similar to multispectral geospatial images), RGB images that mimic CIFAR10, and grayscale images that mimic MNIST. We show how this dataset can be used for prototyping spatial reasoning methods. All datasets, code for extraction, and code for dataset loading can be found at https://starcraftdata.davidinouye.com/.

Intelligent and complex human motion analysis can help design the next generation IoT and AR/VR systems for automated human performance assessment. Such an automated system can help advocate the interpretability and translatability of complex human motions, intelligent motion feedback, and fine-grained motion skill assessment to design next-generation interactive human-machine teaming systems. Motivated by this, we design a wearable sensing framework for assessing the players’ performance and consider a live badminton game as our use case. Generally, the players on the field try to improve their performance by focusing on fast and synchronous coordination of their limbs’ reflex actions to have the ideal body postures to perform the desired shot. Learning the minute dissimilarities and distinctive traits from each limb of the players simultaneously can help assess the players’ performance and specific skillsets during a game. This paper proposes a multi-task learning framework, PerMTL to learn the shared features from each player’s limb. The PerMTL comprises a task-specific regressor output layer that helps to determine the dissimilarities and distinctive traits between the player’s limbs for collective inference in a body sensor network (BSN) environment. We evaluate the PerMTL framework using publicly available Badminton Activity Recognition (BAR) and Daily and Sports Activities (DSA) datasets. Empirical results indicate that PerMTL achieves R2Score of ≈ 82% in predicting the players’ performance.

While both navigation and manipulation are chal-lenging topics in isolation, many tasks require the ability to both navigate and manipulate in concert. To this end, we propose a mobile manipulation system that leverages novel navigation and shape completion methods to manipulate an object with a mobile robot. Our system utilizes uncertainty in the initial estimation of a manipulation target to calculate a predicted next-best-view. Without the need of localization, the robot then uses the predicted panoramic view at the next-best-view location to navigate to the desired location, capture a second view of the object, create a new model that predicts the shape of object more accurately than a single image alone, and uses this model for grasp planning. We show that the system is highly effective for mobile manipulation tasks through simulation experiments using real world data, as well as ablations on each component of our system.

Recently, Reinforcement Learning (RL) has shown great performance in solving sequential decision-making and control in dynamic environment problems. Despite its achievements, deploying Deep Neural Network (DNN)-based RL is expensive in terms of time and power due to the large number of episodes required to train agents with high dimensional image representations. Additionally, at the interference the large energy footprint of deep neural networks can be a major drawback. Embedded edge devices as the main platform for deploying RL applications are intrinsically resource-constrained and deploying deep neural network-based RL on them is a challenging task. As a result, reducing the number of actions taken by the RL agent to learn desired policy, along with the energy-efficient deployment of RL, is crucial. In this article, we propose Energy Efficient Hierarchical Reinforcement Learning (E2HRL), which is a scalable hardware architecture for RL applications. E2HRL utilizes a cross-layer design methodology for achieving better energy efficiency, smaller model size, higher accuracy, and system integration at the software and hardware layers. Our proposed model for RL agent is designed based on the learning hierarchical policies, which makes the network architecture more efficient for implementation on mobile devices. We evaluated our model in three different RL environments with different level of complexity. Simulation results with our analysis illustrate that hierarchical policy learning with several levels of control improves RL agents training efficiency and the agent learns the desired policy faster compared to a non-hierarchical model. This improvement is specifically more observable as the environment or the task becomes more complex with multiple objective subgoals. We tested our model with different hyperparameters to achieve the maximum reward by the RL agent while minimizing the model size, parameters, and required number of operations. E2HRL model enables efficient deployment of RL agent on resource-constraint-embedded devices with the proposed custom hardware architecture that is scalable and fully parameterized with respect to the number of input channels, filter size, and depth. The number of processing engines (PE) in the proposed hardware can vary between 1 to 8, which provides the flexibility of tradeoff of different factors such as latency, throughput, power, and energy efficiency. By performing a systematic hardware parameter analysis and design space exploration, we implemented the most energy-efficient hardware architectures of E2HRL on Xilinx Artix-7 FPGA and NVIDIA Jetson TX2. Comparing the implementation results shows Jetson TX2 boards achieve 0.1 ∼ 1.3 GOP/S/W energy efficiency while Artix-7 FPGA achieves 1.1 ∼ 11.4 GOP/S/W, which denotes 8.8× ∼ 11× better energy efficiency of E2HRL when model is implemented on FPGA. Additionally, compared to similar works our design shows better performance and energy efficiency.

This article presents an energy-efficient and flexible multichannel Electroencephalogram (EEG) artifact identification network and its hardware using depthwise and separable convolutional neural networks. EEG signals are recordings of the brain activities. EEG recordings that are not originated from cerebral activities are termed artifacts . Our proposed model does not need expert knowledge for feature extraction or pre-processing of EEG data and has a very efficient architecture implementable on mobile devices. The proposed network can be reconfigured for any number of EEG channel and artifact classes. Experiments were done with the proposed model with the goal of maximizing the identification accuracy while minimizing the weight parameters and required number of operations. Our proposed network achieves 93.14% classification accuracy using an EEG dataset collected by 64-channel BioSemi ActiveTwo headsets, averaged across 17 patients and 10 artifact classes. Our hardware architecture is fully parameterized with number of input channels, filters, depth, and data bit-width. The number of processing engines (PE) in the proposed hardware can vary between 1 to 16, providing different latency, throughput, power, and energy efficiency measurements. We implement our custom hardware architecture on Xilinx FPGA (Artix-7), which on average consumes 1.4 to 4.7 mJ dynamic energy with different PE configurations. Energy consumption is further reduced by 16.7× implementing on application-specified integrated circuit at the post layout level in 65-nm CMOS technology. Our FPGA implementation is 1.7 × to 5.15 × higher in energy efficiency than some previous works. Moreover, our Application-Specified Integrated Circuit implementation is also 8.47 × to 25.79 × higher in energy efficiency compared to previous works. We also demonstrated that the proposed network is reconfigurable to detect artifacts from another EEG dataset collected in our lab by a 14-channel Emotiv EPOC+ headset and achieved 93.5% accuracy for eye blink artifact detection.

Reinforcement learning (RL) has shown great performance in solving sequential decision-making problems. While a lot of works have done on processing state information such as images, there has been some effort towards integrating natural language instructions into RL. In this paper, we propose an energy-efficient architecture which is designed to receive both images and text inputs as a step towards designing RL agents that can understand human language and act in real-world environments. Different configurations are proposed to illustrate the trade off between the number of parameters and the model accuracy, and a custom low power hardware is designed and implemented on FPGA based on the best configuration. The hardware designed to be configurable with different parameters such as number of processing elements, so that it can easily balance power and performance. The high throughput configuration achieves 217 frames per second throughput with 1.2 mJ energy consumption per classification on Xilinx Artix-7 FPGA, while the low power configuration consumes less than 139 mW for 30 frames per second classification. Compared to the similar works using FPGA for hardware implementation, our design is more energy efficient and need less energy for generating each output.

We present a robot navigation system that uses an imitation learning framework to successfully navigate in complex environments. Our framework takes a pre-built 3D scan of a real environment and trains an agent from pre-generated expert trajectories to navigate to any position given a panoramic view of the goal and the current visual input without relying on map, compass, odometry, or relative position of the target at runtime. Our end-to-end trained agent uses RGB and depth (RGBD) information and can handle large environments (up to 1031m2) across multiple rooms (up to 40) and generalizes to unseen targets. We show that when compared to several baselines our method (1) requires fewer training examples and less training time, (2) reaches the goal location with higher accuracy, and (3) produces better solutions with shorter paths for long-range navigation tasks.

This paper investigates how to utilize different forms of human interaction to safely train autonomous systems in realtime by learning from both human demonstrations and interventions. We implement two components of the Cycle-of Learning for Autonomous Systems, which is our framework for combining multiple modalities of human interaction. The current effort employs human demonstrations to teach a desired behavior via imitation learning, then leverages intervention data to correct for undesired behaviors produced by the imitation learner to teach novel tasks to an autonomous agent safely, after only minutes of training. We demonstrate this method in an autonomous perching task using a quadrotor with continuous roll, pitch, yaw, and throttle commands and imagery captured from a downward-facing camera in a high-fidelity simulated environment. Our method improves task completion performance for the same amount of human interaction when compared to learning from demonstrations alone, while also requiring on average 32% less data to achieve that performance. This provides evidence that combining multiple modes of human interaction can increase both the training speed and overall performance of policies for autonomous systems.

In autonomous embedded systems, it is often vital to reduce the amount of actions taken in the real world and energy required to learn a policy. Training reinforcement learning agents from high dimensional image representations can be very expensive and time consuming. Autoencoders are deep neural network used to compress high dimensional data such as pixelated images into small latent representations. This compression model is vital to efficiently learn policies, especially when learning on embedded systems. We have implemented this model on the NVIDIA Jetson TX2 embedded GPU, and evaluated the power consumption, throughput, and energy consumption of the autoencoders for various CPU/GPU core combinations, frequencies, and model parameters. Additionally, we have shown the reconstructions generated by the autoencoder to analyze the quality of the generated compressed representation and also the performance of the reinforcement learning agent. Finally, we have presented an assessment of the viability of training these models on embedded systems and their usefulness in developing autonomous policies. Using autoencoders, we were able to achieve 4-5X improved performance compared to a baseline RL agent with a convolutional feature extractor, while using less than 2W of power.

While recent advances in deep reinforcement learning have allowed autonomous learning agents to succeed at a variety of complex tasks, existing algorithms generally require a lot oftraining data. One way to increase the speed at which agent sare able to learn to perform tasks is by leveraging the input of human trainers. Although such input can take many forms, real-time, scalar-valued feedback is especially useful in situations where it proves difficult or impossible for humans to provide expert demonstrations. Previous approaches have shown the usefulness of human input provided in this fashion (e.g., the TAMER framework), but they have thus far not considered high-dimensional state spaces or employed the use of deep learning. In this paper, we do both: we propose DeepTAMER, an extension of the TAMER framework that leverages the representational power of deep neural networks inorder to learn complex tasks in just a short amount of time with a human trainer. We demonstrate Deep TAMER’s success by using it and just 15 minutes of human-provided feedback to train an agent that performs better than humans on the Atari game of Bowling - a task that has proven difficult for even state-of-the-art reinforcement learning methods.

As the technology for acquiring and storing images becomes more prevalent, we are faced with a growing need to sort and label these images. At this time, computer vision algorithms cannot parse abstract concepts from images like a human. As a result, there may be performance gains possible from the integration of human analysts with computer vision agents. We present an image triage system which facilitates the collaboration of heterogeneous agents through a novel unsupervised meta-learning technique. The system iteratively allocates images for binary classification among heterogeneous agents according to the Generalized Assignment Problem (GAP) and combines the classification results using the Spectral Meta-Learner (SML). In simulation, we demonstrate that the proposed system achieves significant speed-up over a naive parallel assignment strategy without sacrificing accuracy.

Recent advances in signal processing for the detection of Steady-State Visual Evoked Potentials (SSVEPs) have moved away from traditionally calibrationless methods, such as canonical correlation analysis, and towards algorithms that require substantial training data. In general, this has improved detection rates, but SSVEP-based brain-computer interfaces (BCIs) now suffer from the requirement of costly calibration sessions. Here, we address this issue by applying transfer learning techniques to SSVEP detection. Our novel Adaptive-C3A method incorporates an unsupervised adaptation algorithm that requires no calibration data. Our approach learns SSVEP templates for the target user and provides robust class separation in feature space leading to increased classification accuracy. Our method achieves significant improvements in performance over a standard CCA method as well as a transfer variant of the state-of-the art Combined-CCA method for calibrationless SSVEP detection.

Steady-state visual evoked potentials (SSVEPs) are oscillations of the electroencephalogram (EEG) which are mainly observed over the occipital area that exhibits a frequency corresponding to a repetitively flashing visual stimulus. SSVEPs have proven to be very consistent and reliable signals for rapid EEG-based brain-computer interface (BCI) control. While a subject-specific SSVEP stimulus frequency optimization is ideal, this can be a tedious and time-consuming process. Thus, many studies select SSVEP stimulation frequencies somewhat arbitrarily. There is no standardized set of SSVEP stimulus frequencies or frequency selection method, and some studies even claim conflicting frequency ranges for optimal performance. In this work, 17 subjects were stimulated with an LED array that flashed according to a chirp-modulated signal having a frequency that varied linearly over the typical functional range of SSVEP. The resulting EEG was analyzed using canonical correlation analysis and a genetic algorithm was implemented to determine generalized stimulation frequency sets over a continuum of simulated multiclass BCI classification scenarios. The results show that distinct frequency feature groupings exist over the different multiclass scenarios, and that these groupings result in different information transfer rates. These offline results can provide a guide for generalized stimulus frequency selection for SSVEP-based BCIs with an arbitrary number of targets.