Because it is difficult to precisely specify complex objectives, reinforcement learning policies are often optimized using proxy reward functions that only approximate the true goal. However, optimizing proxy rewards frequently leads to reward hacking: the optimized reward function ceases to be a good proxy and the resulting policy performs poorly with respect to the unspecified true reward. Principled solutions to reward hacking have been impeded by the lack of a good definition for the problem. To address this gap, we introduce a definition of reward hacking based on the correlation between proxy and true rewards for states and actions seen by a “reference policy” that breaks down under optimization. We show that this definition captures reward hacking behavior across several realistic settings, including in reinforcement learning from human feedback (RLHF). Using our formulation, we show theoretically that regularization to the reference policy can effectively prevent reward hacking. While the current practice in RLHF applies a KL penalty between action distributions for this purpose, our theory suggests regularizing the χ2 divergence between the policies’ occupancy measures can be more effective. We intuitively show the benefits of this type of regularization and demonstrate that it better mitigates reward hacking in practice across four realistic settings, including RLHF. Our code is available at https://github.com/cassidylaidlaw/orpo.

Freezing of Gait (FoG) is one of the incapacitating motor symptoms that appear in patients with Parkinson’s Disease (PD). FoG manifests gait impairments and imposes unforeseen difficulties in commencing the locomotion. Frequent episodes of FoG often lead to fall-related injuries and impart dreadful health repercussions. Prediction of FoG before the occurrence could potentially increase the opportunities for pre-emptive cueing to mitigate or abate the episode of FoG. This paper presents a novel algorithm to predict the onset of FoG using a single ankle accelerometer sensor. Principally, we have focused on designing a lightweight algorithm to facilitate the real-time prediction of FoG in resource-limited hardware. A novel Genetic Algorithm (GA) is introduced as a feature selection method to enhance the algorithm’s performance. We have adapted the patient-invariant model which is a more viable approach in practical deployment. The algorithm is evaluated using the Daphnet dataset and achieved 88% FoG prediction accuracy with 1 second prediction time.

Freezing of Gait (FoG) is a paroxysmal and devitalizing symptom associated with Parkinson’s disease (PD). Episodes of FoG impedes gait and augments fall propensity, often leading to serious fall-injury. In this paper, we present a method for online detection of FoG using a wearable motion sensor. The novelty lies in utilizing the Empirical Wavelet Transform for signal denoising and incorporating the two new features to ameliorate the accuracy of the algorithm. Fundamentally, we have focused on a patient-independent model and leveraged a single ankle sensor which makes it a more feasible approach in terms of usability. Our model is evaluated on Daphnet dataset and achieved the average Sensitivity of.95 and Specificity of.70 with only a single sensor, demonstrating its immense potential.

Multi-fingered dexterous hands are versatile and capable of acquiring a diverse set of skills such as grasping, in-hand manipulation, and tool use. To fully utilize their versatility in real-world scenarios, we require algorithms and policies that can control them using on-board sensing capabilities, without relying on external tracking or motion capture systems. Cameras and tactile sensors are the most widely used on-board sensors that do not require instrumentation of the world. In this work, we demonstrate an imitation learning based approach to train deep visuomotor policies for a variety of manipulation tasks with a simulated five fingered dexterous hand. These policies directly control the hand using high dimensional visual observations of the world and propreoceptive observations from the robot, and can be trained efficiently with a few hundred expert demonstration trajectories. We also find that using touch sensing information enables faster learning and better asymptotic performance for tasks with high degree of occlusions. Video demonstration of our results are available at: https://sites.google.com/view/hand-vil/

For robots to operate in a three dimensional world and interact with humans, learning spatial relationships among objects in the surrounding is necessary. Reasoning about the state of the world requires inputs from many different sensory modalities including vision (V) and haptics (H). We examine the problem of desk organization: learning how humans spatially position different objects on a planar surface according to organizational “preference”. We model this problem by examining how humans position objects given multiple features received from vision and haptic modalities. However, organizational habits vary greatly between people both in structure and adherence. To deal with user organizational preferences, we add an additional modality, “utility” (U), which informs on a particular human's perceived usefulness of a given object. Models were trained as generalized (over many different people) or tailored (per person). We use two types of models: random forests, which focus on precise multi-task classification, and Markov logic networks, which provide an easily interpretable insight into organizational habits. The models were applied to both synthetic data, which proved to be learnable when using fixed organizational constraints, and human-study data, on which the random forest achieved over 90% accuracy. Over all combinations of {H, U, V} modalities, UV and HUV were the most informative for organization. In a follow-up study, we gauged participants preference of desk organizations by a generalized random forest organization vs. by a random model. On average, participants rated the random forest models as 4.15 on a 5-point Likert scale compared to 1.84 for the random model.