Real-time recognition and localization of keyboard keys combined with real-time hand tracking and detection of finger-press actions are critical functions of vision-based virtual keyboards. These are not trivial tasks because there are numerous types of keyboard layouts and typist behaviors. Furthermore, key detection is especially difficult in uncontrolled real-world scenarios, where users' hands occlude significant portions of the keyboard while typing. Also, the detection finger press-down actions is further complicated in-the-wild by frequently changing camera viewpoints without direct line of sight to the finger pressing a key. In this work, we address the challenge of complete keyboard segmentation and detection of hand-occluded keys by proposing a deep learning approach for realtime finger-press detection, visible-key detection, and occluded-key recovery. Our models were trained and evaluated on Kaggle's Keyboard Key Detection dataset [1] and further empirically tested on the MSU Typing Behavior Database [3]. Our best models achieved 88% finger press-down detection and a key detection performance of 0.91 IOU and 97.8% mAP@75, while running at 60 fps.

This work aims to boost the performance of 3D human pose estimators trained in a weakly-supervised setting where there are much fewer annotated 3D poses than unlabeled video data. We formulate two self-supervised pose prior regularizers (PPR) - bone proportion and joint mobility constraints that are pose translation, scale, and rotation invariant. These regularizers, combined with bone symmetry loss, reduce overfitting to the 2D reprojection loss commonly used in weakly-supervised settings by optimizing the bone lengths and joint rotations of estimated 3D poses. Consequently, improving the accuracy of 3D pose estimators. The regularizers are network independent and can be applied to any network architecture without modifications. We apply our proposed PPR to VideoPose3D network [1] and show that it decreases the MPJPE by 24% when using ≤5% of annotated H36M [2] 3D data, improving state-of-the-art accuracy by 7.9 mm.