We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. Our setup is simple, cheap, and easily reproducible, comprising of two infrared cameras, diffuse infrared LEDs, and any off-the-shelf retro-reflective material. The retro-reflector aids image segmentation, creating a strong contrast between the surface and any object in proximity. A new highly efficient stereo matching algorithm precisely estimates the 3D contours of interacting objects and the retro-reflective surfaces. A novel pipeline enables 3D finger, hand and object tracking, as well as gesture recognition, purely using these 3D contours. We demonstrate high-precision sensing, allowing robust disambiguation between a finger or stylus touching, pressing or interacting above the surface. This allows many interactive scenarios that seamlessly mix together freehand 3D interactions with touch, pressure and stylus input. As shown, these rich modalities of input are enabled on and above any retro-reflective surface, including custom "physical widgets" fabricated by users. We compare our system with Kinect and Leap Motion, and conclude with limitations and future work.

This paper presents a new sensing and interaction environment for post-stroke and upper extremity limb rehabilitation. The device is a combination of camera-based multitouch sensing and a supporting therapeutic software application that advances the treatment, provides feedback, and records a user's progress. The image-based analysis of hand position provided by a Microsoft Surface is used as an input into a tabletop game environment. Tailored image analysis algorithms assess rehabilitative hand movements. Visual feedback is provided in a game context. Experiments were conducted in a sub-acute rehabilitation center. Preliminary user studies with a stroke-afflicted population determined essential design criteria. Hand and wrist sensing, as well as the goals of the supporting game environment, engage therapeutic flexion and extension as defined by consulted physicians. Participants valued personalization of the activity, novelty, reward and the ability to work at their own pace in an otherwise repetitive therapeutic task. A "character" - game element personifying the participant's movement - was uniquely motivating relative to the media available in the typical therapeutic routine.

MonoFusion allows a user to build dense 3D reconstructions of their environment in real-time, utilizing only a single, off-the-shelf web camera as the input sensor. The camera could be one already available in a tablet, phone, or a standalone device. No additional input hardware is required. This removes the need for power intensive active sensors that do not work robustly in natural outdoor lighting. Using the input stream of the camera we first estimate the 6DoF camera pose using a sparse tracking method. These poses are then used for efficient dense stereo matching between the input frame and a key frame (extracted previously). The resulting dense depth maps are directly fused into a voxel-based implicit model (using a computationally inexpensive method) and surfaces are extracted per frame. The system is able to recover from tracking failures as well as filter out geometrically inconsistent noise from the 3D reconstruction. Our method is both simple to implement and efficient, making such systems even more accessible. This paper details the algorithmic components that make up our system and a GPU implementation of our approach. Qualitative results demonstrate high quality reconstructions even visually comparable to active depth sensor-based systems such as KinectFusion.

Liquid crystal display (LCD) technology will struggle to display high resolution on screens as big as whiteboards. Yet there is demand also for less power consumption, 3-D images, and that the display should be able to see gestures and feel touch. Most of these features are possible with projection and wedge light guides make projection slim, but picoprojectors are so dim that many will be needed for such a big screen. For now, we use instead a hybrid of technologies: light guides look out from behind an LCD to see where the viewers' hands and eyes are, and a collimated backlight lets us illuminate one view at a time to each eye. This lets us synthesize 3-D with achievable increases in LCD frame rate. We expect that this combination of a multiview 3-D display and a view-dependent rendered image will give us the potential to televise the experience of looking through a window.

A pressure sensitive computer keyboard is presented that independently senses the force level on every depressed key. The design leverages existing membrane technologies and is suitable for low-cost, high-volume manufacturing. A number of representative applications are discussed.

CitationBuxton and Myers (1986) compared a specialized document navigation device and a scrollbar. Their device tracked finger movements along two touch-sensitive areas: one for absolute movement and one for relative movement. They found that their system was faster when navigating a 5-page document. We identified finger-tracking issues with the Buxton and Myers device for larger documents (100 pages) and developed an alternative device employing a range sensor and rotary encoder to track finger movement. We ran a new experiment comparing the traditional scrollbar, the Buxton and Myers design, and our new design. Both the Buxton and Myers design and our new design were poorly received by users compared to the scrollbar. Our results indicate that in a large document, factors beyond finger tracking accuracy influence the performance of a device providing absolute movement. From these results, we identify possible improvements necessary to implement effective and practical absolute navigation devices.