HoloDesk is an interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mirror and spatially aligned with the real-world for the viewer. Users easily reach into an interaction volume displaying the virtual image. This allows the user to literally get their hands into the virtual display and to directly interact with an spatially aligned 3D virtual world, without the need for any specialized head-worn hardware or input device. We introduce a new technique for interpreting raw Kinect data to approximate and track rigid (e.g., books, cups) and non-rigid (e.g., hands, paper) physical objects and support a variety of physics-inspired interactions between virtual and real. In particular the algorithm models natural human grasping of virtual objects with more fidelity than previously demonstrated. A qualitative study highlights rich emergent 3D interactions, using hands and real-world objects. The implementation of HoloDesk is described in full, and example application scenarios explored. Finally, HoloDesk is quantitatively evaluated in a 3D target acquisition task, comparing the system with indirect and glasses-based variants.

We introduce dynamic physical properties as an additional degree of freedom for passive tabletop controls. Using electromagnetic actuation, we manipulate attributes of tangibles on the fly, such as perceived weight, spring resistance, friction, and latching. We describe our actuation concepts, prototypes, and measurements showing that magnetic fields can change physical effects in a linear way. Controlled experiments reveal that participants can tactually distinguish four rendered resistance levels of a button prototype and easily detect dynamic detents in a continuous slider. Finally, we describe how adjustable physical properties in tangibles can enhance tabletop interaction.

We introduce FingerFlux, an output technique to generate near-surface haptic feedback on interactive tabletops. Our system combines electromagnetic actuation with permanent magnets attached to the user's hand. FingerFlux lets users feel the interface before touching, and can create both attracting and repelling forces. This enables applications such as reducing drifting, adding physical constraints to virtual controls, and guiding the user without visual output. We show that users can feel vibration patterns up to 35 mm above our table, and that FingerFlux can significantly reduce drifting when operating on-screen buttons without looking.

We present a system for the actuation of tangible magnetic widgets (Madgets) on interactive tabletops. Our system combines electromagnetic actuation with fiber optic tracking to move and operate physical controls. The presented mechanism supports actuating complex tangibles that consist of multiple parts. A grid of optical fibers transmits marker positions past our actuation hardware to cameras below the table. We introduce a visual tracking algorithm that is able to detect objects and touches from the strongly sub-sampled video input of that grid. Six sample Madgets illustrate the capabilities of our approach, ranging from tangential movement and height actuation to inductive power transfer. Madgets combine the benefits of passive, untethered, and translucent tangibles with the ability to actuate them with multiple degrees of freedom.

We present Silicone iLluminated Active Peripherals (SLAP), a system of tangible, translucent widgets for use on multitouch tabletops. SLAP Widgets are cast from silicone or made of acrylic, and include sliders, knobs, keyboards, and buttons. They add tactile feedback to multi-touch tables, improving input accuracy. Using rear projection, SLAP Widgets can be relabeled dynamically, providing inexpensive, battery-free, and untethered augmentations. Furthermore, SLAP combines the flexibility of virtual objects with physical affordances. We evaluate how SLAP Widgets influence the user experience on tabletops compared to virtual controls. Empirical studies show that SLAPWidgets are easy to use and outperform virtual controls significantly in terms of accuracy and overall interaction time.

We present SLAPbook, an application using SLAP, translucent and tangible widgets for use on vision-based multi-touch tabletops in Single Display Groupware (SDG) environments. SLAP stands for Silicone ILluminated Active Peripherals and includes widgets such as sliders, knobs, keyboards, and buttons. The widgets and tactile feedback to multi-touch tables while simultaneously providing dynamic relabeling to tangible objects using the table's rear projection. SLAPbook provides multiple users the ability to add and edit content to a guestbook, browse other peoples' entries, and access personal data using a token-based personalization system. Interaction with the table takes place in the personal and public space so that users can make use of personal and shared controls to perform separate and coordinative actions.

We present DRAGON, a direct manipulation interaction technique for frame-accurate navigation in video scenes. This technique benefits tasks such as professional and amateur video editing, review of sports footage, and forensic analysis of video scenes. By directly dragging objects in the scene along their movement trajectory, DRAGON enables users to quickly and precisely navigate to a specific point in the video timeline where an object of interest is in a desired location. Examples include the specific frame where a sprinter crosses the finish line, or where a car passes a traffic light. Through a user study, we show that DRAGON significantly reduces task completion time for in-scene navigation tasks by an average of 19-42% compared to a standard timeline slider. Qualitative feedback from users is also positive, with multiple users indicating that the DRAGON interaction felt more natural than the traditional timeline slider for in-scene navigation.