Three-Dimensional (3D) range cameras have recently appeared in the marketplace for use in surveillance (e.g. cameras affixed to inanimate objects) applications. We present FreeGlass™ as a wearable hands-free 3D gesture-sensing Digital Eye Glass system. FreeGlass comprises a head-mounted display with an infrared range camera, both connected to a wearable computer. It is based on the MannGlas™ computerized welding glass, which embodies HDR (High Dynamic Range) and AR (Augmented/Augmediated Reality). FreeGlass recontextualizes the 3D range camera as a sousveillance (e.g. cameras attached to people) camera. In this sousveillance context, the range camera is worn by the user and shares the same point-of-view as the user. Computer vision algorithms therefore benefit from the use of the range camera to allow image segmentation by using both the infrared and depth information from the device for 3D hand gesture recognition system. The gesture recognition is then accomplished by using a neural network on the segmented hand. Recognized gestures are used to provide the user with interactions in an augmediated reality environment. Additionally, we present applications of FreeGlass for serendipitous gesture recognition in everyday life, as well as for interaction with real-world objects (with and without gesture recognition). A plurality of FreeGlass units can be used together, each sensor having a different spreading sequence, or the like, so that a number of people can collaborate and share the same or similar Augmediated Reality space(s).

This paper presents the invention and implementation of 3D (Three Dimensional) HDR (High Dynamic Range) sensing, along with examples. We propose a method of 3D HDR veillance (sensing, computer vision, video capture, or the like) by integrating tonal and spatial information obtained from multiple HDR exposures for use in conjunction with one or more 3D cameras. In one embodiment, we construct a 3D HDR camera from multiple 3D cameras such as Kinect sensors. In this embodiment the 3D cameras are arranged in a fixed array, such that the geometric relationships between them remain constant over time. Only a single camera calibration step is required at the initial time of assembling and fixing the cameras into the array. Preferably the cameras either view from the same position through beam splitters, or are fixed close to one another, so that they capture approximately the same subject matter. The system is designed so the cameras each capture a differently exposed image or video of approximately the same subject matter. In one embodiment, two Kinect cameras are attached together facing in the same direction, with an ND (Neutral Density) filter over one of them, so as to obtain a darker exposure. The dark and light exposures are combined to obtain more accurate 3D sensing in high contrast scenes. In another embodiment, a single 3D camera is exposure-sequenced (alternating light and dark exposures). 3D HDR might, more generally, be incorporated into existing 3D cameras, resulting in a new kind of 3D sensor that can work in nearly any environment, including high contrast scenes such as outdoor scenes, or scenes where a bright light is shining directly into the sensor.

We present highly parallelizable and computationally efficient High Dynamic Range (HDR) image compositing, reconstruction, and spatotonal mapping algorithms for processing HDR video. We implemented our algorithms in the EyeTap Digital Glass electric seeing aid, for use in everyday life. We also tested the algorithms in extreme dynamic range situations, such as, electric arc welding. Our system runs in real-time, and requires no user intervention, and no fine-tuning of parameters after a one-time calibration, even under a wide variety of very difficult lighting conditions (e.g. electric arc welding, including detailed inspection of the arc, weld puddle, and shielding gas in TIG welding). Our approach can render video at 1920x1080 pixel resolution at interactive frame rates that vary from 24 to 60 frames per second with GPU acceleration. We also implemented our system on FPGAs (Field Programmable Gate Arrays) for being miniaturized and built into eyeglass frames.

We present a way finding system that uses a range camera and an array of vibrotactile elements which we built into a helmet.

This paper describes a musical instrument consisting of a physical process that acoustically generates sound from the material world (i.e. sound derived from matter such as solid, liquid, gas, or plasma) which is modified by a secondary input from the informatic world. This informatic input selects attributes such as the frequency range of the musical note being sounded, while the acoustic process is kept in close contact with the user, to ensure a high degree of expressivity. In one example, ice skates with acoustic pickups are used to play music while the skater simultaneously controls a bandpass filter with a hand-held keyer and wearable computer. Each skate works much like the bow on a violin, allowing the player to hit, scrape, rub, or "bow", the ice in various ways to create a wide variety of musical textures. Additionally the player can select sound samples on a per-note basis and then "scratch" out a melody or harmony (playing multiple samples at once) on the ice on the rink like a team of disk jockeys (DJs) working together to "scratch" an array of vinyl records. Because the grooves on an ice rink are made by the player in a freeform fashion, there is much more room for variations in musical timbres and textures than with the fixed grooves of a record. Rather than merely using the keyer to trigger musical notes through MIDI note on/note off commands, we create acoustic sound through a physical process such as skating, and then turn those physical sounds into musical notes with the handheld keyer that functions as a modifier input. This combination combines the expressivity of non-electrophonic musical instruments like the violin with the flexibility of electrophones like the sound synthesizer. As a further contribution of the paper, a general taxonomy of acoustic transducers and a link to physical organology is provided, in which the top-level of the taxonomy is the state-of-matter in which the transducer operates.

We introduce a new musical instrument in which computation is used to modify acoustically generated sounds. The acoustically generated sounds originate from real physical objects in the user's environment. These sounds are picked up by one or more microphones connected to a camera phone which filters the sounds using filters whose coefficients change in response to subject matter present in view of the camera. In one example, a row of 12 image processing zones is presented such that sounds originating from real world objects in the first zone are mapped to the first note on a musical scale, sounds originating from the second zone are mapped to the second note of the musical scale, and so on. Thus a user can hit a cement wall or sidewalk, or the ground, and the camera phone will transform the resulting sound (e.g. a dull "thud") into a desired sound, such as the sound of tubular bells, chimes, or the like. Note that the instrument is not an electronic instrument (i.e. not an Electrophone in the Hornbostel Sachs sense) because the sound originates acoustically and is merely filtered toward the desired note. This plays upon the acoustic qualities and physicality of the originating media. For example, if we strike the ground abruptly, the sound resembles that of a bell being hit abruptly. If we rub the ground, the sound resembles that of rubbing a bell. We can scrape the ground in various ways to obtain various sounds that differ depending on which of the camera's zones we're in, as well as the physical properties of the ground itself. These experiences can be shared across "cyborgspace" to effectively blur the boundary between the real and virtual worlds. We present an aquatic instrument that plays upon jets of water, where it is the filter coefficients of the transform that are shared. This allows both users to play the instrument in the jets of water of different public fountains but still experience the same musical qualities of the instrument, and share the physical experience of playing in a fountain despite geographic distances.

We present "equiveillance" as a conceptual framework for understanding the balance between surveillance and sousveillance. In addition to this conceptual framework we also present a practical embodiment of equiveillance in the form of a new program called "cyborglogger" ('glogger) that runs on most modern camera phones, along with a server architecture to support 'glogger. Finally we show how the 'glogger program and server architecture create sousveillance communities. Cyborglogger implements features that are ideal for sousveillance such as including continuous capture and real-time upload aimed at communicating personal day-to-day narratives of everyday experiences. The server architecture includes a custom-built community web site that allows a sousveillance community to interact, in real time, with 'glogs from various users of the system. Participants can use their camera phones to display output from other camera phones, resulting in peer-to-peer sharing of visual narratives. This real-time live monitoring creates a social commentary and discourse that runs parallel to the widespread surveillance already present in the world around us. Unlike surveillance, which often happens in secrecy, our tools for sousveillance are freely available and moves personal experience capture into the realm of the everyday, with an open forum for public discourse. We thus explore how users engaging in sousveillance with the 'glogger application provide a balance to existing well-established surveillance practices by examining the philosophical questions that arise from the new artistic practice of sousveillance.