Recent combinations of interactive technology, humans, and water have resulted in “WaterHCI”. WaterHCI design seeks to complement the many benefits of engagement with the aquatic domain, by offering, for example, augmented reality systems for snorkelers, virtual reality in floatation tanks, underwater musical instruments for artists, robotic systems for divers, and wearables for swimmers. We conducted a workshop in which WaterHCI experts articulated the field's grand challenges, aiming to contribute towards a systematic WaterHCI research agenda and ultimately advance the field.

We introduce a method to enhance the performance of the high dynamic range (HDR) technique on audio signals by automatically controlling the gains of the individual signal channels. Automatic gain control (AGC) compensates the receiver's dynamic range by ensuring that the incoming signal is contained within the desired range while the HDR utilizes these multi-channel gains to extend the dynamic range of the composited signal. The results validate that the benefits given by each method are compounded when they are used together. In effect, we produce a dynamic high dynamic range (DHDR) composite signal. The HDR AGC method is simulated to show performance gains under various conditions. The method is then implemented using a custom PCB and a microcontroller to show feasibility in real-world and real-time applications.

We introduce a novel use of wearable technology to reconstruct and output high dynamic range (HDR) audio signals. The purpose of designing an end-to-end HDR audio wearable device is to create an audio device capable of replicating and even extending the dynamic range of human hearing. We also introduce a different method of visualizing audio by way of a rotary Sequential Wave Imprinting Machine (SWIM). The results show that real-time HDR audio processing and visualization are possible at the standard audio sampling rate of 44.1 kHz. Additionally, the quality of the recorded audio improved from the originally captured data after HDR processing. We further analyze the dependence of different audio gain combinations as well as the influence of the number of audio input channels on the quality of the HDR signal reconstruction in various acoustic environments.

We present an investigation into the relationship between steady-state visually-evoked potentials (SSVEPs) and the magnitude, distance, shape, and spatial location of the flashing stimulus relative to the participant. We use a wearable electroen-cephalography (EEG) device with the addition of an external occipital electrode for the experiments. SSVEP responses are extracted using the lock-in amplifier and fast Fourier transform algorithms. We then map the responses to what the human eye sees. Our experiments pinpoint the optimal range of stimulus parameters required for stable SSVEP response, identify failure states for flashing stimulus, as well as create a visual map, a vidmap, of the participant's ability to see. The results show that locating the stimulus at the participant's central vision elicits stronger SSVEP response compared to the peripheral vision.

The design and implementation of an integrated wearable face recognition and training system for prosopagnosia patients are presented. The purpose of this assistive technology is to provide real-time memory assistance and long-term rehabilitation. The real-time face recognition mode provides audio and visual notification of people who interact with the subject, while the at-home training mode combines features of mnemonic and perceptual training to help with prosopagnosia rehabilitation. In addition, a custom eye tracker is developed to determine the person whom the subject is making eye contact with within a crowd. Using the inverted face effect to mimic the difficulties of prosopagnosia patients, clinically healthy participants have shown improvements in their face-naming abilities. Early results indicate the system's potential to enrich the well-being of prosopagnosia patients.

We propose the use of SSVEP (Steady State Visual Evoked Potentials) in such a way as to cause the eye itself to function as a camera. By recording brainwaves in response to flickering visual stimuli, we read the “mind's eye” and are able to successfully reconstruct a visual image of subject matter that a person is looking at. In addition to causing the eye itself to function as a camera (see Fig 1), we also propose a new way to visualize and photograph human vision and human perception, i.e. a new way to see and understand human vision. This new “meta-vision” (vision of vision) has many applications in healthcare, from testing human vision to furthering our understanding of the brain.

The Sequential Wave Imprinting Machine (SWIM), invented by Steve Mann in the 1970s, offers naked-eye augmentedreality overlays with perfect alignment, through a Persistence of Exposure (PoE) of human vision, or photographic/videographic media.

We introduce the concept of coupled dynamic dynamic-range (D2R) compositing - that is, assembling sensor information, such as images or audio, from multiple dynamic-ranges that are allowed to move and change over time, as lighting conditions or sound conditions change over time in their amplitude-domain properties. We propose a feedback-control method to automatically adjust multiple exposure-value settings for HDR compositing, to increase the dynamic range of a sensory process, such as video capture. The method uses a cost function to express measurement uncertainty, and a "cross-uncertainty" metric between different exposures, fed into a dynamic control system. The system is designed to asymptotically approach an optimal distribution of camera exposure control settings, under varying lighting conditions and motion, to capture an extremely high dynamic range for HDR compositing.

We introduce a multidimensional space to quantify a signal or sensor, expressed in three sets of dimensions: the physical space-time domain, the Fourier space-time domain, and the amplitude domain. When combined with veillametrics we can measure a sensor's capacity-to-sense, as that capacity-to-sense propagates through space. Physical-Fourier-Amplitude visualizations can be generated for sensors, transducers, signal-processing systems, or physical phenomena, to yeild further insight beyond conventional frequency spectra or sensitivity charts. We examine applications in the sonic and optical modalities.

We propose a method to automatically adjust multiple exposure-value settings for HDR video compositing. The method uses imagespace-to-lightspace post-conversion and tonal extrapolation to iteratively select optimal exposure settings, as opposed to present systems, which use manually set settings. The limiting factor becomes the image sensor as opposed to the implementation of the algorithm, as in present systems. By choosing the exposure settings with the described algorithm, the high dynamic range sampling process can be adapted to various lighting environments. This algorithm is especially useful for ultra low power capture of optimally selected exposures which can be processed later using well-known HDR compositing methods.

Wearable computers and Generation-5 Digital Eye Glass easily recognize a user's own gestures, forming the basis for shared AR (Augmediated Reality). This Studio-workshop presents the latest in wearable AR, plus an historical perspective with new insights. Participants will sculpt 3D objects using hand gestures and create Unity 3D art+game objects using computational lightpainting.

Video cameras can only take photographs with limited dynamic range. One method to overcome this is to combine differently exposed images of the same subject matter (i.e. a Wyckoff Set), producing a High Dynamic Range (HDR) result. HDR digital photography started almost 20 years ago. Now, it is possible to produce HDR video in real-time, on both high-power CPU/GPU systems, as well as low-power FPGA boards. However, other FPGA implementations have relied upon methods that are less accurate than current CPU and GPU-based methods. Namely, the earlier FPGA approaches used weighted sum for image compositing. In this paper we provide a novel method for real-time HDR com-positing. As an essential part of an upgraded HDR video production system, the resulting system combines differently exposed video stream (of the same subject matter) in Full HD (1080p at 60fps) on a Kintex-7 FPGA. The proposed work flow, implemented with software written in C, estimates the camera response function according to its quadtree representation and generates the compositing circuit in Verilog HDL from a Wyckoff Set. This circuit consists of parts that perform addressing using multiplexer networks and estimation with bilinear interpolation. It is parameterizable by user-specified error constraints, allowing us to explore the trade-offs in resource usage and precision of the implementation. Here is an MD5 hash function sum generated for the rest of the paper: 07897e61027d15dc3600fadbccfbd67d, citation date: December 18, 2013.

Wearable computing can be used to both extend the range of human perception, and to share sensory experiences with others. For this objective to be made practical, engineering considerations such as form factor, computational power, and power consumption are critical concerns. In this work, we consider the design of a low-power visual seeing aid, and how to implement computationally-intensive computational photography algorithms in a small form factor with low power consumption. We present realtime an FPGA-based HDR (High Dynamic Range) video processing and filtering by integrating tonal and spatial information obtained from multiple different exposures of the same subject matter. In this embodiment the system captures, in rapid succession, sets of three exposures, “dark”, “medium”, and “light”, over and over again, e.g. “dark”, “medium”, “light”, “dark”, “medium”, “light”, and so on, at 60 frames per second. These exposures are used to determine an estimate of the photoquantity every 1/60th of a second (each time a frame comes in, an estimate goes out). This allows us to build a seeing aid that helps people see better in high contrast scenes, for example, while welding, or in outdoor scenes, or scenes where a bright light is shining directly into the eyes of the wearer. Our system is suitable for being built into eyeglasses or small camera-based, lifeglogging, or gesture-sensing pendants, and other miniature wearable devices, with low-power and compact circuits that can be easily mounted on the body.

Surveillance is a French word that means “to watch from above” (e.g. guards watching prisoners, police watching citizens, etc.). Another form of veillance (watching) is sousveillance, which means “to watch from below”. Whereas surveillance often means cameras on large entities (e.g. buildings and land), sousveillance often means cameras on small entities (e.g. individual people). The importance of sousveillance has come to the forefront recently with advancements in wearable computing and AR (augmented or augmediated reality). We characterize sousveillance from both an economic and moral perspective. We argue that societies that reject sousveil-lance will be impoverished, relative to those accepting sousveil-lance. We further argue that sousveillance as a form of social action has positive survival characteristics, so that in the long run, assuming that social and technological trends continue, the widespread adoption of sousveillance is inevitable.

Hospital acquired infections (HAIs) occur frequently in hospitalized patients. Staff compliance with hand hygiene (HH) policy during patient care has been shown to reduce HAIs. Currently, hospitals evaluate adherence to HH policies through direct observation by human auditors. The auditors do not have authority over staff members; thus, this process is more akin to sousveillance (watching from below) than surveillance (watching from above). When behaviour change occurs due to awareness of being observed, it is referred to as the “Hawthorne effect”. We quantified the effect of sousveillance by comparing the frequency of HH events with an auditor present to when no auditor was present. The data analysed in the present work is from an ongoing study on hand hygiene compliance monitoring. A monitoring network recorded 290,000 hand hygiene events over 6 months; auditors were present on five occasions for about an hour each visit. When using an exponential underlying distribution we found that the change in the HH event rate was significant (p <; 0.01) in 4 of the 5 auditor visits. Finally, with a hyper-exponential underlying distribution, 5 of 5 were significant (p <; 0.01). There was no significant change in the HH event rate among dispensers located within patient rooms (not visible to the auditor), irrespective of auditor's presence.

Identity awareness of research data has been introduced as a requirement for open research, transparency and reusability of research data in the context of eScience. This requirement includes the capability of linking research data to researchers, research projects and publications, and identifying the license for the data. This connectivity between research data and other elements in research ecosystems is required in order to make the data available and reusable beyond the initial research. In this paper, we examine these capabilities in the domains of veillance and social computing. The dataset cases presented in this paper articulate the challenges that researchers face as they seek to expose data created as a result of research activities.

The needs of a realtime vision aid require that it functions immediately, not merely for production of a picture or video record to be viewed later. Therefore the vision system must offer a high dynamic range (often hundreds of millions to one) that functions in real time. In compliment with the existing efficient and real-time HDR compositing algorithms, we propose a novel method for compressing High Dynamic Range (HDR) images by Per-Pixel Exposure Mapping (PPEM). Unlike any existing methods, PPEM only varies exposure to achieve tone mapping. It takes advantage of the camera response to enable exposure synthesis which we call Wyckoff set expansion. The method evaluates the synthetic exposures in a recursive pairwise process to generate a tone mapped HDR image. The results can be approximated using a look-up table, which can be used for real-time HDR applications.

Three-Dimensional (3D) range cameras have recently appeared in the marketplace for use in surveillance (e.g. cameras affixed to inanimate objects) applications. We present FreeGlass™ as a wearable hands-free 3D gesture-sensing Digital Eye Glass system. FreeGlass comprises a head-mounted display with an infrared range camera, both connected to a wearable computer. It is based on the MannGlas™ computerized welding glass, which embodies HDR (High Dynamic Range) and AR (Augmented/Augmediated Reality). FreeGlass recontextualizes the 3D range camera as a sousveillance (e.g. cameras attached to people) camera. In this sousveillance context, the range camera is worn by the user and shares the same point-of-view as the user. Computer vision algorithms therefore benefit from the use of the range camera to allow image segmentation by using both the infrared and depth information from the device for 3D hand gesture recognition system. The gesture recognition is then accomplished by using a neural network on the segmented hand. Recognized gestures are used to provide the user with interactions in an augmediated reality environment. Additionally, we present applications of FreeGlass for serendipitous gesture recognition in everyday life, as well as for interaction with real-world objects (with and without gesture recognition). A plurality of FreeGlass units can be used together, each sensor having a different spreading sequence, or the like, so that a number of people can collaborate and share the same or similar Augmediated Reality space(s).

This paper presents the invention and implementation of 3D (Three Dimensional) HDR (High Dynamic Range) sensing, along with examples. We propose a method of 3D HDR veillance (sensing, computer vision, video capture, or the like) by integrating tonal and spatial information obtained from multiple HDR exposures for use in conjunction with one or more 3D cameras. In one embodiment, we construct a 3D HDR camera from multiple 3D cameras such as Kinect sensors. In this embodiment the 3D cameras are arranged in a fixed array, such that the geometric relationships between them remain constant over time. Only a single camera calibration step is required at the initial time of assembling and fixing the cameras into the array. Preferably the cameras either view from the same position through beam splitters, or are fixed close to one another, so that they capture approximately the same subject matter. The system is designed so the cameras each capture a differently exposed image or video of approximately the same subject matter. In one embodiment, two Kinect cameras are attached together facing in the same direction, with an ND (Neutral Density) filter over one of them, so as to obtain a darker exposure. The dark and light exposures are combined to obtain more accurate 3D sensing in high contrast scenes. In another embodiment, a single 3D camera is exposure-sequenced (alternating light and dark exposures). 3D HDR might, more generally, be incorporated into existing 3D cameras, resulting in a new kind of 3D sensor that can work in nearly any environment, including high contrast scenes such as outdoor scenes, or scenes where a bright light is shining directly into the sensor.

This paper explores the interplay between surveillance cameras (cameras affixed to large-entities such as buildings) and sousveillance cameras (cameras affixed to small entities such as individual people), laying contextual groundwork for the social implications of Augmented/Augmediated Reality, Digital Eye Glass, and the wearable camera as a vision and visual memory aid in everyday life. We now live in a society in which we have both “the few watching the many” (surveillance), AND “the many watching the few” (sousveillance). Widespread sousveillance will cause a transition from our one-sided surveillance society back to a situation akin to olden times when the sheriff could see what everyone was doing AND everyone could see what the sheriff was doing. We name this neutral form of watching “veillance” - from the French word “veiller” which means “to watch”. Veillance is a broad concept that includes both surveillance (oversight) and sousveillance (undersight), as well as dataveillance, uberveillance, etc. It follows that: (1) sousveillance (undersight) is necessary to a healthy, fair, and balanced society whenever surveillance (oversight) is already being used; and (2) sousveillance has numerous moral, ethical, socioeconomic, humanistic/humanitarian, and practical justifications that will guarantee its widespread adoption, despite opposing sociopolitical forces.

Existential tinkering as a form of inquiry must be brought into the engineering curriculum at the university level, as well as into the education curricula in general, including early childhood education. This paper presents a methodology of education for people of all ages and abilities, including engineering education, through unstructured play, personal involvement (authenticity), expression, and exploration - playful tinkering - as forms of inquiry. Current methods of engineering education have too much emphasis on structure, creating rigidity that destroys the capacity for creativity and radical innovation and invention. We introduce “existinquiry/praxistemology” (existential tinkering as inquiry) as a learning methodology consisting of three parts: learning by thinking, learning by doing, and “learning by being” (existential education). The goal of this learning methodology is to create lateral thinkers who integrate ideas and methodologies normally associated with play, the arts, and the sciences, into the the creative thinking process of engineering and design. Our hope is that (1) existinquiry in engineeing education will create more competitive and versatile thinkers capable of solving more sophisticated problems; and (2) that combining concepts of engineering education with concepts of unstructured play that are normally associated with early childhood education, will result in more groundbreaking inventions. We playfully explore topics of Veillance (surveillance, sousveillance, reciprocal transparency, equiveillance/omniveillance, uberveillance, and dataveillance) and Natural User Interfaces with the fundamental Elements (earth, water, air, etc.). The methodologies are applicable to teaching engineering to children or adults of any age or ability.

The Society of Mind is the theory, emerging in the 1970s, that natural intelligence arises from the interactions of numerous simple agents, each of which, taken individually, is “mindless”, but, collectively, give rise to intelligence: “What magical trick makes us intelligent? The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle.” [Minsky 1988]

We propose a novel computational method for compositing low-dynamic-range (LDR) images into an high-dynamic-range (HDR) image by the use of a comparametric camera response function (CCRF), which is the response of a virtual HDR camera to multiple inputs. We demonstrate the use of this method with a simple probabilistic joint estimation model, that accounts for Gaussian noise, using iterative non-linear optimization to compute the CCRF. We achieve a speedup of ≈2500×, relative to direct calculation using the probabilistic model. This method can be implemented as a multidimensional lookup table, and enables realtime HDR video with any camera response function model, and any compositing algorithm based on pixel value and exposure.

We present highly parallelizable and computationally efficient High Dynamic Range (HDR) image compositing, reconstruction, and spatotonal mapping algorithms for processing HDR video. We implemented our algorithms in the EyeTap Digital Glass electric seeing aid, for use in everyday life. We also tested the algorithms in extreme dynamic range situations, such as, electric arc welding. Our system runs in real-time, and requires no user intervention, and no fine-tuning of parameters after a one-time calibration, even under a wide variety of very difficult lighting conditions (e.g. electric arc welding, including detailed inspection of the arc, weld puddle, and shielding gas in TIG welding). Our approach can render video at 1920x1080 pixel resolution at interactive frame rates that vary from 24 to 60 frames per second with GPU acceleration. We also implemented our system on FPGAs (Field Programmable Gate Arrays) for being miniaturized and built into eyeglass frames.

Technology has put us out of touch with nature. A goal of the CCEI (the Centre for Nature and Technology) is to invent, research, study, and teach technologies that facilitate connection with our natural world. One project of CCEI is Hydraulikos, the Water Labs, for people to touch and be touched by the most primordial of all media = water. Hydraulikos aims to be a place where science, quantum physics, and fluid mechanics come together with nature, the environment, the arts, culture and society, health, wellness, and innovation, as therapy for the mind and body... where music meets math, and the compartmentalized silos of academia are washed away with lateral thinking in a setting where the boundary between work and play can also dissolve.

In 2001 the term "Natural User Interface" (NUI) was coined to denote the use of wearable computing or of physical matter (solids, liquids, and gases) as direct user interfaces for metaphor-free computing ["Intelligent Image Processing", S. Mann, John Wiley & Sons, Inc., 2001]. An example of NUI is the idioscope, a highly expressive musical instrument based on continuous ("undigital") scratch input ["Natural Interfaces for Musical Expression...", S. Mann, in Proc. NIME 2007, Jun6--10, New York, NY, USA.].

We present a way finding system that uses a range camera and an array of vibrotactile elements which we built into a helmet.

We propose a water-based multitouch multimedia user-interface based on total-internal reflection as viewed by an underwater camera. The underwater camera is arranged so that nothing above the water surface is visible until a user touches the water, at which time anything that penetrates the water's surface becomes clearly visible. Our contribution is twofold: (1) computer vision using underwater cameras aided by total internal reflection; (2) hyperacoustic signal processing (frequency shifting) to capture the natural, acoustically-originating sounds of water rather than using synthetic sounds.

We propose the use of multiple sensors of different sensitivity that simultaneously sense the same signal. Outputs of these sensors are then combined in a way that allows the simultaneous sensing of large-signal and small-signal phenomena. This sensing methodology is applied to the andantephone, a musical instrument that allows a player to physically step through the notes of a song as if they were walking along the song's timeline. When you stop walking the music stops. If you walk faster the music plays faster. A new, more expressive design of andantephone was created using a wideband complementary set of geophones to detect seismic waves transmitted from human footsteps. Each tile in the andantephone has one or more high-frequency piezoelectric geophones that respond to small-signals, as well as one or more low-frequency carbon geophones that respond to large-signals. These sensors are subsequently connected to a real-time frequency-shifting system that shifts each geophone's output to the correct musical pitch or chord for a particular note in a song. The proposed HDR sensing principle may be applied to many different sensing scenarios.

Water hammer, a well known phenomenon occurring in water pipes and plumbing fixtures, is generally considered destructive and undesirable. We propose the use of water hammer for a musical instrument akin to hammered percussion instruments like hammered dulcimer, piano, etc. In one embodiment, the instrument comprises an array of mouths each for being struck with the open palm or fingers, each mouth connected to a separate hydraulic resonator. In another embodiment, we use a basin or pool of water as a multitouch user-interface where sounds made by water are acoustically sensed by an array of hydrophones (underwater listening devices). Using water itself as a touch surface creates a fun and playful user interface medium that captures the fluidity of the water's ebb and flow.

The andantephone is an instrument that allows a performed to physically step through a piece of music by walking. Each note or chord of the piece is assigned to one footstep, so expressively varying velocity varies the tempo in turn.

Graphics and vision are approximate inverses of each other: ordinarily graphics processing units (GPUs) are used to convert ldquonumbers into picturesrdquo (i.e. computer graphics). In this paper, we discuss the use of GPUs in approximately the reverse way: to assist in ldquoconverting pictures into numbersrdquo (i.e. computer vision). For graphical operations, GPUs currently provide many hundreds of gigaflops of processing power. This paper discusses how this processing power is being harnessed for image processing and computer vision, thereby providing dramatic speedups on commodity, readily available graphics hardware. A brief review of algorithms mapped to the GPU by using the graphics API for vision is presented. The NVIDIA CUDA programming model is then introduced as a way of expressing program parallelism without the need for graphics expertise.

This paper describes a musical instrument consisting of a physical process that acoustically generates sound from the material world (i.e. sound derived from matter such as solid, liquid, gas, or plasma) which is modified by a secondary input from the informatic world. This informatic input selects attributes such as the frequency range of the musical note being sounded, while the acoustic process is kept in close contact with the user, to ensure a high degree of expressivity. In one example, ice skates with acoustic pickups are used to play music while the skater simultaneously controls a bandpass filter with a hand-held keyer and wearable computer. Each skate works much like the bow on a violin, allowing the player to hit, scrape, rub, or "bow", the ice in various ways to create a wide variety of musical textures. Additionally the player can select sound samples on a per-note basis and then "scratch" out a melody or harmony (playing multiple samples at once) on the ice on the rink like a team of disk jockeys (DJs) working together to "scratch" an array of vinyl records. Because the grooves on an ice rink are made by the player in a freeform fashion, there is much more room for variations in musical timbres and textures than with the fixed grooves of a record. Rather than merely using the keyer to trigger musical notes through MIDI note on/note off commands, we create acoustic sound through a physical process such as skating, and then turn those physical sounds into musical notes with the handheld keyer that functions as a modifier input. This combination combines the expressivity of non-electrophonic musical instruments like the violin with the flexibility of electrophones like the sound synthesizer. As a further contribution of the paper, a general taxonomy of acoustic transducers and a link to physical organology is provided, in which the top-level of the taxonomy is the state-of-matter in which the transducer operates.

A robust method of camera response function estimation applicable to auto-exposure cameras is presented. The method uses the superposition property of light to solve for the response function directly using superposition constraints imposed by using different combinations of two (or more) lights that illuminate the same subject matter in varying proportions. An iterative optimization is utilised to simultaneously recover the exposure ratios of the images and the camera response function. Previously published multiple exposure methods that simultaneously estimate exposure ratio and response function suffer from a fundamental ambiguity. The use of the proposed superposition constraints solves this problem. We introduce a simple method for combining both superposition and homogeneity (from multiple exposure techniques) to accurately recover the response function from auto-exposure cameras.

The paper proposes a method of compressing floating-point images of arbitrary precision. The concept of floating point images is used frequently in such areas as high dynamic range imaging, where pixel data stored as 8 or 12-bit integers are insufficient. The compression scheme presented in the paper organizes the floating point data in a manner such that already existing compression algorithms such as JPEG or Zlib compression may be used once the data re-organization has taken place. The paper compares the result to a popular (but restrictive) form of image compression, openEXR, and shows significant gains over this format. Furthermore, the compression scheme presented is scalable to deal with floating point images of arbitrary precision.

The hydraulophone is a fun-to-play self-cleaning keyboard instrument in which each key is a water jet. Many hydraulophones are already equipped with an array of underwater microphones (hydrophones), to pick up the turbulent sound from water inside musical sounding mechanisms under each water jet. Accordingly, we propose to make greater use of the sound of the water flow.

Present-day sampling music keyboards are electronic instruments that fall under the last (5th) category of the Hornbostel Sachs musical instrument classification scheme. Conversely, we first propose an entirely acoustic/mechanical mellotron-like sampling keyboard instrument that neither uses nor involves electricity in any way. Instrument voice/voicing is changed by replacing mechanical storage media similar to Edison phonograph cylinders, gramophone disks, or vinyl records that were commonly used from 1870 to 1980. We next propose a fluid version of our instrument in which hydraulic (water) action is used to fluidly index into the mechanically stored samples, again, without the use of electrical components. Finally, we present a computerized version of our instrument in which digital signal processing is used to obtain fluidly continuous control of musical sampling from a hydraulic keyboard in which each key is a water jet. The final result gives rise to new musically expressive capabilities for continuously flowing manipulation of music samples. Moreover, we propose versions of the computerized instrument that derive the initial sound source from the water itself, such that the instrument is not an electrophone.

We introduce a new musical instrument in which computation is used to modify acoustically generated sounds. The acoustically generated sounds originate from real physical objects in the user's environment. These sounds are picked up by one or more microphones connected to a camera phone which filters the sounds using filters whose coefficients change in response to subject matter present in view of the camera. In one example, a row of 12 image processing zones is presented such that sounds originating from real world objects in the first zone are mapped to the first note on a musical scale, sounds originating from the second zone are mapped to the second note of the musical scale, and so on. Thus a user can hit a cement wall or sidewalk, or the ground, and the camera phone will transform the resulting sound (e.g. a dull "thud") into a desired sound, such as the sound of tubular bells, chimes, or the like. Note that the instrument is not an electronic instrument (i.e. not an Electrophone in the Hornbostel Sachs sense) because the sound originates acoustically and is merely filtered toward the desired note. This plays upon the acoustic qualities and physicality of the originating media. For example, if we strike the ground abruptly, the sound resembles that of a bell being hit abruptly. If we rub the ground, the sound resembles that of rubbing a bell. We can scrape the ground in various ways to obtain various sounds that differ depending on which of the camera's zones we're in, as well as the physical properties of the ground itself. These experiences can be shared across "cyborgspace" to effectively blur the boundary between the real and virtual worlds. We present an aquatic instrument that plays upon jets of water, where it is the filter coefficients of the transform that are shared. This allows both users to play the instrument in the jets of water of different public fountains but still experience the same musical qualities of the instrument, and share the physical experience of playing in a fountain despite geographic distances.

We propose a single figure-of-merit measure of resolution of a digital imaging system based on the work of Gabor in communication theory. Gabor's work was largely inspired by Heisenberg's developments in quantum theory, most notably his uncertainty theorem of quantum mechanics. Gabor's results look simultaneously at the frequency and spatial domain of a signal, making it ideal for the measure of the modulation transfer function and point-spread function of an imaging system. As opposed to the crude "megapixel" measure which is bantered about in the marketplace, we suggest a figure-of-merit which more accurately represents the resolution of the system. Given that the resolution measure we propose is condensed into a single number rather than a function such as the modulation transfer function or the point spread function, it is our intent to propose this scientific evaluation as a means for typical consumers to fairly judge the resolution of a camera. Finally, we use this measure to compare common digital SLR cameras with varying lenses

This paper describes the construction of a MIDI interface that can be attached to an acoustic hydraulophone, or pneumatophone, or to an ordinary water fountain, or air fountain. This allows the fountain to be used as a haptic/tactile control surface or highly expressive fluid keyboard that provides gentle, soothing (soft) tactile feedback quite different from either the hard (solid) feedback of plastic keys, or the total lack of tactile feedback inherent in camera-based, vision-based, or proximity-sensing user-interfaces. We describe some of the difficulties in constructing such a multimedia input device, and some system architecture and signalling protocol considerations. In particular, we describe a modular self-healing architecture for fluid-based user interfaces. The architecture uses small microcontrollers, strung together, to obtain a large number of analog inputs in a small size low-power distributed system that is long and slender and can be easily waterproofed and sealed inside water pipes, manifolds, fluid chests, or fountains

The paper demonstrates a method of simultaneously testing the spatial and tonal resolution of a camera. Unlike the modulation transfer function which has been used in the past, the measure proposed rates the resolution as a single number rather than a response over varied resolution, simplifying the measure and making it possible to compare the performance of different cameras easily. Previous work was subject to perturbations caused by aliasing at high frequencies. The measure proposed is immune to such perturbations, and is therefore appropriate for measuring a wide range of image sensing systems

When the user of a digital camera records an image, there is some presumption that when displayed, there is an accurate tonal representation. However, this is almost never the case. The paper shows how capture and tonal reproduction of an image rarely match, as well as a brief history as to why this is true. Techniques for calibrating digital cameras tonally as well as calibrating viewing devices are presented. The paper demonstrates a method of recovering camera response curves easily and with precision. Monitor calibration is demonstrated in a means that allows an accurate display of light-linear images. Ultimately, the paper shows how the proposed methods of calibration for digital cameras and display devices results in the accurate representation of images

We propose a hydraulic user interface consisting of an array of spray jets and the appropriate fluid sensing and fluid flow control systems for each jet, so that the device functions as a fluid-based tactile user interface. Our array of fluid streams work like the keys on a keyboard, but where each fluid stream can also provide tactile feedback by dynamically modulating the pressure of the fluid spray, so that the keyboard is actually bi-directional (i.e. is both an input and an output device). A 104-jet version can be used as a fun and tactile "QWERTY..." style keyboard. More importantly, however, the device can also be used for applications, such as musical instruments, where its more expressive multi-dimensional input capabilities can be put to full use. One such instrument, the hydraulophone (hydrauliphone, hydraulaphone), is a hollow tubular object with a row of holes in it. It is played much like one would play a tin flute or recorder, by covering up the holes to restrict fluid flow. This gives rise to a fun new way of playing music by successively blocking water jets in a fountain, or while frolicking in a pool, or splash pad. Additionally, the hydraulophone can be used as a teaching tool to help children learn music by playing in the water. We demonstrate this teaching capability by way of an implementation of the arcade game "Touch Me" using a hydraulic user interface. We describe some of our present and upcoming installations of these devices in public parks, pools, and aquatic play areas

We propose the use of a hand-held camera flash as a tangible user interface to a photographic lightpainting algorithm/ system. Our system includes a method of photographic multiple-exposure lighting using a single camera flash to obtain the results that would otherwise require complicated and more expensive professional multiple-light photographic studio setups. The system is simple in its approach, because it provides a direct metaphor-free interaction between the user, the camera, and the computer. The system mimics traditional multiple exposure photography by computationally combining images linearly to produce visual art works.

I present a new way of teaching musical tempo and rhythm by writing out the music on a timeline along the ground, with, for example, chalk, in a form in which each beat of the music corresponds to one footstep. In some setups I use computer vision to track participants so that the music is actually generated by their footsteps moving through the space. In other embodiments I installed patio stones, leading to a musical garden, and outfitted each stone with a pressure sensor. I connected the pressure sensors to a central computer, which I programmed to step through a song, as people walk to the garden. Each footstep activates the next note in the song, so that there is perfect synchronization between the music and the speed of your walking (i.e. if you walk faster the song plays faster, if you stop walking the song stops, etc.). In one embodiment the computer controls an outdoor pipe-organ sculpture that I made from PVC pipes. Another provides a MIDI output to control a piano or other sound-producing device. Some versions of the sculpture are human-powered, either electrically, or wholly acoustically without the use of a computer.I also arranged various musical compositions suitable to this new form of art.This teaching method, together with various sculptural embodiments of it were found to break down social barriers and create cross-cultural and cross-generational ties. For example, children and their grand parents enjoyed walking through the gardens at Pine Hill Estates where a version of my sculpture is permanently installed.Other variations of the sculpture include arrays of hydraulophonic fountain jets that play a song in a water park when a person walks on the water. Each note or beat is triggered by a water pressure increase when one of the water jets is blocked by the foot of a user stepping on it.

We present "equiveillance" as a conceptual framework for understanding the balance between surveillance and sousveillance. In addition to this conceptual framework we also present a practical embodiment of equiveillance in the form of a new program called "cyborglogger" ('glogger) that runs on most modern camera phones, along with a server architecture to support 'glogger. Finally we show how the 'glogger program and server architecture create sousveillance communities. Cyborglogger implements features that are ideal for sousveillance such as including continuous capture and real-time upload aimed at communicating personal day-to-day narratives of everyday experiences. The server architecture includes a custom-built community web site that allows a sousveillance community to interact, in real time, with 'glogs from various users of the system. Participants can use their camera phones to display output from other camera phones, resulting in peer-to-peer sharing of visual narratives. This real-time live monitoring creates a social commentary and discourse that runs parallel to the widespread surveillance already present in the world around us. Unlike surveillance, which often happens in secrecy, our tools for sousveillance are freely available and moves personal experience capture into the realm of the everyday, with an open forum for public discourse. We thus explore how users engaging in sousveillance with the 'glogger application provide a balance to existing well-established surveillance practices by examining the philosophical questions that arise from the new artistic practice of sousveillance.

We present a musical keyboard that is not only velocity-sensitive, but in fact responds to absement (presement), displacement (placement), velocity, acceleration, jerk, jounce, etc. (i.e. to all the derivatives, as well as the integral, of displacement).Moreover, unlike a piano keyboard in which the keys reach a point of maximal displacement, our keys are essentially infinite in length, and thus never reach an end to their key travel. Our infinite length keys are achieved by using water jet streams that continue to flow past the fingers of a person playing the instrument. The instrument takes the form of a pipe with a row of holes, in which water flows out of each hole, while a user is invited to play the instrument by interfering with the flow of water coming out of the holes. The instrument resembles a large flute, but, unlike a flute, there is no complicated fingering pattern. Instead, each hole (each water jet) corresponds to one note (as with a piano or pipe organ). Therefore, unlike a flute, chords can be played by blocking more than one water jet hole at the same time. Because each note corresponds to only one hole, different fingers of the musician can be inserted into, onto, around, or near several of the instrument's many water jet holes, in a variety of different ways, resulting in an ability to independently control the way in which each note in a chord sounds.Thus the hydraulophone combines the intricate embouchure control of woodwind instruments with the polyphony of keyboard instruments.Various forms of our instrument include totally acoustic, totally electronic, as well as hybrid instruments that are acoustic but also include an interface to a multimedia computer to produce a mixture of sounds that are produced by the acoustic properties of water screeching through orific plates, as well as synthesized sounds.

The FUNtain (TM) is a new input device in which a user inputs data by direct interaction with fluid, such as with one or more air or water jets. There is usually a (re)strictometer for each jet that measures the degree of restriction of fluid flow for that jet, such that the time-varying changes in restrictometric quantities provide an input to another process. For example, 61 restrictometers are used to provide a directacting waterpipe organ as a musical instrument in which notes are played by blocking fluid (air or water) emerging from each of the 61 holes spaced along a copper pipe that forms the housing for the musical instrument. When used with liquids, such a fluid-user interface based on the image processing of water sprays is proposed to create a new form of input device. In particular, the methodology exploits the unique optical properties of fluids that become manifest when a fountain or spray is illuminated as it would typically be for an exhibit, display, or performance. Most notably, the form of illumination that is most desirable for aesthetic reasons turns out to assist simple image processing of the water flow. In this way, an interactive performance environment is created that is ideally suited to real-time image processing as a means of data input. The resulting methodology gives rise to a liquid-user-interface, played by one or more people, interacting with water jets, such as by blocking, partially blocking, diverting, or otherwise engaging with the spray.

Graphics and vision are approximate inverses of each other: ordinarily Graphics Processing Units (GPUs) are used to convert into (i.e. computer graphics). In this paper, we propose using GPUs in approximately the reverse way: to assist in converting pictures into numbers (i.e. computer vision). The OpenVIDIA project uses single or multiple graphics cards to accelerate image analysis and computer vision. It is a library and API aimed at providing a graphics hardware accelerated processing framework for image processing and computer vision. OpenVIDIA explores the creation of a parallel computer architecture consisting of multiple Graphics Processing Units (GPUs) built entirely from commodity hardware. OpenVIDIA uses multiple Graphic.Processing Units in parallel to operate as a general-purpose parallel computer architecture. It provides a simple API which implements some common computer vision algorithms. Many components can be used immediately and because the project is Open Source, the code is intended to serve as templates and examples for how similar algorithms are mapped onto graphics hardware. Implemented are image processing techniques (Canny edge detection, filtering), image feature handling (identifying and matching features) and image registration, to name a few.

"flUId" is a system for fluid-based tactile user interfaces with an array of fluid streams that work like the keys on a keyboard, but that can also provide a much richer and more expressive form of input by virtue of the infinitely diverse ways in which each fluid jet can be pressed, hit, restricted, or otherwise manipulated by a user. Additionally, if desired, flUId can provide tactile feedback by dynamically modulating the pressure of the fluid spray, so that the keyboard is actually bi-directional (i.e. is both an input and an output device). A 104-jet version can be used as a fun and tactile "QWERTY..." style keyboard. More importantly, however, flUId can also be used for applications, such as musical instruments, where its more expressive multi-dimensional input capabilities can be put to full use. One such instrument, the "FUNtain", is a hollow tubular object with a row of holes in it. It is played much like one would play a tin flute or recorder, by covering up the holes to restrict fluid flow. The FUNtain's fluid-based "keys" embody features of a keyboard instrument (piano or organ) as well as features of the tin flute, to create a hybrid water-pipe organflut ("waterpipe florgan") instrument. This gives rise to a fun new way of playing music by successivly blocking water jets in a fountain, while sitting in a hot tub, or while frolicking in a pool, or lake. Other examples of fluid-user-interface systems that were invented, designed, and built by the author, to enable direct interaction with fluids, as input media, are also discussed. Some of the input devices will work with either air or water, to provide the benefits of richly expressive input and dynamic tactile feedback in settings where use of wet fluid is inappropriate. FUNtains that use no computer or electricity are also presented as wholly acoustic musical instruments. Some of these embody "back to basics" postcyborg/undigital multimedia design elements such as fire, water, and air.

This paper shows speedups attained by using computer graphics hardware for implementation of computer vision algorithms by efficiently mapping mathematical operations of computer vision onto modem computer graphics architecture. As an example computer vision algorithm, we implement a real-time projective camera motion tracking routine on modern, GeForce FX class GPUs (graphics processing units). Algorithms are implemented using OpenGL and the nVIDIA Cg fragment shaders. Trade-offs between computer vision requirements and GPU resources are discussed. Algorithm implementation is examined closely, and hardware bottlenecks are addressed to examine the performance of GPU architecture for computer vision. It is shown that significant speedups can be achieved, while leaving the CPU free for other signal processing tasks. Applications of our work include wearable, computer mediated reality systems that use both computer vision and computer graphics, and require realtime processing with low-latency and high throughput provided by modem GPUs.

Realtime image processing provides a general framework for robust mediated reality problems. This paper presents a realtime mediated reality system that is built upon realtime image processing algorithms. It has been shown that the graphics processing unit (GPU) is capable of efficiently performing image processing tasks. The system presented uses a parallel GPU architecture for image processing that enables realtime mediated reality. Our implementation has many benefits; the graphics hardware has high throughput and low latency; the GPU's are not prone to jitter. Additionally, the CPU is kept available for user applications. The system is easily constructed, consisting of readily available commodity hardware.

A robust method of camera response function estimation, using pictures taken by differently illuminating the same subject matter, is presented. The method solves for the response function directly using superposition constraints imposed by different combinations of two (or more) lights to illuminate the same subject matter. Previous methods of computing camera response functions typically used differing exposures of identical subject matter, leading to uniqueness problems (underconstrained due to comparametric periodicity or fractal ambiguity). The method used in this paper overcomes this problem. Finally, we compare the method of the paper to previous methods and find the new method outperforms the previous work.

This is a personal narrative that began 30 years ago as a childhood hobby, of wearing and implanting various sensors, effectors, and multimedia computation in order re-define personal space and modify sensory perception computationally. This work involved the creation of various computational seeing aids that evolved into a new kind of visual art, using multimedia cyborglogs. Becoming at one with the machine, the author was able explore a new humanity at the nexus of cyberspace and the real world. The author presents what was discovered accidentally, as a result of facing discrimination. In particular, over the past 30 years, peer discrimination has decreased, while institutional and organized discrimination has intensified. Most notably, it was discovered that cyborg discrimination was most intense in establishments having the most surveillance. Rather than avoid such establishments, the author was able explore and capture unique aspects understand surveillance in new ways. The word sur-veillance denotes a God's eye view from on high (i.e. French for to watch from above). An inverse, called sous-veillance (French for to watch from below) explores what happens when cameras move from lamp posts and ceilings down eye level. Finally, it is suggested that new personal multimedia technologies, like mass-produced wearable cameraphones, can be used as tools for artists explore equiveillance by shifting this equilibrium between surveillance and sousveillance with inverse/reverse/accountability/recountability/continuability of continuous sur/sousveillance.

For many years, the brightness constancy constraint equation (BCCE) has been used for optical flow and related computer vision computations. However, almost all cameras have some kind of automatic exposure feature such as automatic gain control (AGC), so that the overall exposure level of the image varies as the camera is aimed at brighter or darker portions of a scene. Moreover, because most cameras have some kind of unknown nonlinear response function, the change due to AGC cannot be captured by merely applying a multiplicative constant to the pixels of each image. We propose, therefore, a lightspace change constraint equation (LCCE) that accounts for exposure change (AGC) together with the nonlinear response function of the camera. The response function can be automatically "learned" by an intelligent image processing system presented with differently exposed captures of the same subject matter in overlapping regions of registered images. Most importantly, a logarithmic lightspace change constraint equation (LLCCE) is shown to have a very simple mathematical formulation. The LCCE (and log LCCE) is applied to the estimation of the projective coordinate transformation between pairs of images in a sequence, and is compared with examples where the BCCE fails.

Personal Cybernetics and Humanistic Intelligence are new and rapidly growing fields of research in the area of human-computer interaction. These areas of research involve personal wearable imaging devices with intelligence that arises from the existence of a human user in the feedback loop of a computational process, in which the human user and the computational process are inextricably intertwined. Unlike the typical goal of Artificial Intelligence (AI), which is to emulate human intelligence with computers, Humanistic Intelligence (HI) creates a close synergy in which Intelligent Signal Processing is used to harness the processing power of the human brain. HI gives rise to a symbiosis between human and computer in which each uses the other within a closely coupled signal processing feedback loop. The computer performs basic low-level signal processing functions, using data obtained from a first person perspective (wearable camera, microphones, miniature wearable radar, biosensors, etc.), while the human performs high-level cognitive tasks, providing a computationally “mediated reality.” Personal Cybernetics and HI form a basis for augmenting, deliberately diminishing, or otherwise mediating the visual perception of reality. Although the visual modality is most often used in mediated reality systems based on current technology, other modalities such as touch, taste, and olfaction may be mediated as well. In the visual domain, a system that can augment, diminish, or otherwise alter the visual perception of reality is called a “Reality Mediator” (RM). By way of explanation, “virtual reality” creates a completely computer-generated environment, “augmented reality” uses an existing, real-life environment, and adds computer-generated information (virtual objects) thereto, “diminished reality” filters the environment (i.e., it alters real objects, replaces them with virtual ones, or renders them imperceptible), and mediated reality combines augmented and dimin

We present a way of making the wearing of a lifelong electrocardiographic health monitor fun for a user. The health monitor is coupled with a reality mediator device to create physiologically mediated reality, i.e. mediated reality which alters a user's audiovisual perception of the world based upon their own electrocardiographic waveform. This creates an interesting audiovisual experience for the user, playing upon the poetic narrative of combining cardio-centric metaphors pervasive in everyday life (the heart as a symbol of love and centrality, e.g. "get to the heart of the matter") with ubiquitous occular-centric metaphors such as "see the world from my point of view". This audiovisual experience is further enhanced by combining music which alters the visual perception and also heightens the user's emotional response to their experience and, in doing so, further affects their heart(beat).

We present a system which allows wearable computer users to share their views of their current environments with each other. Our system uses an EyeTap: a device which allows the eye of the wearer to function both as a camera and a display. A wearer, by looking around his/her environment, "paints" or "builds" an environment map composed of images from the EyeTap device, along with head-tracking information recording the orientation of each image. The head-tracking algorithm uses a featureless image motion estimation algorithm coupled with a head mounted gyroscope. The environment map is then transmitted to another user, who, through their own head-tracking EyeTap system, browses the first user's environment solely by head motion, seeing the environment as though it were their own. As a result of browsing the transmitted environment map, the viewer builds and extends his/her own environment map, and thus this is a data-producing head-tracking system. These environment maps can then be shared reciprocally between wearers.

When we ask the fundamental question "What does a camera measure?", we arrive at the concept of quantimetric imaging, which uses a new quantimetric unit, q, characteristic of a particular camera (e.g. each kind of camera defines its own quantimetric unit q based on its spectral response, etc.). Fluctuations in interframe exposures, along a sequence of images, give rise to a comparametric relationship between successive pairs of images. This allows us to estimate the response function of the camera (to derive the quantimetric unit q) as well as the relative differences in exposure. A new method of video image processing that exploits multiple differently exposed pictures (frames of the video sequence) of overlapping subject matter is thus possible. The method may be used whenever a video camera having automatic exposure captures multiple frames of video with the same subject matter appearing in regions of overlap between at least some of the successive video frames. Since almost all cameras have an automatic exposure feature, typically center weighted, when a light object falls in the center of the frame the exposure is automatically decreased, whereas the exposure is automatically increased when the camera swings around to point at a darker object. Such fluctuations in gain may be used to estimate the camera's response function, to estimate exposure differences, to do quantimetric processing, as well as to obtain images having both extended dynamic range and extended dynamic domain.

Humanistic Intelligence (HI) is defined an having two embodying elements. (1) It is a signal processing framework in which the human and the computer use each other as peripherals in a feedback loop. (2) The HI processing apparatus is Inextricably intertwined with the oat, natural capabilities of the human mind and body. Biofeedback (or biocybernetic) interfaces made possible through analysis of physiological signals, such as electroencephalograms (EEG), electrocardiograms (ECG) skin conductance (SC), blood pressure (BP) and respiration can provide a means to realize HI. The HI-Comp project builds upon previous work (e.g. the WearComp project) to develop a wearable computer that is physiologically responsive. HI-Comp requires signal analysis research In both offline and online situations to develop appropriate pattern detection algorithms. EEG signal analysis can be performed using Fast Fiourier Transform (FFT) for power analysis, and the Hilbert Transform for phase analysis. The first HI-Comp, the HI-Cam, is a wearable personal imaging application of HI that uses FFT power analysis on the EEG signal to control various features of an biological Interfaces that will lead to true extensions of the mind and body.

Multiple differently exposed pictures of the same subject matter arise naturally whenever a video camera having automatic exposure captures multiple frames of video with the same subject matter appearing in regions of overlap between at least some of the successive video frames. Almost all cameras have some kind of automatic exposure feature. Generally automatic exposure is center weighted, so that when a light object falls in the center of the frame the exposure is automatically decreased, whereas the exposure is automatically increased when the camera swings around to point at a darker object. In this paper, it is assumed that the spatial (e.g. projective) coordinate transformation between successive frames of the sequence is known (or equivalently that it is the identity), and the contribution of the paper is an efficient way to estimate the tonal relationship between successive frames of the sequence. In particular methods are proposed to simultaneously estimate the unknown camera response function, as well as the set of unknown relative exposure changes among images, up to a single unknown scalar constant. The method comprises a succession of guesses each of which is a refinement of the previous. The first guess is often sufficient, so that no initial solution needs to be provided by the user. Each subsequent guess is a least squares solution so that no computationally expensive optimization is required. Since the method makes use of all the data, it is extremely immune to noise. The method is tested against state-of-the art laboratory measurement instruments to confirm the accuracy of the results.

It is argued that, hidden within the flow of signals from typical cameras, through image processing, to display media, is a homomorphic filter. While homomorphic filtering is often desirable, there are some occasions where it is not. Thus, cancellation of this implicit homomorphic filter is proposed, through the introduction of an antihomomorphic filter. This concept gives rise to the principle of quantigraphic image processing, wherein it is argued that most cameras can be modeled as an array of idealized light meters each linearly responsive to a semi-monotonic function of the quantity of light received, integrated over a fixed spectral response profile. This quantity depends only on the spectral response of the sensor elements in the camera. A particular class of functional equations, called comparametric equations, is introduced as a basis for quantigraphic image processing. These are fundamental to the analysis and processing of multiple images differing only in exposure. The "gamma correction" of an image is presented as a simple example of a comparametric equation, for which it is shown that the underlying quantigraphic function does not pass through the origin. Thus, it is argued that exposure adjustment by gamma correction is inherently flawed, and alternatives are provided. These alternatives, when applied to a plurality of images that differ only in exposure, give rise to a new kind of processing in the "amplitude domain". The theoretical framework presented in this paper is applicable to the processing of images from nearly all types of modern cameras. This paper is a much revised draft of a 1992 peer-reviewed but unpublished report by the author, entitled "Lightspace and the Wyckoff principle."

Humanistic computing is proposed as a new signal processing framework in which the processing apparatus is inextricably intertwined with the natural capabilities of our human body and mind. The emphasis of this paper is on personal imaging applications of humanistic computing, to take a first step toward an intelligent wearable camera system that can allow one to effortlessly capture our day-to-day experiences, help us remember and see better, provide us with personal safety through crime reduction, and facilitate new forms of communication through collective connected humanistic computing. The author's wearable signal processing hardware currently provides the computational power of a UNIX workstation concealed within ordinary-looking eyeglasses and clothing. Thus it may be worn continuously during all facets of ordinary day-to-day living, so that, through long-term adaptation, it begins to function as a true extension of the mind and body.

An eyeglass-mounted camera system with wearable multimedia computer ('WearCam') was previously proposed. In particular, 'WearCam' contains two miniature cameras: one (wide-angle in landscape orientation) provides the overall contextual information from the wearer's perspective, while the other (telephoto, in portrait orientation) provides close-up details, such as faces. This 'bi-foveated' scheme was found to work well within the context of a proposed model of image motion characterized by a projective (homographic) coordinate transformation together with a gain transformation. Applications of 'WearCam' include personal safety (crime prevention and personal documentary), perceptual intelligence/situational awareness (in the context of personal wearable multimedia), and homographic modeling (wearable, tetherless computer-mediated reality). A pencigraphic image representation is presented where the photometric response function of the camera is determined to within a constant, and the registered images are assembled into a photometric environment map, yielding an estimate, to within a constant scale factor, of the number of photons of light coming from each angle, toward the wearer.

We present direct featureless methods for estimating the eight parameters of an "exact" projective (homographic) coordinate transformation to register pairs of images, together with the application of seamlessly combining a plurality of images of the same scene, resulting in a single image (or new image sequence) of greater resolution or spatial extent. The approach is "exact" for two cases of static scenes: (1) images taken from the same location of an arbitrary three-dimensional (3-D) scene, with a camera that is free to pan, tilt, rotate about its optical axis, and zoom, or (2) images of a flat scene taken from arbitrary locations. The featureless projective approach generalizes interframe camera motion estimation methods that have previously used a camera model (which lacks the degrees of freedom to "exactly" characterize such phenomena as camera pan and tilt) and/or which have relied upon finding points of correspondence between the image frames. The featureless projective approach, which operates directly on the image pixels, is shown to be superior in accuracy and the ability to enhance the resolution. The proposed methods work well on image data collected from both good-quality and poor-quality video under a wide variety of conditions (sunny, cloudy, day, night). These new fully automatic methods are also shown to be robust to deviations from the assumptions of static scene and no parallax.

Article "Smart clothing": wearable multimedia computing and "personal imaging" to restore the technological balance between people and their environments Share on Author: Steve Mann MIT Media Lab, Building E15-389, 20 Ames Street, Cambridge, MA MIT Media Lab, Building E15-389, 20 Ames Street, Cambridge, MAView Profile Authors Info & Claims MULTIMEDIA '96: Proceedings of the fourth ACM international conference on MultimediaFebruary 1997 Pages 163–174https://doi.org/10.1145/244130.244184Online:01 February 1997Publication History 42citation2,407DownloadsMetricsTotal Citations42Total Downloads2,407Last 12 Months48Last 6 weeks7 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Cameras with bellows give photographers flexibility for controlling perspective, but once the picture is taken, its perspective is set. We introduce 'virtual bellows' to provide control over perspective after a picture has been taken. Virtual bellows can be used to align images taken from different viewpoints, an important initial step in applications such as creating a high-resolution still image from video. We show how the virtual bellows, which implements the projective group, is an exact model fit to both pan and tilt. Specifically, we identify two important classes of image sequences accommodated by the virtual bellows. Examples of constructing high-quality stills are shown for the two cases: multiple frames taken of a flat object, and multiple frames taken from a fixed point.>

The authors have developed an expansion they call the chirplet transform. It has been successfully applied to a wide variety of signal processing applications, including radar and image processing. There has been a recent debate as to the relative merits of an affine-in-time (wavelet) transform and the classical short-time Fourier transform (STFT) for the analysis of nonstationary phenomena. Chirplet filters embody both the wavelet and STFT as special cases by decoupling the filter bandwidths and center frequencies. Chirplets, by their embodiment of affine geometry in the time-frequency (TF) plane, may also include shears in time and frequency (chirps) and even time-bandwidth product variation (noise bursts) if desired. The most general chirplets may be derived from one or more basic ('mother') chirplets by the transformations or perspective geometry in the TF plane.>