Graphics and vision are approximate inverses of each other: ordinarily graphics processing units (GPUs) are used to convert ldquonumbers into picturesrdquo (i.e. computer graphics). In this paper, we discuss the use of GPUs in approximately the reverse way: to assist in ldquoconverting pictures into numbersrdquo (i.e. computer vision). For graphical operations, GPUs currently provide many hundreds of gigaflops of processing power. This paper discusses how this processing power is being harnessed for image processing and computer vision, thereby providing dramatic speedups on commodity, readily available graphics hardware. A brief review of algorithms mapped to the GPU by using the graphics API for vision is presented. The NVIDIA CUDA programming model is then introduced as a way of expressing program parallelism without the need for graphics expertise.

We introduce a new musical instrument in which computation is used to modify acoustically generated sounds. The acoustically generated sounds originate from real physical objects in the user's environment. These sounds are picked up by one or more microphones connected to a camera phone which filters the sounds using filters whose coefficients change in response to subject matter present in view of the camera. In one example, a row of 12 image processing zones is presented such that sounds originating from real world objects in the first zone are mapped to the first note on a musical scale, sounds originating from the second zone are mapped to the second note of the musical scale, and so on. Thus a user can hit a cement wall or sidewalk, or the ground, and the camera phone will transform the resulting sound (e.g. a dull "thud") into a desired sound, such as the sound of tubular bells, chimes, or the like. Note that the instrument is not an electronic instrument (i.e. not an Electrophone in the Hornbostel Sachs sense) because the sound originates acoustically and is merely filtered toward the desired note. This plays upon the acoustic qualities and physicality of the originating media. For example, if we strike the ground abruptly, the sound resembles that of a bell being hit abruptly. If we rub the ground, the sound resembles that of rubbing a bell. We can scrape the ground in various ways to obtain various sounds that differ depending on which of the camera's zones we're in, as well as the physical properties of the ground itself. These experiences can be shared across "cyborgspace" to effectively blur the boundary between the real and virtual worlds. We present an aquatic instrument that plays upon jets of water, where it is the filter coefficients of the transform that are shared. This allows both users to play the instrument in the jets of water of different public fountains but still experience the same musical qualities of the instrument, and share the physical experience of playing in a fountain despite geographic distances.

We present "equiveillance" as a conceptual framework for understanding the balance between surveillance and sousveillance. In addition to this conceptual framework we also present a practical embodiment of equiveillance in the form of a new program called "cyborglogger" ('glogger) that runs on most modern camera phones, along with a server architecture to support 'glogger. Finally we show how the 'glogger program and server architecture create sousveillance communities. Cyborglogger implements features that are ideal for sousveillance such as including continuous capture and real-time upload aimed at communicating personal day-to-day narratives of everyday experiences. The server architecture includes a custom-built community web site that allows a sousveillance community to interact, in real time, with 'glogs from various users of the system. Participants can use their camera phones to display output from other camera phones, resulting in peer-to-peer sharing of visual narratives. This real-time live monitoring creates a social commentary and discourse that runs parallel to the widespread surveillance already present in the world around us. Unlike surveillance, which often happens in secrecy, our tools for sousveillance are freely available and moves personal experience capture into the realm of the everyday, with an open forum for public discourse. We thus explore how users engaging in sousveillance with the 'glogger application provide a balance to existing well-established surveillance practices by examining the philosophical questions that arise from the new artistic practice of sousveillance.

Graphics and vision are approximate inverses of each other: ordinarily Graphics Processing Units (GPUs) are used to convert into (i.e. computer graphics). In this paper, we propose using GPUs in approximately the reverse way: to assist in converting pictures into numbers (i.e. computer vision). The OpenVIDIA project uses single or multiple graphics cards to accelerate image analysis and computer vision. It is a library and API aimed at providing a graphics hardware accelerated processing framework for image processing and computer vision. OpenVIDIA explores the creation of a parallel computer architecture consisting of multiple Graphics Processing Units (GPUs) built entirely from commodity hardware. OpenVIDIA uses multiple Graphic.Processing Units in parallel to operate as a general-purpose parallel computer architecture. It provides a simple API which implements some common computer vision algorithms. Many components can be used immediately and because the project is Open Source, the code is intended to serve as templates and examples for how similar algorithms are mapped onto graphics hardware. Implemented are image processing techniques (Canny edge detection, filtering), image feature handling (identifying and matching features) and image registration, to name a few.

This paper shows speedups attained by using computer graphics hardware for implementation of computer vision algorithms by efficiently mapping mathematical operations of computer vision onto modem computer graphics architecture. As an example computer vision algorithm, we implement a real-time projective camera motion tracking routine on modern, GeForce FX class GPUs (graphics processing units). Algorithms are implemented using OpenGL and the nVIDIA Cg fragment shaders. Trade-offs between computer vision requirements and GPU resources are discussed. Algorithm implementation is examined closely, and hardware bottlenecks are addressed to examine the performance of GPU architecture for computer vision. It is shown that significant speedups can be achieved, while leaving the CPU free for other signal processing tasks. Applications of our work include wearable, computer mediated reality systems that use both computer vision and computer graphics, and require realtime processing with low-latency and high throughput provided by modem GPUs.

Realtime image processing provides a general framework for robust mediated reality problems. This paper presents a realtime mediated reality system that is built upon realtime image processing algorithms. It has been shown that the graphics processing unit (GPU) is capable of efficiently performing image processing tasks. The system presented uses a parallel GPU architecture for image processing that enables realtime mediated reality. Our implementation has many benefits; the graphics hardware has high throughput and low latency; the GPU's are not prone to jitter. Additionally, the CPU is kept available for user applications. The system is easily constructed, consisting of readily available commodity hardware.

For many years, the brightness constancy constraint equation (BCCE) has been used for optical flow and related computer vision computations. However, almost all cameras have some kind of automatic exposure feature such as automatic gain control (AGC), so that the overall exposure level of the image varies as the camera is aimed at brighter or darker portions of a scene. Moreover, because most cameras have some kind of unknown nonlinear response function, the change due to AGC cannot be captured by merely applying a multiplicative constant to the pixels of each image. We propose, therefore, a lightspace change constraint equation (LCCE) that accounts for exposure change (AGC) together with the nonlinear response function of the camera. The response function can be automatically "learned" by an intelligent image processing system presented with differently exposed captures of the same subject matter in overlapping regions of registered images. Most importantly, a logarithmic lightspace change constraint equation (LLCCE) is shown to have a very simple mathematical formulation. The LCCE (and log LCCE) is applied to the estimation of the projective coordinate transformation between pairs of images in a sequence, and is compared with examples where the BCCE fails.

We present a way of making the wearing of a lifelong electrocardiographic health monitor fun for a user. The health monitor is coupled with a reality mediator device to create physiologically mediated reality, i.e. mediated reality which alters a user's audiovisual perception of the world based upon their own electrocardiographic waveform. This creates an interesting audiovisual experience for the user, playing upon the poetic narrative of combining cardio-centric metaphors pervasive in everyday life (the heart as a symbol of love and centrality, e.g. "get to the heart of the matter") with ubiquitous occular-centric metaphors such as "see the world from my point of view". This audiovisual experience is further enhanced by combining music which alters the visual perception and also heightens the user's emotional response to their experience and, in doing so, further affects their heart(beat).

We present a system which allows wearable computer users to share their views of their current environments with each other. Our system uses an EyeTap: a device which allows the eye of the wearer to function both as a camera and a display. A wearer, by looking around his/her environment, "paints" or "builds" an environment map composed of images from the EyeTap device, along with head-tracking information recording the orientation of each image. The head-tracking algorithm uses a featureless image motion estimation algorithm coupled with a head mounted gyroscope. The environment map is then transmitted to another user, who, through their own head-tracking EyeTap system, browses the first user's environment solely by head motion, seeing the environment as though it were their own. As a result of browsing the transmitted environment map, the viewer builds and extends his/her own environment map, and thus this is a data-producing head-tracking system. These environment maps can then be shared reciprocally between wearers.

When we ask the fundamental question "What does a camera measure?", we arrive at the concept of quantimetric imaging, which uses a new quantimetric unit, q, characteristic of a particular camera (e.g. each kind of camera defines its own quantimetric unit q based on its spectral response, etc.). Fluctuations in interframe exposures, along a sequence of images, give rise to a comparametric relationship between successive pairs of images. This allows us to estimate the response function of the camera (to derive the quantimetric unit q) as well as the relative differences in exposure. A new method of video image processing that exploits multiple differently exposed pictures (frames of the video sequence) of overlapping subject matter is thus possible. The method may be used whenever a video camera having automatic exposure captures multiple frames of video with the same subject matter appearing in regions of overlap between at least some of the successive video frames. Since almost all cameras have an automatic exposure feature, typically center weighted, when a light object falls in the center of the frame the exposure is automatically decreased, whereas the exposure is automatically increased when the camera swings around to point at a darker object. Such fluctuations in gain may be used to estimate the camera's response function, to estimate exposure differences, to do quantimetric processing, as well as to obtain images having both extended dynamic range and extended dynamic domain.