As Internet information portals become prevalent for both Internet and Intranet, most existing Internet Application Server architectures are not scalable to support the large amount of personalization, customization and content adaptation required. We propose a framework to capture the information and content dissemination process. Furthermore, we propose a methodology to map this process to a distributed application server environment. By fully exploiting the intersections of user preference at multiple content processing stages, this new framework enables high hit ratio on processing, storage, and transmission of content and thus scales well to support a large number of clients.

Growing numbers of pervasive devices are gaining access to the Internet and other information sources. However, much of the rich multimedia content cannot be easily handled by the client devices with limited communication, processing, storage and display capabilities. In order to improve access, we are developing a system for scalable delivery of multimedia. The system uses an InfoPyramid for managing and manipulating multimedia content composed of video, images, audio and text. The InfoPyramid manages the different variations of media objects with different fidelities and modalities and generates and selects among the alternatives in order to adapt the delivery to different client devices. We describe a system for scalable multimedia delivery for a variety of client devices, including PDAs, HHCs, smart phones, TV browsers and color PCs.

Content delivery over the Internet needs to address both the multimedia nature of the content and the capabilities of the diverse client platforms the content is being delivered to. We present a system that adapts multimedia Web documents to optimally match the capabilities of the client device requesting it. This system has two key components. 1) A representation scheme called the InfoPyramid that provides a multimodal, multiresolution representation hierarchy for multimedia. 2) A customizer that selects the best content representation to meet the client capabilities while delivering the most value. We model the selection process as a resource allocation problem in a generalized rate distortion framework. In this framework, we address the issue of both multiple media types in a Web document and multiple resource types at the client. We extend this framework to allow prioritization on the content items in a Web document. We illustrate our content adaptation technique with a web server that adapts multimedia news stories to clients as diverse as workstations, PDA's and cellular phones.

There is a growing need for developing a content description language for multimedia that improves searching, indexing and managing of the multimedia content. The MPEG group established the MPEG-7 effort to standardize the multimedia content interface. The proposed interface will bridge the gap between various types of content meta-data, such as content features, annotations, relationships, and the search engines. We develop a method of handling multimedia content description in a new multi-abstraction, multi-modal content representation framework called the InfoPyramid. The InfoPyramid facilitates the search, retrieval, manipulation, and transmission of multimedia data by providing a hierarchy for content descriptors. We illustrate the suitability of the InfoPyramid multimedia content description to MPEG-7 by examining four multimedia retrieval applications: a Web-image search engine, a satellite image retrieval system, an Internet content delivery system, and a TV news storage and retrieval system.

We present a novel scheme to match a video clip against a large database of videos. Unlike previous schemes that match videos based on image similarity, this scheme matches videos based on similarity of temporal activity, i.e., it finds similar "actions". Furthermore, it provides precise temporal localization of the actions in the matched videos. Video sequences are represented as a sequence of feature vectors called fingerprints. The fingerprint of the query video is matched against the fingerprints of videos in a database using sequential matching. The fingerprints are computed directly from compressed MPEG videos. The matching is much faster than real-time. We have used this scheme to find similar actions in sporting events, such as diving and baseball.

The authors present an automatic product 1D system ("VeggieVision"), intended to ease the produce checkout process. The system consists of an integrated scale and imaging system with a user-friendly interface. When a produce item is placed on the scale, an image is taken. A variety of features, color, texture (shape, density), are then extracted. These features are compared to stored "signatures" which were obtained by prior system training (either on-line or off-line). Depending on the certainty of the classification, the final decision is made either by the system or by a human from a number of choices selected by the system. Over 95% of the time, the correct produce classification is in the top four choices.

This paper introduces an analytical framework for studying some properties of model acquisition and recognition techniques based on indexing. The goal is to demonstrate that several problems previously associated with the approach can be attributed to the low dimensionality of invariants used. These include limited index selectivity. Excessive accumulation of votes in the look-up table buckets, and excessive sensitivity to quantization parameters. Theoretical results demonstrate that using high-dimensional, highly descriptive global invariants produces better results in terms of accuracy, false positive suppression, and computation time. A practical example of high-dimensional global invariants is introduced and used to implement a 2-D shape acquisition/recognition system. The acquisition/recognition system is based on a two-step table look-up mechanism. First, local curve descriptors are obtained by correlating image contour information at short range. Then, seven-dimensional global invariants are computed by correlating triplets of local curve descriptors at longer range. This experimental system is meant to illustrate the behavior of a high-dimensional indexing scheme. Indeed, its performance shows good agreement with the analytical model with respect to database size, fault tolerance, and recognition speed. Model acquisition time is linear to cubic in the number of object features. Object recognition time is constant to linear in the number of models in the database and linear to cubic in the number of features in the image. The system has been tested extensively. With more than 250 arbitrary shapes in the database. Unsupervised shape and subpart acquisition is demonstrated.>

The authors analyze how parameters of indexing based recognition systems affects their performance. The main result is that increasing the dimensionality of the indices leads to significantly improved discrimination, false positive suppression and reduced recognition times. With increase in index dimensionality, coarser quantization is required to allow index match in the presence of noise. The authors' analysis also allows estimation of votes thresholds for recognition and estimation of the amount of occlusion that can be tolerated by indexing schemes for given levels of recognition confidence.>

The authors present a method for 3-D object recognition from 2-D image sequences. The system uses feature points tracked over three or more views to compute structural invariants, which serve as 3-D shape representations. Object recognition is performed by using these Euclidean invariants as indices into a high-dimensional shape table. The use of indexing eliminates any need for matching models to images. In addition, the representation of 3-D objects is extracted from 2-D views, eliminating the cumbersome burden of having to obtain 3-D models. The proposed scheme was implemented using a mixed database of real and simulated objects. Experiments are outlined that show good recognition results on real objects and simulated objects corrupted with noise.>

A data-driven system for segmenting scenes into objects and their components is presented. This segmentation system generates hierarchies of features that correspond to structural elements such as boundaries and surfaces of objects. The technique is based on perceptual organization, implemented as a mechanism for exploiting geometrical regularities in the shapes of objects as projected on images. Edges are recursively grouped on geometrical relationships into a description hierarchy ranging from edges to the visible surfaces of objects. These edge groupings, which are termed collated features, are abstract descriptors encoding structural information. The geometrical relationships employed are quasi-invariant over 2-D projections and are common to structures of most objects. Thus, collations have a high likelihood of corresponding to parts of objects. Collations serve as intermediate and high-level features for various visual processes. Applications of collations to stereo correspondence, object-level segmentation, and shape description are illustrated.>

A homogeneous approach for acquisition, storage, and recognition of nonparametric shapes from images, using a novel shape representation based on shape autocorrelation operators is presented. A theoretical and experimental analysis of the computational complexity, recognition performance with increasing database size, and fault tolerance of the approach is presented. The system has been tested extensively with more than 300 arbitrary shapes in the database. Using a set of complex shapes, the recognition behavior with respect to occlusion, geometric transformation, and cluttered environments is studied. Unsupervised shape and subpart acquisition is demonstrated.>

A novel approach is presented for pruning the amount of search needed to match image features to object models. The technique relies on active networks which capture various visibility and geometric constraints between features of a model to prune these features from search space during matching. The networks, which can be efficiently implemented in Boolean logic, integrate harmoniously with the previous work in feature recognition and object matching. A method is proposed for clustering model features (vsets) and four types of constraints which assist in building the networks. The authors show, both analytically and empirically, the dramatic reduction in search provided by activation nets.>

The authors present a description framework, motivated by perceptual organization, which consists of representations of the geometrical organizations of intensity discontinuities. The descriptors in this framework are called collated features, and are groupings identified by perceptual organization. The processes that operate on the image to obtain these descriptors and the visual processes that utilize them are discussed. The detection of collated features is robust to local problems. The structural information encoded in them aids various visual tasks such as object segmentation, correspondence processes (stereo, motion, and model matching), and shape inferences. Two primary grouping processes, cocurvilinearity and symmetry are applied to intensity edge contours to generate the collated features, including curves, symmetries, and ribbons. These collations can be used to segment into visible surfaces of objects and to describe the 2D shapes of those surfaces.>

An algorithm is presented for error detection and correction of disparity, as a process separate from stereo matching, with the contention that matching is not necessarily the best way to utilize all the physical constraints characteristic to stereopsis. As a result of the bias in stereo research towards matching, vision tasks like surface interpolation and object modeling have to accept erroneous data from the stereo matchers without the benefits of any intervening stage of error correction. An algorithm which identifies all errors in disparity data that can be detected on the basis of figural continuity and corrects them is presented. The algorithm can be used as a postprocessor to any edged-based stereo matching algorithm, and can additionally be used to automatically provide quantitative evaluations on the performance of matching algorithms of this class.>

The authors describe an approach to perceptual grouping for detecting and describing 3-D objects in complex images and apply it to the task of detecting and describing complex buildings in aerial images. They argue that representations of structural relationships in the arrangements of primitive image features, as detected by the perceptual organization process, are essential for analyzing complex imagery. They term these representations collated features. The choice of collated features is determined by the generic shape of the desired objects in the scene. The detection process for collated features is more robust than the local operations for region segmentation and contour tracing. The important structural information encoded in collated features aids various visual tasks such as object segmentation, correspondence processes, and shape description. The proposed method initially detects all reasonable feature groupings. A constraint satisfaction network is then used to model the complex interactions between the collations and select the promising ones. Stereo matching is performed on the collations to obtain height information. This aids in further reasoning on the collated features and results in the 3-D description of the desired objects.>