Infants receiving care in the Neonatal Intensive Care Unit (NICU) experience several painful procedures during their hospitalization. Assessing neonatal pain is difficult because the current standard for assessment is subjective, inconsistent, and discontinuous. The intermittent and inconsistent assessment can induce poor treatment and, therefore, cause serious long-term outcomes. In this paper, we present a comprehensive pain assessment system that utilizes facial expressions along with crying sounds, body movement, and vital sign changes. The proposed automatic system generates a standardized pain assessment comparable to those obtained by conventional nurse-derived pain scores. The system achieved 95.56 percent accuracy using decision fusion of different pain responses that were recorded in a challenging clinical environment. In addition to the decision fusion, we present the performance of multimodal assessment using other fusion schemes as well as a unimodal assessment approach. We also discuss the impact of different factors (e.g., gestational age) on pain, propose several group-specific models for pain assessment (e.g., pre-term and full-term models), and compare the performance of these models with the performance of general models. While further research is needed, our results show that the automatic assessment of neonatal pain is a viable and more efficient alternative to the manual assessment.

This paper presents the first investigation into the use of fully automated deep learning framework for assessing neonatal postoperative pain. It specifically investigates the use of Bilinear Convolutional Neural Network (B-CNN) to extract facial features during different levels of postoperative pain followed by modeling the temporal pattern using Recurrent Neural Network (RNN). Although acute and postoperative pain have some common characteristics (e.g., visual action units), postoperative pain has a different dynamic, and it evolves in a unique pattern over time. Our experimental results indicate a clear difference between the pattern of acute and postoperative pain. They also suggest the efficiency of using a combination of bilinear CNN with RNN model for the continuous assessment of postoperative pain intensity.

The current standard for assessing neonatal pain is discontinuous and suffers from inter-observer variations, which can result in delayed intervention and inconsistent treatment of pain. Therefore, it is critical to address the shortcomings of the current standard and develop continuous and less subjective pain assessment tools. Convolutional Neural Networks have gained much popularity in the last decades due to the wide range of its successful applications in medical image analysis, object recognition, and emotion recognition. In this paper, we propose a Neonatal Convolutional Neural Network, designed and trained end-to-end to detect neonatal pain. We evaluated the proposed network in two data sets of neonates and compared its performance to the performance of ResNet architecture in the same data sets. Our proposed method outperformed ResNet in recognizing neonates' pain and achieved around 91.00% accuracy. While further research is needed, our preliminary results suggest that the presented network can be used for automatic pain assessment, and possibly similar applications. It also suggests that the automatic recognition of neonatal pain provides a viable and more efficient alternative to the current standard of pain assessment.

Neonates do not have the ability to either articulate pain or communicate it non-verbally by pointing. The current clinical standard for assessing neonatal pain is intermittent and highly subjective. This discontinuity and subjectivity can lead to inconsistent assessment, and therefore, inadequate treatment. In this paper, we propose a multi-channel deep learning framework for assessing neonatal pain from videos. The proposed framework integrates information from two pain indicators or channels, namely facial expression and body movement, using convolutional neural network (CNN). It also integrates temporal information using a recurrent neural network (LSTM). The experimental results prove the efficiency and superiority of the proposed temporal and multi-channel framework as compared to existing similar methods.

The objective of this work is to estimate 3D human pose from a single RGB image. Extracting image representations which incorporate both spatial relation of body parts and their relative depth plays an essential role in accurate3D pose reconstruction. In this paper, for the first time, we show that camera viewpoint in combination to 2D joint locations significantly improves 3D pose accuracy without the explicit use of perspective geometry mathematical models. To this end, we train a deep Convolutional Neural Net-work (CNN) to learn categorical camera viewpoint. To make the network robust against clothing and body shape of the subject in the image, we utilized 3D computer rendering to synthesize additional training images. We test our framework on the largest 3D pose estimation bench-mark, Human3.6m, and achieve up to 20% error reduction on standing-pose activities compared to the state-of-the-art approaches that do not use body part segmentation.

Current practices of assessing infants' pain depends on the observer's subjective and potentially inconsistent judgment and requires continuous monitoring by care providers. Therefore, pain may be misinterpreted or totally missed leading to misdiagnosis and over/under treatment. To address these shortcomings, current practices can be augmented with a machine-based assessment system that monitors various pain cues and provides an objective and continuous assessment of pain. Although several machine-based pain assessment approaches have been introduced, the majority of these approaches assess pain based on analysis of a single pain indicator (i.e., unimodal). In this paper, we propose an automated multimodal approach that utilizes a combination of both behavioral and physiological pain indicators to assess infants' pain. We also present a unimodal approach that depends on a single pain indicator for assessment. Recogsnizing pain using a single indicator yielded 88%, 85%, and 82% overall accuracies for facial expression, body movement, and vital signs, respectively. Combining facial expression, body movement, and changes in vital signs (i.e., the multimodal approach) for assessment achieved 95% overall accuracy. These preliminarily results indicate that utilizing both behavioral and physiological pain indicators could provide a better and more reliable assessment of infants' pain.

The U.S. Defense Advanced Research Projects Agency's (DARPA) Neovision2 program aims to develop artificial vision systems based on the design principles employed by mammalian vision systems. Three such algorithms are briefly described in this paper. These neuromorphic-vision systems' performance in detecting objects in video was measured using a set of annotated clips. This paper describes the results of these evaluations including the data domains, metrics, methodologies, performance over a range of operating points and a comparison with computer vision based baseline algorithms.

We propose a novel method by using three new character features to detect text objects comprising two or more isolated characters in images and videos. A new text model is constructed to describe text objects. Each character is a part in the model and every two neighboring characters are connected by a link. Two characters and the link connecting them are defined as a text unit. For every candidate part, we compute character energy based on our observation that each character stroke forms two edges with high similarities in length, curvature, and orientation. For every candidate link, we compute link energy based on the similarities in color, size, stroke width, and spacing between characters that are aligned along a particular direction. For every candidate text unit, we combine character and link energies to compute text unit energy which measures the likelihood that the candidate is a text object. We evaluated the performance of the proposed method on ICDAR 2003/2005 dataset, Microsoft Street view dataset, and VACE video dataset. The experimental results demonstrate that our method can capture the inherent properties of characters and discriminate text from other objects effectively.

A novel wire detection algorithm for use by unmanned aerial vehicles (UAV) in low altitude urban reconnaissance is presented. This is of interest to urban search and rescue and military reconnaissance operations. Detection of wires plays an important role, because thin wires are hard to discern by tele-operators and automated systems. Our algorithm is based on identification of linear patterns in images. Most existing methods that search for linear patterns use a simple model of a line, which does not take into account the line surroundings. We propose the use of a robust Gaussian model to approximate the intensity profile of a line and its surroundings which allows effective discrimination of wires from other visually similar linear patterns. The algorithm is able to cope with highly cluttered urban backgrounds, moderate rain, and mist. Experimental results show a 17.7% detection improvement over the baseline.

Shape analysis is an active and important branch in computer vision research field. In recent years, many geometrical, topological, and statistical features have been proposed and widely used for shape-related applications. In this paper, based on the properties of Distance Transform, we present a new shape feature, weight of boundary point. By computing the shortest distances between boundary points and distance contours of a transformed shape, every boundary point is assigned a weight, which contains the interior structure information of the shape. To evaluate the proposed new shape feature, we tested the weighted boundary points on shape matching and shape decomposition. The experimental results demonstrated the validity.

In this paper, we propose a new unsupervised text detection approach which is based on Histogram of Oriented Gradient and Graph Spectrum. By investigating the properties of text edges, the proposed approach first extracts text edges from an image and localize candidate character blocks using Histogram of Oriented Gradients, then Graph Spectrum is utilized to capture global relationship among candidate blocks and cluster candidate blocks into groups to generate bounding boxes of text objects in the image. The proposed method is robust to the color and size of text. ICDAR 2003 text locating dataset and video frames were used to evaluate the performance of the proposed approach. Experimental results demonstrated the validity of our approach.

Visual surveillance is an active research topic in image processing. Transit systems are actively seeking new or improved ways to use technology to deter and respond to accidents, crime, suspicious activities, terrorism, and vandalism. Human behavior-recognition algorithms can be used proactively for prevention of incidents or reactively for investigation after the fact. This paper describes the current state-of-the-art image-processing methods for automatic-behavior-recognition techniques, with focus on the surveillance of human activities in the context of transit applications. The main goal of this survey is to provide researchers in the field with a summary of progress achieved to date and to help identify areas where further research is needed. This paper provides a thorough description of the research on relevant human behavior-recognition methods for transit surveillance. Recognition methods include single person (e.g., loitering), multiple-person interactions (e.g., fighting and personal attacks), person-vehicle interactions (e.g., vehicle vandalism), and person-facility/location interactions (e.g., object left behind and trespassing). A list of relevant behavior-recognition papers is presented, including behaviors, data sets, implementation details, and results. In addition, algorithm's weaknesses, potential research directions, and contrast with commercial capabilities as advertised by manufacturers are discussed. This paper also provides a summary of literature surveys and developments of the core technologies (i.e., low-level processing techniques) used in visual surveillance systems, including motion detection, classification of moving objects, and tracking.

Common benchmark data sets, standardized performance metrics, and baseline algorithms have demonstrated considerable impact on research and development in a variety of application domains. These resources provide both consumers and developers of technology with a common framework to objectively compare the performance of different algorithms and algorithmic improvements. In this paper, we present such a framework for evaluating object detection and tracking in video: specifically for face, text, and vehicle objects. This framework includes the source video data, ground-truth annotations (along with guidelines for annotation), performance metrics, evaluation protocols, and tools including scoring software and baseline algorithms. For each detection and tracking task and supported domain, we developed a 50-clip training set and a 50-clip test set. Each data clip is approximately 2.5 minutes long and has been completely spatially/temporally annotated at the I-frame level. Each task/domain, therefore, has an associated annotated corpus of approximately 450,000 frames. The scope of such annotation is unprecedented and was designed to begin to support the necessary quantities of data for robust machine learning approaches, as well as a statistically significant comparison of the performance of algorithms. The goal of this work was to systematically address the challenges of object detection and tracking through a common evaluation framework that permits a meaningful objective comparison of techniques, provides the research community with sufficient data for the exploration of automatic modeling techniques, encourages the incorporation of objective evaluation into the development process, and contributes useful lasting resources of a scale and magnitude that will prove to be extremely useful to the computer vision research community for years to come.

Text extraction in video documents, as an important research field of content-based information indexing and retrieval, has been developing rapidly since 1990s. This has led to much progress in text extraction, performance evaluation, and related applications. By reviewing the approaches proposed during the past five years, this paper introduces the progress made in this area and discusses promising directions for future research.

In surveillance situations, computer vision systems are often deployed to help humans perform their tasks more effectively. In a typical installation human observers are required to simultaneously monitor a number of video signals. Psychophysical research indicates that there are severe limitations in the ability of humans to monitor simultaneous signals. Do these same limitations extend to surveillance? We present a method for evaluating human surveillance performance in a situation that mimics demands of real world surveillance. A single computer monitor contained either nine display cells or four display cells. Each cell contained a stream of 2 to 4 moving objects. Observers were instructed to signal when a target event occurred - - when one of the objects entered a small square ldquoforbiddenrdquo region in the center of the display. Target events could occur individually or in groups of 2 or 3 temporally close events. The results indicate that the observers missed many targets (60%) when required to monitor 9 displays and many fewer when monitoring 4 displays (20%). Further, there were costs associated with target events occurring in close temporal succession. Understanding these limitations would help computer visions researchers to design algorithms and human-machine interfaces that result in improved overall performance.

We present a system to retrieve all clips from a meeting archive that show a particular individual speaking, using a single face or voice sample as the query. The system incorporates three novel ideas. One, rather than match the query to each individual sample in the archive, samples within a meeting are grouped first, generating a cluster of samples per individual. The query is then matched to the cluster, taking advantage of multiple samples to yield a robust decision. Two, automatic audio-visual association is performed which allows a bi-modal retrieval of clips, even when the query is uni-modal. Three, the biometric recognition uses individual-specific score distributions learnt from the clusters, in a likelihood ratio based decision framework that obviates the need for explicit normalization or modality weighting. The resulting system, which is completely automated, performs with 92.6% precision at 90% recall on a dataset of 16 real meetings spanning a total of 13 hours.

In this paper, we propose a new edge-based text verification approach for video. Based on the investigation of the relation between candidate blocks and their neighbor areas, the proposed approach first detects background edges in candidate blocks, then erases them by an edge tracking technique, and finally the candidate blocks containing too few remaining edges are eliminated as false alarms. Three measures for text detection evaluation in video were used to assess the performance of the proposed text verification approach. Experimental results on 50 broadcast news video clips demonstrate the validity of our approach.

This paper presents a bottom-up approach that combines audio and video to simultaneously locate individual speakers in the video (2D source localization) and segment their speech (speaker diarization), in meetings recorded by a single stationary camera and a single microphone. The novelty lies in using motion information from the entire body rather than just the face to perform these tasks, which permits processing nonfrontal views, unlike previous work. Since body movements do not exhibit instantaneous signal-level synchrony with speech, the approach targets long term co-occurrences between audio and video subspaces. First, temporal clustering of the audio produces a large number of intermediate clusters, each containing speech from only a single speaker. Then, spatial clustering is performed in the video frames of each cluster by a novel eigen-analysis method to find the region of dominant motion. This region is associated with the speech assuming that a speaker exhibits more movement than the listeners. Thus, partial diarization and localization is obtained from the intermediate clusters. Speech from an intermediate cluster is modeled by a mixture of Gaussians and the speaker's location is represented by an eigen-blob model. In the ensuing iterative clustering stage, the diarization and localization results are progressively refined by merging the closest pair of clusters and updating the models until a stop criterion is met. Ideally, each final cluster contains all the speech from a single speaker and the corresponding eigen-blob model localizes the speaker in the image. Experiments conducted on 21 h of real data indicate that the proposed localization approach leads to a relative improvement of 40% over mutual information-based localization and that speaker diarization improves by 16% by incorporating visual information. The proposed approach does not require training and does not rely onapriorihand/face/person detection.

We present a theory for constructing linear subspace approximations to face-recognition algorithms and empirically demonstrate that a surprisingly diverse set of face-recognition approaches can be approximated well by using a linear model. A linear model, built using a training set of face images, is specified in terms of a linear subspace spanned by, possibly nonorthogonal vectors. We divide the linear transformation used to project face images into this linear subspace into two parts: 1) a rigid transformation obtained through principal component analysis, followed by a nonrigid, affine transformation. The construction of the affine subspace involves embedding of a training set of face images constrained by the distances between them, as computed by the face-recognition algorithm being approximated. We accomplish this embedding by iterative majorization, initialized by classical MDS. Any new face image is projected into this embedded space using an affine transformation. We empirically demonstrate the adequacy of the linear model using six different face-recognition algorithms, spanning template-based and feature-based approaches, with a complete separation of the training and test sets. A subset of the face-recognition grand challenge training set is used to model the algorithms and the performance of the proposed modeling scheme is evaluated on the facial recognition technology (FERET) data set. The experimental results show that the average error in modeling for six algorithms is 6.3% at 0.001 false acceptance rate for the FERET fafb probe set which has 1195 subjects, the most among all of the FERET experiments. The built subspace approximation not only matches the recognition rate for the original approach, but the local manifold structure, as measured by the similarity of identity of nearest neighbors, is also modeled well. We found, on average, 87% similarity of the local neighborhood. We also demonstrate the usefulness of the linear model for algorithm-dependent indexing of face databases and find that it results in more than 20 times reduction in face comparisons for Bayesian, elastic bunch graph matching, and one proprietary algorithm.

Regeneration of templates from match scores has security and privacy implications related to any biometric authentication system. We propose a novel paradigm to reconstruct face templates from match scores using a linear approach. It proceeds by first modeling the behavior of the given face recognition algorithm by an affine transformation. The goal of the modeling is to approximate the distances computed by a face recognition algorithm between two faces by distances between points, representing these faces, in an affine space. Given this space, templates from an independent image set (break-in) are matched only once with the enrolled template of the targeted subject and match scores are recorded. These scores are then used to embed the targeted subject in the approximating affine (non-orthogonal) space. Given the coordinates of the targeted subject in the affine space, the original template of the targeted subject is reconstructed using the inverse of the affine transformation. We demonstrate our ideas using three, fundamentally different, face recognition algorithms: Principal Component Analysis (PCA) with Mahalanobis cosine distance measure, Bayesian intra-extrapersonal classifier (BIC), and a feature-based commercial algorithm. To demonstrate the independence of the break-in set with the gallery set, we select face templates from two different databases: Face Recognition Grand Challenge (FRGC) and Facial Recognition Technology (FERET) Database (FERET). With an operational point set at 1 percent False Acceptance Rate (FAR) and 99 percent True Acceptance Rate (TAR) for 1,196 enrollments (FERET gallery), we show that at most 600 attempts (score computations) are required to achieve a 73 percent chance of breaking in as a randomly chosen target subject for the commercial face recognition system. With similar operational set up, we achieve a 72 percent and 100 percent chance of breaking in for the Bayesian and PCA based face recognition systems, respectively. With three different levels of score quantization, we achieve 69 percent, 68 percent and 49 percent probability of break-in, indicating the robustness of our proposed scheme to score quantization. We also show that the proposed reconstruction scheme has 47 percent more probability of breaking in as a randomly chosen target subject for the commercial system as compared to a hill climbing approach with the same number of attempts. Given that the proposed template reconstruction method uses distinct face templates to reconstruct faces, this work exposes a more severe form of vulnerability than a hill climbing kind of attack where incrementally different versions of the same face are used. Also, the ability of the proposed approach to reconstruct actual face templates of the users increases privacy concerns in biometric systems.

A computer vision-based system using images from an airborne aircraft can increase flight safety by aiding the pilot to detect obstacles in the flight path so as to avoid mid-air collisions. Such a system fits naturally with the development of an external vision system proposed by NASA for use in high-speed civil transport aircraft with limited cockpit visibility. The detection techniques should provide high detection probability for obstacles that can vary from subpixels to a few pixels in size, while maintaining a low false alarm probability in the presence of noise and severe background clutter. Furthermore, the detection algorithms must be able to report such obstacles in a timely fashion, imposing severe constraints on their execution time. For this purpose, we have implemented a number of algorithms to detect airborne obstacles using image sequences obtained from a camera mounted on an aircraft. This paper describes the methodology used for characterizing the performance of the dynamic programming obstacle detection algorithm and its special cases. The experimental results were obtained using several types of image sequences, with simulated and real backgrounds. The approximate performance of the algorithm is also theoretically derived using principles of statistical analysis in terms of the signal-to-noise ration (SNR) required for the probabilities of false alarms and misdetections to be lower than prespecified values. The theoretical and experimental performance are compared in terms of the required SNR.

Vehicle text marks are unique features which are useful for identifying vehicles in video surveillance applications. We propose a method for finding such text marks. An existing text detection algorithm is modified such that detection is increased and made more robust to outdoor conditions. False alarm is reduced by introducing a binary image test which remove detections that are not likely to be caused by text. The method is tested on a captured video of a typical street scene.

Network Of Workstations (NOW) platforms put together with off-the-shelf workstations and networking hardware have become a cost effective, scalable, and flexible platform for video processing applications. Still, one has to manually schedule an algorithm to the available processors of the NOW to make efficient use of the resources. However, this approach is time-consuming and impractical for a video processing system that must perform a variety of different algorithms, with new algorithms being constantly developed. Improved support for program development is absolutely necessary before the full benefits of parallel architectures can be realized for video processing applications. Toward this goal, an automatic compile-time scheduler has been developed to schedule input tasks of video processing applications with precedence constraints onto available processors. The scheduler exploits both spatial (parallelism) and temporal (pipelining) concurrency to make the best use of machine resources. Two important scheduling problems are addressed. First, given a task graph and a desired throughput, a schedule is constructed to achieve the desired throughput with the minimum number of processors. Second, given a task graph and a finite set of available resources, a schedule is constructed such that the throughput is maximized while meeting the resource constraints. Results from simulations show that the scheduler and proposed optimization techniques effectively tackle these problems by maximizing processor utilization. A code generator has been developed to generate parallel programs automatically. The tools developed in this paper make it much easier for a programmer to develop video processing applications on these parallel architectures.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

Automatic content-based video indexing is an important research problem. One approach is to extract text appearing in video as an indication of a scene's semantic content. Most work so far has focused only on detecting the spatial extent of text instances in individual video frames. But text occurring in video usually persists for several seconds. This constitutes a text event that should be entered only once in the video index. Therefore it is necessary to determine the temporal extent of text events by combining the results of text detection on individual frames, over time. This is a nontrivial problem because a text event may move, rotate, grow, shrink, or otherwise change throughout its lifetime. Such text effects are common in television programs and commercials to attract viewer attention, but have so far been ignored in the literature. We present a method for detecting and tracking moving, changing caption text events in MPEG-1 compressed video.

Many computes vision applications are computationally challenging especially when they need to meet real-time constraints. A major problem with special purpose systems is that they require the developers of image-processing applications to be aware of the low-level hardware design, making the task cumbersome. To avoid inflexible and expensive hardware designs, another possible alternative is a network of workstations (NOW) platform put together with off-the-shelf workstations and networking hardware. Still, one had to manually schedule an algorithm to the available processors of the NOW to make efficient use of the resources. However, this approach is time consuming and impractical for a vision system that must perform a variety of different algorithms, with new algorithms being constantly developed. Improved support for program development is absolutely necessary before the full benefits of parallel architectures can be realized for vision applications. Towards this goal, an automatic compile-time scheduler has been developed to schedule input tasks of vision applications with precedence constraints onto available processors. The scheduler exploits both spatial (parallelism) and temporal (pipelining) concurrency to make the best use of machine resources. Two important scheduling problems are addressed. First, given a task graph and a desired throughput, a schedule is constructed to achieve the desired throughput with the minimum number of processors. Second, given a task graph and a finite set of available resources, a schedule is constructed such that the throughput is maximized while meeting the resource constraints. Results from simulations show that the scheduler and proposed optimization techniques effectively tackle these problems by maximizing the processor utilization. A code generator has been developed to generate parallel programs automatically. The execution profiles of the resulting parallel programs demonstrate the feasibility of the scheduler. The tools developed in this paper make it much easier for a programmer to develop vision applications on these high-performance platforms.

The National Aeronautics and Space Administration (NASA), along with members of the aircraft industry, recently developed technologies for a new supersonic aircraft. One of the technological areas considered for this aircraft is the use of video cameras and image processing equipment to aid the pilot in detecting other aircraft in the sky. The detection techniques should provide high detection probability for obstacles that can vary from sub-pixel to a few pixels in size, while maintaining a low false alarm probability in the presence of noise and severe background clutter Furthermore, the detection algorithms must be able to report such obstacles in a timely fashion, imposing severe constraints on their execution time. This paper describes approaches to detect airborne obstacles on collision course and crossing trajectories in video images captured from an airborne aircraft. In both cases the approaches consist of an image processing stage to identify possible obstacles followed by a tracking stage to distinguish between true obstacles and image clutter, based on their behavior. The crossing target detection algorithm was also implemented on a pipelined architecture from DataCube and runs in real time. Both algorithms have been successfully tested on flight tests conducted by NASA.

Despite advances in the archiving of digital video, we are still unable to efficiently search and retrieve the portions that interest us. Video indexing by shot segmentation has been a proposed solution and several research efforts are seen in the literature. Shot segmentation alone cannot solve the problem of content based access to video. Recognition of text in video has been proposed as an additional feature. Several research efforts are found in the literature for text extraction from complex images and video with applications for video indexing. We present an update of our system for detection and extraction of an unconstrained variety of text from general purpose video. The text detection results from a variety of methods are fused and each single text instance is segmented to enable it for OCR. Problems in segmenting text from video are similar to those faced in detection and localization phases. Video has low resolution and the text often has poor contrast with a changing background. The proposed system applies a variety of methods and takes advantage of the temporal redundancy in video resulting in good text segmentation.

This paper explores an approach for extracting scene text from a sequence of images with relative motion between the camera and the scene. It is assumed that the scene text lies on planar surfaces, whereas the other features are likely to be at random depths or undergoing independent motion. The motion model parameters of these planar surfaces are estimated using gradient based methods, and multiple motion segmentation. The equations of the planar surfaces, as well as the camera motion parameters are extracted by combining the motion models of multiple planar surfaces. This approach is expected to improve the reliability and robustness of the estimates, which are used to perform perspective correction on the individual surfaces. Perspective correction can lead to improvement in OCR performance. This work can be useful for detecting road signs and bill-boards from a moving vehicle.

In this paper a method is proposed for locating horizontal, uniform-colored text in video frames. It was observed that when a row of pixels across such a text region is clustered in perceptually uniform L*a*b* color space, the pixels of one of these clusters would belong to the text strokes. These pixels would appeal as a line of short streaks on the row since a typical text region has many vertical and diagonal strokes. The proposed method examines every third row of the the image and checks whether this row passes through a horizontal text region. For a given row R, the pixels of R are hierarchically clustered in L*a*b* space and each cluster is tested whether similar-colored pixels in R's vicinity are possibly part of a text region. Candidate text blocks are marked by heuristics using information about the cluster's line of shell streaks. The detected text blocks are fused with the text regions. The method was tested on key frames of several video sequences and was able to locate a wide variety of text.

A number of automated shot-change detection methods for indexing a video sequence to facilitate browsing and retrieval have been proposed. Many of these methods use color histograms or features computed from block motion or compression parameters to compute frame differences. It is important to evaluate and characterize their performance so as to deliver a single set of algorithms that may be used by other researchers for indexing video databases. We present the results of a performance evaluation and characterization of a number of shot-change detection methods that use color histograms, block motion matching, or MPEG compressed data.

Video indexing is an important problem that has occupied recent research efforts. The text appearing in video can provide semantic information about the scene content. Detecting and recognizing text events can provide indices into the video for content based querying. We describe a system for detecting, tracking, and extracting artificial and scene text in MPEG-1 video. Preliminary results are presented.

Temporal segmentation of video is a necessary first step to indexing digital video for browsing and retrieval. A number of different video temporal segmentation algorithms have been published in the literature. There has been little effort to evaluate and characterize their performance so as to deliver a single (or set of) algorithms that may be used by other researchers for indexing video databases. The present results of evaluating a number of these algorithms and characterizing their performance, specifically with respect to robustness to encoder and bitrate changes. The lessons learnt have relevance to algorithm development and evaluation in general.

Like shot changes, the presence of text in digital video is an important event that can be used to index digital video and provide extremely useful semantic information about the scene content. The special characteristics of digital video compared to document images both require and allow new robust approaches to recognition of text in video. We discuss the characteristics and special challenges of text in video and present a strategy of detecting, localizing, and segmenting text from video data for the text indexing problem. Preliminary results from our approach are presented.

Traditional maps contain information about land usage, transportation networks, political boundaries, topographic and hydrographic features, and annotations. The rapid growth in the volume and the complexity of maps has led to the development of computerized geographical and land information systems (GIS and LIS) to facilitate information management. Computerized representation of data contained in maps facilitates efficient searching and updating of information in such systems. Such a representation is also useful for creation of specialized maps to serve a particular need. However the conversion of existing paper-based maps into computer-readable databases is a time consuming and tedious task. This paper addresses one particular aspect of this task, viz., the segmentation of text from other graphical objects in digitized images of color maps.

The paper describes a method for interpretation of lines in distributing frame drawings of a telephone company. The lines are composed of hierarchical components: intersections, segments and line links. First, intersections are located. Segments of lines are then detected. Adjacent segments are finally linked into a line. A post processing step refines the lines. The line extraction and classification are insensitive to touching characters and broken lines. A number of experimental results using real drawings demonstrate good performance of the algorithms.

We propose a method of invoice document structure analysis that provides a means to extract the relevant information from an unknown invoice. Our method uses a combination of textual and graphical processing by analyzing the line and line intersection features in the document as well as searching for possible keywords such as item number, quantity, total, etc. Valid keyword search regions are determined by a specialized connected-component analysis before any OCR is performed. The results of the the keyword search and the line analysis are combined to give the search regions for extracting the relevant data contained in the invoice. This analysis will become part of a larger invoice interpretation system which is currently under development.

The paper proposes a method for obstacle detection on a runway for autonomous navigation and landing of an aircraft. Detection is done in the presence of extraneous features such as tire marks. Suitable features are extracted from the image and warping using approximately known camera and plane parameters is performed in order to compensate ego-motion as far as possible. Residual disparity after warping is estimated using an optical flow algorithm. Features are tracked from frame to frame so as to obtain more reliable estimates of their motion. Corrections are made to motion parameters with the residual disparities using a robust method, and features having large residual disparities are signalled as obstacles. Sensitivity analysis of the procedure is also studied. A Bayesian framework is used at every stage so that the confidence in the estimates can be determined.

In this paper we present the idea of using customizable tools to develop systems to interpret telephone company drawings. These drawings are composed primarily of horizontal and vertical lines. By selecting the intersections as primitives, routines are developed to establish spatial relations of the primitives which are used to describe the structures. Another aspect that as presented is the benefit of using run length encoding to reduce the amount of memory and the processing time needed to process the large files resulting from scanning the forementioned drawings. Examples of the interpretation of telephone company distributing frame and front equipment drawings are presented.

This paper presents the super relation graph (SRG) method for extracting prismatic features from the CAD boundary representation of a machined part. Using the definition of super relations and the validity of a feature volume, this method recognizes features with all three types of interactions: face splitting, face merging, and edge truncation.

THE IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) has become the premier journal in our field due to the vision and leadership provided by past editors- in-chief, King-Sun Fu, Theo Pavlidis, Steve Tanimoto and Anil Jain. Thus, it is with a great sense of honor I have accepted the responsibilities of becoming PAMI's fifth Editorin- Chief. My primary goal is to maintain its excellent reputation and quality while enhancing its value to the subscribers. In this editorial I would like to share some of my plans to achieve this goal.

A system for the interpretation of telephone company drawings is described. Two different types of drawings are interpreted: manhole and table drawings. Each drawing is interpreted by using a different approach to suit its characteristics. Manhole drawings represent the interconnection of telephone lines in the field. The approach used is the vectorization of the line segments to extract continuous lines. The table drawings contain information related to the cables in the manhole. The approach is to locate lines and their intersections to extract the information for entries. An Intelligent Character Recognition (ICR) system is used to interpret text information by using grammars.>

This paper describes a methodology to interpret central office drawings using basic features in order to increase the efficiency of the interpretation. To interpret these drawings, the intersections between vertical and horizontal lines are located and related to obtain the position of boxes. Once the graphical information is interpreted, a grammar driven intelligent character recognition (ICR) system is used to extract text information in order to complete the interpretation of the drawing.

A computer vision system to automatically analyze and assemble an image of the pieces of a jigsaw puzzle is presented. The system, called Automatic Puzzle Solver (APS), derives a new set of features based on the shape and color characteristics of the puzzle pieces. A combination of the shape dependent features and color cues is used to match the puzzle pieces. Matching is performed using a modified iterative labeling procedure in order to reconstruct the original picture represented by the jigsaw puzzle. Algorithms for obtaining shape description and matching are explained with experimental results.

This correspondence presents a system for detecting dimension sets in engineering drawings that are drawn to ANSI drafting standards. A new rule-based text/graphics separation algorithm and a model-based procedure for detecting arrowheads in any orientation have been developed. Arrowhead tracking and search methods are used to extract leaders, tails, and witness lines from segmented images containing only graphics. Text blocks and feature control frames extracted from the segmented images are than associated with their corresponding leaders to obtain complete dimension sets. Experimental results are presented.>

A method is presented for identifying the views in an engineering drawing along with their associated view points in preparation for 3D interpretation of the object shape. A formal procedure is developed for constructing a set of view-based coordinate systems that act as intermediate 2.5D coordinates to relate the 2D drawing-based coordinates to the 3D object-based coordinates. The method can accommodate auxiliary views in addition to the standard orthogonal set, and the number of views and their layout in the drawing need not be known a priori. Moderate errors in line placement and view alignment can also be accommodated.>

The authors present efficient methodologies for interpreting manhole drawings of a telephone company for information extraction. They have developed a skew correction algorithm for deskewing scanned images, a new rule-based text/graphics separation algorithm for separating annotations in the drawings, and a graphics interpretation system which includes model-based arrow head detection, splice detection, and recognition of interconnections of cables and stubs. The final result from the interpretation system is a table which includes an ordered list of all cables and their interconnections which are contained in a manhole. Experimental results are used throughout to describe the algorithms.>

A system for extraction of structural information of a table from its image is discussed. Following the initial binarization and deskewing operations, the image is scanned to extract all horizontal and vertical lines that are present. The table's dimensions are estimated based on these lines. Unlike other systems, the procedure described does not depend on the sole existence of lines to mark the item blocks. White streams are recognized in both the horizontal and vertical direction as substitutes for any missing demarcation lines. A structure interpretation procedure uses the extracted demarcation information to identify each of the item blocks in the table. Subsequently, the interrelations of these item blocks are used to recognize the structure of the tabulated data.>

A system for detecting dimension sets in engineering drawings that are drawn to ANSI drafting standards is presented. A new rule-based text/graphics separation algorithm and a model-based procedure for detecting arrowheads in any orientation have been developed. Arrowhead tracking and search methods are used to extract leaders, tails, and witness lines from segmented images containing only graphics. Text blocks and feature control frames extracted from the segmented images are then associated with their corresponding leaders to obtain complete dimension sets. Object lines are then separated from centerlines and hatching lines. Experimental results are presented.>

A system for reading text data from microfilm images of punched cards is described. The input is a high resolution gray level image obtained by scanning the card image from the microfilm. Noise due to the poor quality of microfilm data and similarity in gray levels of noise patches and punches are the major problems for text extraction. Thresholding, skew correction and morphological operations are performed on the input gray level image. Card parameters such as positions of punches, etc., are calculated and used along with the knowledge about the contents of the card to separate punched holes from other artifacts. Text data are recognized by locating the punched holes and errors are corrected by a context-based approach. The algorithm has been implemented in software and tested on several images.>

The authors investigate a way of improving Hough transform performance using the spatial relations that exist between model features. Clusters of scene features possibly belonging to a model are grouped by a series of one-dimensional Hough transforms. The results presented illustrate the effectiveness of the methods.>

A method developed for classifying each location in a set of magnetic resonance (MR) images by tissue type is described. Three MR images of a region of interest are acquired using spin-echo pulse sequences. The sequences used to acquire these images are specifically defined to allow the calculation of MR-related physical parameters from the image intensity data. After preprocessing operators are applied to the original images, the image intensity data are used to calculate three MR-related parameters of each location. Then, in a supervised training environment, this calculated data set is used with the acquired image data set in a minimum-distance classifier to assign a class-specific color or gray level to each location in the image. Following the classification and formation of the tissue-map image, a set of edge detection routines is applied to generate tissue boundary images for all or a selected-set of tissue types. Experimental results verify that the method is capable of accurately distinguishing between major tissue types in a region of interest.>

A method for interpreting engineering drawings with multiple orthogonal views is presented. The number of views required is not fixed and can range from a minimum of two to a maximum of six using a standard drawing layout. The method call handle input views with small errors or inaccuracies and still produce a valid interpretation. The solution is based on a bottom-up approach in which candidate vertices and edges are used to generate a set of possible faces which are in turn assembled into enclosures representing the final object. All possible interpretations consistent with the input views are found, and inconsistent input views are recognized by the fact that no possible consistent interpretation can be obtained.>

A system for interpretation of images of paper-based line drawings is described. Since a typical drawing contains both text strings and graphics, an algorithm has been developed to locate and separate text strings of various font sizes, styles, and orientations. This is accomplished by applying the Hough transform to the centroids of connected components in the image. The graphics in the segmented image are processed to represent thin entities by their core-lines and thick objects by their boundaries. The core-lines and boundaries are segmented into straight line segments and curved lines. The line segments and their interconnections are analyzed to locate minimum redundancy loops which are adequate to generate a succinct description of the graphics. Such a description includes the location and attributes of simple polygonal shapes, circles, and interconnecting lines, and a description of the spatial relationships and occlusions among them. Hatching and filling patterns are also identified. The performance of the system is evaluated using several test images, and the results are presented. The superiority of these algorithms in generating meaningful interpretations of graphics, compared to conventional data compression schemes, is clear from these results.>

A system for automatic recognition and description of graphical primitives and their interconnections in a mixed text/graphics image is described. The input to the system is a high-resolution binary image obtained by scanning paper-based documents. The Hough transform is applied to the connected components of the image to locate and separate text strings of various font size and orientation. The graphics in the segmented image is processed to represent thin entities by their core lines and thick objects by their boundaries. Lines of various types are identified and segmented into straight line and curved line segments during this process. The line segments and their interconnections are analyzed to locate minimum redundancy loops which are adequate to generate a succinct description of the graphics. Various hatching and filling patterns in the image are identified and described. The algorithm has been implemented as software, and its robustness and performance have evaluated tested using a number of test images. The results for a simple test image are reported.>

The development and implementation of an algorithm for automated text string separation that is relatively independent of changes in text font style and size and of string orientation are described. It is intended for use in an automated system for document analysis. The principal parts of the algorithm are the generation of connected components and the application of the Hough transform in order to group components into logical character strings that can then be separated from the graphics. The algorithm outputs two images, one containing text strings and the other graphics. These images can then be processed by suitable character recognition and graphics recognition systems. The performance of the algorithm, both in terms of its effectiveness and computational efficiency, was evaluated using several test images and showed superior performance compared to other techniques.>

The design of a system to extract information automatically from paper-based maps and answer queries related to spatial features and structure of geographic data is considered. The foundation of such a system is a set of image-analysis algorithms for extracting spatial features. Efficient algorithms to detect symbols, identify and track various types of lines, follow closed contours, compute distances, find shortest paths, etc. from simplified map images have been developed. A query processor analyzes the queries presented by the user in a predefined syntax, controls the operation of the image processing algorithms, and interacts with the user. The query processor is written in Lisp and calls image-analysis routines written in Fortran.>

A number of adaptive estimators to restore images degraded by signal-dependent film grain noise have been derived. These estimators are characterized by their ability to adapt to the nonstationary properties of the image signal distribution. This adaptability results in restored images with better contrast and noise suppression properties when compared with nonadaptive estimators. Lookup tables were extensively used in order to reduce the computation time. In addition to the optimal Bayesian estimators, two suboptimal estimators were presented. Results of computer simulations conducted to evaluate the performances of the estimators using various image quality measures were also presented.