Technologies that track a specific person as they traverse a network of surveillance cameras can be used as the basis for a multitude of video surveillance applications including mass transit monitoring, large venue security, building security, and the like. In order to continue supporting the development robust people tracking technologies, the first AVSS Multiple Camera Person Tracking (MCPT) Challenge Evaluation was established to provide data and evaluation resources for researchers to build Single Person Tracking (SPT) technologies. This special session will focus on the AVSS MCPT Challenge Evaluation which will include a description of the evaluation task, the i-LIDS Multiple-camera tracking scenario data set used for the evaluation, and presentations by the challenge evaluation participants describing their systems.

This paper is a summary of the 2008 TRECVid Event Detection evaluation track. TRECVid is a laboratory-style evaluation that aims to model real world situations or significant component tasks. The event detection evaluation was organized to address detection of a set of specific events that would be of potential interest to an operator in the surveillance domain. This paper describes the video data, evaluation tasks, evaluation metrics, and results of the event detection evaluation.

Common benchmark data sets, standardized performance metrics, and baseline algorithms have demonstrated considerable impact on research and development in a variety of application domains. These resources provide both consumers and developers of technology with a common framework to objectively compare the performance of different algorithms and algorithmic improvements. In this paper, we present such a framework for evaluating object detection and tracking in video: specifically for face, text, and vehicle objects. This framework includes the source video data, ground-truth annotations (along with guidelines for annotation), performance metrics, evaluation protocols, and tools including scoring software and baseline algorithms. For each detection and tracking task and supported domain, we developed a 50-clip training set and a 50-clip test set. Each data clip is approximately 2.5 minutes long and has been completely spatially/temporally annotated at the I-frame level. Each task/domain, therefore, has an associated annotated corpus of approximately 450,000 frames. The scope of such annotation is unprecedented and was designed to begin to support the necessary quantities of data for robust machine learning approaches, as well as a statistically significant comparison of the performance of algorithms. The goal of this work was to systematically address the challenges of object detection and tracking through a common evaluation framework that permits a meaningful objective comparison of techniques, provides the research community with sufficient data for the exploration of automatic modeling techniques, encourages the incorporation of objective evaluation into the development process, and contributes useful lasting resources of a scale and magnitude that will prove to be extremely useful to the computer vision research community for years to come.

NIST has been conducting a series of evaluations in the automatic analysis of information in video since 2001. These began within the NIST text retrieval evaluation (TREC) as a pilot track in searching for information in large collections of video. The evaluation series was spun off into its own evaluation/workshop series called TRECVID. TRECVID continues to examine the challenge of extracting features for search technologies. In 2004, NIST also began an evaluation series dedicated to assessing video object detection and tracking technologies using training and test sets that were significantly larger than those used in the past -facilitating novel machine learning approaches and supporting statistically-informative evaluation results. Eventually this effort was merged with other video processing evaluations being implemented in Europe under the classification of events, activities, and relationships (CLEAR) consortium. NIST's goal is to evolve these evaluations of video processing technologies towards a focus on the detection of visually observable events and 3D modeling and to help the computer vision community make strides in the areas of accuracy, robustness, and efficiency.

Pervasive computing devices, sensors, and networks, provide infrastructure for context aware smart meeting rooms that sense ongoing human activities and respond to them. This requires advances in areas including networking, distributed computing, sensor data acquisition, signal processing, speech recognition, human identification, and natural language processing. Open interoperability and metrology standards for the sensor and recognition technologies can aid R&D programs in making these advances. The NIST (National Institute of Standards and Technology) Smart Space and Meeting Room projects are developing tools for data formats, transport, distributed processing, and metadata. We are using them to create annotated multi modal research corpora and measurement algorithms for smart meeting rooms, which we are making available to the research and development community.

During 2000 and 2001 two large data collections were performed, with paid users. We analyze the effects of speech recognition accuracy, as measured by Word Error Rate (WER), on other metrics. Analysis shows a linear correlation between WER and the Task Completion metrics, and (unexpectedly) this relationship remains more or less linear even for quite high values of WER. The picture for User Satisfaction metrics is more complex, and a linear model derived from the data by using the PARADISE framework [1] is given by Walker et al. [2]. We present evidence suggesting a somewhat linear relationship between WER and User Satisfaction for WER less than 35% or 40% in 2001, compared to stronger correlations in 2000. Finally, we note that the size of effect of increasing WER on Task Completion (slope of the least-squares regression line) appears to be about half as large in 2001 as in 2000, which we attribute to improved strategies for accomplishing tasks despite speech recognition errors. We consider this to be an important accomplishment of the research groups who built the Communicator implementations.

This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

This paper describes the evaluation methodology and results of the 2001 DARPA Communicator evaluation. The experiment spanned 6 months of 2001 and involved eight DARPA Communicator systems in the travel planning domain. It resulted in a corpus of 1242 dialogs which include many more dialogues for complex tasks than the 2000 evaluation. We describe the experimental design, the approach to data collection, and the results. We compare the results by the type of travel plan and by system. The results demonstrate some large differences across sites and show that the complex trips are clearly more difficult.

This paper describes results of an experiment with 9 different DARPA Communicator Systems who participated in the June 2000 data collection. All systems supported travel planning and utilized some form of mixed-initiative interaction. However they varied in several critical dimensions: (1) They targeted different back-end databases for travel information; (2) The used different modules for ASR,NLU,TTS and dialog management. We describe the experimental design, the approach to data collection, the metrics collected, and results comparing the systems. 1.

This paper presents a summary of the 1998 Topic Detection and Tracking (TDT) tasks and the results of the 1998 TDT evaluation. The purpose of TDT is to develop technologies for retrieval and automatic organization of Broadcast News and Newswire stories and to evaluate the performance of those technologies. The TDT project builds on and extends the technologies of Automatic Speech Recognition and Document Retrieval with three tasks: 1) Story Segmentation, 2) Topic Detection and 3) Topic Tracking. Each of the tasks simulates a hypothetical operational system that requires incoming data to be processed time synchronously. The 1998 TDT evaluation (TDT2) continues the work of the TDT pilot study conducted in 1997 (TDT1) and is the first open evaluation of TDT tasks.

This paper reports on the use of materials derived from radio For the 1996 tests, a detailed annotation convention was and television news broadcasts for research and testing implemented by the LDC to capture these effects. This purposes for large vocabulary Continuous Speech Recogniconvention permitted the community to implement a tion (CSR) technology. Tests using these materials have been particular form of partitioned evaluation, and for which implemented by NIST on behalf of the DARPA-funded NIST’s analyses of results could be partitioned into speech recognition research community in 1995 and 1996, corresponding categories. and are expected to continue for the next several years. Four research groups participated in the 1995 tests, and nine For all systems, perhaps most striking feature of these tests groups (at eight sites) participated in the 1996 tests. This is that from segment to segment the word error rates paper documents properties of the training and test materials, sometimes vary over a wide range. The challenge that is describes a detailed annotation and transcription protocol that presented by the broadcast materials is significant. In the has been used for more than 100 hours of recorded data that 1996 tests, involving both radio and TV broadcast news has been made available through the Linguistic Data Consormaterials, the system with the lowest measured error rate had tium (LDC), and discusses test protocols and results of both an overall word error rate of 27.1%. the 1995 and 1996 Benchmark Tests.

The performance of large vocabulary speech recognition systems is currently thought to be limited by the size of the corpus used to train the recognition system. Hence several very large speech corpora have been created recently and many more are planned. A significant problem in the generation of these corpora is the definition of their format to minimize distribution costs and maximize ease of use. This paper describes the development of a "standard" lossless compressed waveform file format which minimizes the media required for corpora distribution while maximizing accessibility. This paper contains two primary contributions: 1) The use of a "standard" file format for speech corpora which supports embedded compression and the development of a software interface toolkit which supports automatic waveform compression/decompression; 2) The use of lossless data compression for speech corpora. This task differs from mainstream speech coding in that the compression must be fast and lossless. Fast approximations to the standard techniques of linear prediction and residual coding have been developed and are employed.>