It is disruptive for instrumentalists to turn the page of music sheet when they are playing instruments. The purpose of this study is to investigate how real-time music score alignment can serve as a tool for a computer-assisted page turner. We proposed a simple system which can be set up easily and quickly for use to solve the problem. The framework of the system has two parts: off-line preprocessing stage and online alignment stage. In the first stage, the system extracts chroma feature vectors from the reference recording. In the second stage, the system receives audio signals of live performance and extracts chroma feature vectors from them. Finally, the system uses Dynamic Time Warping (DTW) to find an alignment between those two sets of chroma feature vectors to mark the current measure of the score. The prototype system was evaluated by musicians in different music ensembles like string quartet and orchestra. Most musicians agreed that the system is helpful and can indicate the current measure of a live performance correctly. Some musicians, however, disagreed that the system turned the page at right time. The user survey showed that the best timing for page turning is user-dependent because it is highly to do with musicians' sight reading skills and speed.

This paper presents a system, Web-based Multimedia Annotation (WMA) system, for English as Foreign Language learning in writing skills. The whole correcting process, including the instructor's voice and navigation events (i.e., tele-pointer (cursor), highlight, pen strokes, markings and annotations), can be captured through our system for later access. We address the issues of exploring involved media correlation to benefit adaptable presentation in a synchronization manner from temporal, spatial and content domains. The proposed computed synchronization techniques include speech-event binding process in the temporal domain, tele-pointer movement interpolation and adaptable handwriting presentation in the spatial domain, and visualized annotation erasing in the content domain. The experimental results show that in the speech-event binding process 74% of speech access entries for accessible visualized annotations are found. The acceptable rate of human perception of tele-pointer movement is higher than 85% if time interval is selected carefully. The accuracy of visualized annotation erasing for content removal is about 71%. Our user study shows that students can devote their efforts to writing practice because they can better understand their own mistakes corrected by the instructors using this multimedia presentation.

Nowadays, many applications support the capturing of lecturing information such as speech/video and the writings on the whiteboard to integrate with the presentation slides for later access. This method provides a meaningful way to keep a piece of information presented to audience in a classroom or seminar because they usually have to spend much time on perceiving the complex concepts conveyed by the instructor or speaker. With the successful integration of captured materials and access interfaces, students could review the lectures again and again. In this paper, we keep the core ideas of automated capture for classroom experience and structural access for multimedia documents in mind to develop an application which can be used to record the lecturing process when teaching/learning a language. All recorded materials of a lecture are packaged in the form of Web documents which can be accessed via the Internet (online) or from the local computers (offline).

This article investigates the correlations between multimedia objects (particularly speech and text) involved in language lectures in order to design an effective presentation mechanism for web-based learning. The cross-media correlations are classified into implicit relations (retrieved by computing) and explicit relations (recorded during the preprocessing stage). The implicit temporal correlation between speech and text is primarily to help to negotiate supplementary lecture navigations like tele-pointer movement, lips-sync movement, and content scrolling. We propose a speech-text alignment framework, using an iterative algorithm based on local alignment, to probe many-to-one temporal correlations, and not the one-to-one only. The proposed framework is a more practical method for analyzing general language lectures, and the algorithm's time complexity conforms to the best-possible computation cost, O(nm) , without introducing additional computation. In addition, we have shown the feasibility of creating vivid presentations by exploiting implicit relations and artificially simulating some explicit media. To facilitate the navigation of integrated multimedia documents, we develop several visualization techniques for describing media correlations, including guidelines for speech-text correlations, visible-automatic scrolling, and levels of detail of timeline, to provide intuitive and easy-to-use random access mechanisms. We evaluated the performance of the analysis method and human perceptions of the synchronized presentation. The overall performance of the analysis method is that about 99.5% of the words analyzed are of a temporal error within 0.5 sec and the subjective evaluation result shows that the synchronized presentation is highly acceptable to human beings.

This paper presents a multimedia objects sharing framework on the basis of Internet news infrastructure (NNTP) to facilitate e-learning content access and distribution. Through this framework, a Web-based e-learning system can publish sharable learning objects by posting them to a news server nearby, so that the objects can be automatically spread to world-wide news servers through NNTP forwarding mechanism. Instead of hunting for the desired materials from many different e-learning systems, users can access learning objects shared from different e-learning platforms easily, and in a unified way. Moreover, we deal with severe data redundancy problem occurring especially when the shared object contains multimedia data (e.g., instructor's speech, image, and video). A system prototype has been implemented in our university to share English lectures with others.

In this paper we propose a marking-based synchronized multimedia tutoring system (MSMT) to enhance our previous work, the Web-based synchronized multimedia lecturing system (WSML), for language learning in writing skill. For non-native users, writing is usually a barrier when they are learning a foreign language. Therefore we simulate several symbols which are common use in the traditional paper-based writing correction by state-of-the-art technologies. The system not only provides an easy-to-use interface for instructors to simplify the composition editing tasks, but also provides visualized editing markings for learners to grasp the writing mistakes. A system prototype has been implemented to support ESL composition studies. An overview page that presents the system features with demonstrations can be accessed at http://english-csie.ncnu.edu.tw/OWL/.

This demonstration presents an integrated web-based synchronized scenario for many-to-one cross-media correlations between speech (an EFL, English as Foreign Language, lecture with free-style lecturing behaviors) and the corresponding textual content. The analysis/presentation of the temporal correlations enable the vivid web-based language learning through the interactive functions: browsing speech via content, word-by-word pointer guidance, synchronized scrolling/highlighting, and listening training mode. We regularly analyze and repackage the multimedia content of VoA (Voice of America) [1], ICRT (International Community Radio Taipei) [2], and Online Lectures in our University [3]. Through the subjective experiments, this repackaged synchronized speech/text content does facilitate the learning for EFL learners.

This paper is devoted to explore media correlation and media synchronization in a composite multimedia document, the so-called navigated hypermedia document in our language learning system, to facilitate the multimedia authoring, presentation, and access. Two levels of media correlation in temporal, spatial, and content domains are investigated: syntactic level correlation and semantic level correlation. We devise a capturing mechanism to record all the media streams and relations between them, including voice and event streams, for replaying the lecturing in a form as close as possible to the original classroom experience. The syntactic level correlation is based on specific timestamps of the media stream and used to reconstruct the recorded lecture for synchronized presentation. Furthermore, to integrate media objects with specific segments within the media stream, some computed synchronization processes are required to discover semantic content of the media. The proposed computed synchronization techniques, including speech-event binding process for temporal domain, tele-pointer (i.e. cursor) movement interpolation and adaptable handwriting presentation for spatial domain, and erasing handling for content domain, will be addressed. Experimental results show that in the speech-event binding process 74% of speech access entries for accessible visualized events are found. The acceptable rate of human perception on tele-pointer movement is higher than 85% if time interval is selected carefully. Finally, the accuracy of erasing handling for content removing is about 71%.

Synchronization anomalies, or so-called out-of-sync problems, may occur in a composite multimedia document presentation especially when a presentation tool receives unexpected events that destroy intermedia relations in the document. This work proposes an adaptive synchronization framework (ASF) to solve this problem by adopting a resynchronization procedure. In contrast to the existing approaches focusing on temporal synchronization only, spatial and content relations for a more general presentation are also considered and incorporated into the proposed synchronization core. The experimental results show that the proposed framework can adapt a document to an unsupervised presentation environment and the performance is satisfactory (85%) under some synchronization-killer operations.

The research issues on multiple media correlation have arisen with more and more integrated multimedia applications. The multimedia correlation is used to coordinate different media and facilitate cross-media access. This paper presents our work on two types of multimedia correlation: explicit and implicit relations. We develop a system to carefully capture explicit relations and devise some computed synchronization processes to discover implicit relations between media objects. The proposed computed synchronization techniques, including speech-text alignment process in temporal domain, automatic scrolling process in spatial domain, and content dependency check process in content domain, will be addressed. Experimental results show that in the speech-text alignment process 80% of forced alignment are in-sync even the speech recognition accuracy is as low as 25%. The automatic scrolling process effectively maintains a resynchronization mechanism in different displaying environments.

This demonstration presents a web-based multimedia lecture system that perfectly integrates multimedia lecturing with static HTML pages and enables their presentation in synchronization. We develop key techniques to reproduce vivid web-teaching scenarios for on-demand access, including events capturing scheme and synchronized mechanism. To facilitate multimedia lectures creation, the easy-to-use authoring and managing tools specially designed for teachers have been also included in the development. In addition, integrated presentation and efficient access issues are also discussed for the student's aspect.

In this article, we have introduced a “Live Synchronized Hypermedia Live Lecture (SHLL) System” using RTP to synchronize the live presentation of streaming video lecture, HTML-based lecture notes, and HTML page Navigation Events. The SHLL framework consists of three major modules: (1) SHLL Recorder- for recording the temporal information of the AV lecture and the HTML-based lecture notes navigation processes. (2) SHLL Event Server- for receiving, depositing, and multicasting SHLL events. (3) SHLL Browser- for presentation of the synchronized AV lecture and HTML-based lecture notes navigation.

Synchronization among various media sources is one of the most important issues in multimedia communications and various audio/video (A/V) applications. For continuous playback (such as lip synchronization) under a time-sharing multiprocessing operating system (such as UNIX), the synchronization quality of traditional synchronization mechanisms employed on single processes may vary according to the workload of the system. When the system encounters an overload situation, the synchronization usually fails and, even worse, results in two fatal defects in human perception: the audio discontinuity (audio break) and the out-of-synchronization (synchronization anomaly). In order to overcome these problems, a novel media synchronization model employed on multiple processes (or threads) in a multiprocessing environment is proposed. The problem of asynchronism due to system overload is solved by assigning a higher priority to more important media and adopting a delay-or-drop policy to treat the lower priority ones. Some experimental results are presented to show the effectiveness of the proposed model and the implementation mechanisms under a UNIX, X-Windows environment. On the basis of the proposed model, a continuous media playback (CMP) module, which acted as the key component of some popular multimedia systems such as multimedia authoring system, multimedia E-mail system, multimedia bulletin board system (BBS), and video-on-demand (VoD) System, was implemented.