We describe TOBY Playpad, an early intervention program for children with Autism Spectrum Disorder (ASD). TOBY teaches the teacher -- the parent -- during the crucial period following diagnosis, which often coincides with no access to formal therapy. We reflect on TOBY's evolution from table-top aid for flashcards to an iPad app covering a syllabus of 326 activities across 51 skills known to be deficient for ASD children, such imitation, joint attention and language. The design challenges unique to TOBY are the need to adapt to marked differences in each child's skills and rate of development (a trait of ASD) and teach parents unfamiliar concepts core to behavioural therapy, such as reinforcement, prompting, and fading. We report on three trials that successively decrease oversight and increase parental autonomy, and demonstrate clear evidence of learning. TOBY's uniquely intertwined Natural Environment Tasks are found to be effective for children and popular with parents.

The scale and dynamicity of social media, and interaction between traditional news sources and online communities, has created challenges to information retrieval approaches. Users may have no clear information need or be unable to express it in the appropriate idiom, requiring instead to be oriented in an unfamiliar domain, to explore and learn. We present a novel data-driven visualization, termed Eventscape, that combines time, visual media, mood, and controversy. Formative evaluation highlights the value of emotive facets for rapid evaluation of mixed news and social media topics, and a role for such visualizations as pre-cursors to deeper search.

We demonstrate an open multimedia-based system for delivering early intervention therapy for autism. Using flexible multi-touch interfaces together with principled ways to access rich content and tasks, we show how a syllabus can be translated into stimulus sets for early intervention. Media stimuli are able to be presented agnostic to language and media modality due to a semantic network of concepts and relations that are fundamental to language and cognitive development, which enable stimulus complexity to be adjusted to child performance. Being open, the system is able to assemble enough media stimuli to avoid children over-learning, and is able to be customised to a specific child which aids with engagement. Computer-based delivery enables automation of session logging and reporting, a fundamental and time-consuming part of therapy.

Although tagging has become increasingly popular in online image and video sharing systems, tags are known to be noisy, ambiguous, incomplete and subjective. These factors can seriously affect the precision of a social tag-based web retrieval system. Therefore improving the precision performance of these social tag-based web retrieval systems has become an increasingly important research topic. To this end, we propose a shared subspace learning framework to leverage a secondary source to improve retrieval performance from a primary dataset. This is achieved by learning a shared subspace between the two sources under a joint Nonnegative Matrix Factorization in which the level of subspace sharing can be explicitly controlled. We derive an efficient algorithm for learning the factorization, analyze its complexity, and provide proof of convergence. We validate the framework on image and video retrieval tasks in which tags from the LabelMe dataset are used to improve image retrieval performance from a Flickr dataset and video retrieval performance from a YouTube dataset. This has implications for how to exploit and transfer knowledge from readily available auxiliary tagging resources to improve another social web retrieval system. Our shared subspace learning framework is applicable to a range of problems where one needs to exploit the strengths existing among multiple and heterogeneous datasets.

The blogosphere has grown to be a mainstream forum of social interaction as well as a commercially attractive source of information and influence. Tools are needed to better understand how communities that adhere to individual blogs are constituted in order to facilitate new personal, socially-focused browsing paradigms, and understand how blog content is consumed, which is of interest to blog authors, big media, and search. We present a novel approach to blog subcommunity characterization by modeling individual blog readers using mixtures of an extension to the LDA family that jointly models phrases and time, Ngram Topic over Time (NTOT), and cluster with a number of similarity measures using Affinity Propagation. We experiment with two datasets: a small set of blogs whose authors provide feedback, and a set of popular, highly commented blogs, which provide indicators of algorithm scalability and interpretability without prior knowledge of a given blog. The results offer useful insight to the blog authors about their commenting community, and are observed to offer an integrated perspective on the topics of discussion and members engaged in those discussions for unfamiliar blogs. Our approach also holds promise as a component of solutions to related problems, such as online entity resolution and role discovery.

The amount of multimedia content available online constantly increases, and this leads to problems for users who search for content or similar communities. Users in Flickr often self-organize in user communities through Flickr Groups. These groups are particularly interesting as they are a natural instantiation of the content~+~relations social media paradigm. We propose a novel approach to group searching through hypergroup discovery. Starting from roughly 11,000 Flickr groups' content and membership information, we create three different bag-of-word representations for groups, on which we learn probabilistic topic models. Finally, we cast the hypergroup discovery as a clustering problem that is solved via probabilistic affinity propagation. We show that hypergroups so found are generally consistent and can be described through topic-based and similarity-based measures. Our proposed solution could be relatively easily implemented as an application to enrich Flickr's traditional group search.

This paper examines the recovery of user context in indoor environmnents with existing wireless infrastructures to enable assistive systems. We present a novel approach to the extraction of user context, casting the problem of context recovery as an unsupervised, clustering problem. A well known density-based clustering technique, DBSCAN, is adapted to recover user context that includes user motion state, and significant places the user visits from WiFi observations consisting of access point ID and signal strength. Furthermore, user rhythms or sequences of places the user visits periodically are derived from the above low level contexts by employing a state-of-the-art probabilistic clustering technique, the Latent Dirichlet Allocation (LDA), to enable a variety of application services. Experimental results with real data are presented to validate the proposed unsupervised learning approach and demonstrate its applicability.

We present a video browsing approach, termed Temporal Semantic Compression (TSC), that uses automated measures of interest to support today's foraging behaviours. Conventional browsers 'compress' a video stream using simple 2x or 8x fast-forward. TSC browsers dynamically filter video based on a single user gesture to leave out more or less of the boring bits. We demonstrate a browser with an example interest measure, derived from an automated estimate of movie tempo, to forage in terms of narrative structures such as crises, climaxes, and action sequence book-ends. Media understanding algorithms facilitate browsing, and interactivity enables the human-in-the-loop to cope when those algorithms fail to cross the semantic gap.

Recent growth in broadband access and proliferation of small personal devices that capture images and videos has led to explosive growth of multimedia content available everywhere - from personal disks to the Web. While digital media capture and upload has become nearly universal with newer device technology, there is still a need for better tools and technologies to search large collections of multimedia data and to find and deliver the right content to a user according to her current needs and preferences. A renewed focus on the subjective dimension in the multimedia lifecycle, from creation, distribution, to delivery and consumption, is required to address this need beyond what is feasible today. Integration of the subjective aspects of the media itself - its affective, perceptual, and physiological potential (both intended and achieved), together with those of the users themselves will allow for personalizing the content access, beyond today's facility. This integration, transforming the traditional multimedia information retrieval (MIR) indexes to more effectively answer specific user needs, will allow a richer degree of personalization predicated on user intention and mode of interaction, relationship to the producer, content of the media, and their history and lifestyle. In this paper, we identify the challenges in achieving this integration, current approaches to interpreting content creation processes, to user modelling and profiling, and to personalized content selection, and we detail future directions. The structure of the paper is as follows: In Section I, we introduce the problem and present some definitions. In Section II, we present a review of the aspects of personalized content and current approaches for the same. Section III discusses the problem of obtaining metadata that is required for personalized media creation and present eMediate as a case study of an integrated media capture environment. Section IV presents the MAGIC system as a case study of capturing effective descriptive data and putting users first in distributed learning delivery. The aspects of modelling the user are presented as a case study in using user's personality as a way to personalize summaries in Section V. Finally, Section VI concludes the paper with a discussion on the emerging challenges and the open problems.

We present online algorithms to extract social context: Social spheres are labeled locations of significance, represented as convex hulls extracted from GPS traces. Colocation is determined from Bluetooth and GPS to extract social rhythms, patterns in time, duration, place, and people corresponding to real-world activities. Social ties are formulated from proximity and shared spheres and rhythms. Quantitative evaluation is performed for 10+ million samples over 45 man-months. Applications are presented with assessment of perceived utility: Socio-Graph , a video and photo browser with filters for social metadata, and Jive , a blog browser that uses rhythms to discover similarity between entries automatically.

This paper describes a novel suggestive interface embedded in a smart camera prototype aimed at aiding home movie makers. We focus on the problem of generating shot capture suggestions suitable to the user filming context, intended audience and style, and formulate a novel aesthetic measure by which to judge proposed suggestions. Tight coupling between media and software allows the aesthetic measure to be sensitive to previous footage captures, including those taken without the system prompting, in a manner allowing flexible, end-to-end migration of the authoring task from user to machine. An approximate method is used to find timely, near-optimal solutions to the aesthetic measure. Qualitative evaluation in the form of a user study shows it to be a promising approach to the flexible home movie authoring context

This paper presents novel techniques for using panoramas as spatial context to enhance browsing of personal media archives. This context, scenes where frequent media capture takes place, is present in the disparate photos and videos, but not leveraged by traditional browsing techniques (e.g. thumbnails or zoomable interfaces). Coarse geo-position is often an insufficient index at such media capture hotspots. We experiment with panoramic video, which presents archive video organically blended with panoramas of media capture hotspots; Immersive browsing and filtering with media items projected onto spherical panoramas; and Detection and representation of links between panoramas to enable browsing of situated media in quasi-3D. We present proof-of-concept implementations and observations of their effectiveness, limitations, and open problems. Experiments confirm the intuition that each holds promise for augmenting traditional browsing environments.

Personal media collections are often viewed and managed along the social dimension, the places we spend time at and the people we see, thus tools for extracting and using this information are required. We present novel algorithms for identifying socially significant places termed social spheres unobtrusively from GPS traces of daily life, and label them as one of Home, Work, or Other, with quantitative evaluation of 9 months taken from 5 users. We extract locational co-presence of these users and formulate a novel measure of social tie strength based on frequency of interaction, and the nature of spheres it occurs within. Comparative user studies of a multimedia browser designed to demonstrate the utility of social metadata indicate the usefulness of a simple interface allowing navigation and filtering in these terms. We note the application of social context is potentially much broader than personal media management, including context-aware device behaviour, life logs, social networks, and location-aware information services.

This paper details the design of an algorithm for automatically manipulating the important aesthetic element of video, visual tempo. Automatic injection, detection and repair of such aesthetic elements, it is argued, is vital to the next generation of amateur multimedia authoring tools. We evaluate the performance of the algorithm on a battery of synthetic data and demonstrate its ability to return the visual tempo of the final media a considerable degree closer to the target signal. The novelty of this work lies chiefly in the systematic manipulation of this high level aesthetic element of video.

This paper presents novel additions to our existing amateur media creation framework. The framework provides at-capture guidance to enable the home movie maker to realize their aesthetic and narrative goals and automation of post-production editing. A common problem with the amateur filming context is its contingent nature, which often results in the failure to gain footage vital to the user's goals, even with at-capture software embedding. Accordingly, we have modelled minimizing the difference between target and captured footage at a given time during filming as a probability distribution divergence problem. We apply two policies of feedback to the user on thier performance, passive communication via a suggestion desirability measure, and active filtering of undesirable suggestions. We demonstrate the framework using each policy with a simulation of various user and filming situations with promising results.

This paper describes a novel interactive media authoring framework, MediaTE, that enables amateurs to create videos of higher narrative or aesthetic quality with a completely mobile lifecycle. A novel event bootstrapping dialog is used to derive shot suggestions that yield both targetted footage and annotation enabling an automatic Computational Media Aesthetics-aware editing phase, the manual performance of which is typically a barrier to the amateur. This facilitates a move away from requiring a prior-conception of the events or locale being filmed, in the form of a template, to at-capture bootstrapping of this information. Metadata gathered as part of the critical path of media creation also has implications for the longevity and reuse of captured media assets. Results of an evaluation performed on both the usability and delivered media aspects of the system are discussed, which highlight the tenability of the proposed framework and the quality of the produced media.

We discuss the design goals for an integrated media creation environment (IMCE) aimed at enabling the average user to create media artifacts with professional qualities. The resulting requirements are implemented and we demonstrate the efficacy of the resulting system with the generation of two simple home movies. The significance for the average user seeking to create home movies lies in the flexible and automatic application of film principles to the task, removal of tedious low-level editing by means of well-formed media transformations in terms of high-level film constructs (e.g., tempo), and content repurposing powered by those same transformations added to the rich semantic information maintained at each phase of the process.

We discuss the design and implementation of an integrated media creation environment, and demonstrate its efficacy in the generation of two simple home movies. The significance for the average user seeking to create home movies lies in the flexible and automatic application of film principles to the task, removal of tedious low-level editing by means of well-formed media transformations in terms of high-level film constructs (e.g. tempo), and content repurposing powered by those same transformations added to the rich semantic information maintained at each phase of the process.

Director in your pocket

In this paper, we present an application designed to improve the quality of amateur video production. The majority of home movie material is negatively impacted by two factors: lack of narrative content - "what to shoot?", and the absence or inappropriate use of cinesthestic elements for effective reinforcement of content - "how to shoot?" We leverage the age-old communicative powers of story to answer the what. For the second problem, the how, we turn to the corpus of aesthetic principles that constitute the film profession, which impact both technical and cinematic considerations for a given project.

This work constitutes the first attempt to extract an important narrative structure, the 3-Act story telling paradigm, in film. This narrative structure is prevalent in the domain of film as it forms the foundation and framework in which the film can be made to function as an effective tool for story telling, and its extraction is a vital step in automatic content management for film data. A novel act boundary likelihood function for Act 1 is derived using a Bayesian formulation under guidance from film grammar, tested under many configurations and the results are reported for experiments involving 25 full length movies. The formulation is shown to be a useful tool in both the automatic and semi-interactive setting for semantic analysis of film.

In this paper we describe research aimed at enabling amateur video makers to improve both the technical quality and communicative capacity of their work. Motivated by the recognition that untold hours of home video are simply abandoned after capture, we have formulated the problem as one of defining the what and how of footage capture.We have implemented a framework that answers the first problem, the what, by means of the age-old communicative powers of Story; the second problem, the how, is addressed by means of well documented aesthetic principles that constitute the film profession, which impact both technical and cinematic considerations for a given project.We provide a brief overview of the process, beginning with the narrative template, embodying a chosen story, through the principal phases of generating a storyboard, directing, and editing, resulting in the finished product. We demonstrate the interplay of narrative, purpose for the production, and aesthetic agents, and their influence on the automatically generated storyboard with examples.

This paper examines film rhythm, an important expressive element in motion pictures, based on our ongoing study to exploit film grammar as a broad computational framework for the task of automated film and video understanding.Of the many, more or less elusive, narrative devices contributing to film rhythm, this paper discusses motion characteristics that form the basis of our analysis, and presents novel computational models for extracting rhythmic patterns induced through a perception of motion.In our rhythm model, motion behaviour is classified as being either nonexistent, fluid or staccato for a given shot.Shot neighbourhoods in movies are then grouped by proportional makeup of these motion behavioural classes to yield seven high-level rhythmic arrangements that prove to be adept at indicating likely scene content (e.g.dialogue or chase sequence) in our experiments.Underlying causes for this level of codification in our approach are postulated from film grammar, and are accompanied by detailed demonstration from real movies for the purposes of clarification.

This paper presents an original computational approach for the extraction of movie tempo for deriving story sections and events that convey high level semantics of stories portrayed in motion pictures, thus enabling better video annotation and interpretation systems. This approach, inspired by the existing cinematic conventions known as film grammar, uses the attributes of motion and shot length to define and compute a novel continuous measure of the tempo of a movie. Tempo flow plots are derived for several full-length motion pictures and edge detection is performed to extract dramatic story sections and events occurring in the movie, underlined by their unique tempo. The results confirm the reliable detection of actual distinct tempo changes and serve as a useful index for the dramatic development and narration of the story in motion pictures.

This work seeks to lay the framework of film grammar over the video to be analysed. We use the shot attributes of motion and shot length to produce a novel continuous measure of one of the aesthetic elements of films, namely the movie tempo. We refer to our previous work detailing the study of this construct and its automatic derivation, and also demonstrating its usefulness as an expressive element and as a sound basis for higher semantic descriptions such as dramatic events and story elements. Initial assessment of tempo was performed in our study on the basis that the relative importance of both shot length and motion in formulating the tempo function was the same. In this paper, we analyze their relative contributions to tempo, and demonstrate how these two factors can be manipulated to influence audience perception of movie time.