The recent growth in the consumption of online media by children during early childhood necessitates data-driven tools enabling educators to filter out appropriate educational content for young learners. This paper presents an approach for detecting educational content in online videos. We focus on two widely used educational content classes: literacy and math. For each class, we choose prominent codes (sub-classes) based on the Common Core Standards. For example, literacy codes include ‘letter names’, ‘letter sounds’, and math codes include ‘counting’, ‘sorting’. We pose this as a finegrained multilabel classification problem as videos can contain multiple types of educational content and the content classes can get visually similar (e.g., ‘letter names’vs ‘letter sounds’). We propose a novel class prototypes based supervised contrastive learning approach that can handle fine-grained samples associated with multiple labels. We learn a class prototype for each class and a loss function is employed to minimize the distances between a class prototype and the samples from the class. Similarly, distances between a class prototype and the samples from other classes are maximized. As the alignment between visual and audio cues are crucial for effective comprehension, we consider a multimodal transformer network to capture the interaction between visual and audio cues in videos while learning the embedding for videos. For evaluation, we present a dataset, APPROVE, employing educational videos from YouTube labeled with fine-grained education classes by education researchers. APPROVE consists of 193 hours of expert-annotated videos with 19 classes. The proposed approach outperforms strong baselines on APPROVE and other benchmarks such as Youtube-8M, and COIN. The dataset is available at https://nusci.csl.sri.com/project/APPROVE.

We present a Human Body model based IDentification system (HMID) system that is jointly trained for shape, pose and biometric identification. HMID is based on the Human Mesh Recovery (HMR) network and we propose additional losses to improve and stabilize shape estimation and biometric identification while maintaining the pose and shape output. We show that when our HMID network is trained using additional shape and pose losses, it shows a significant improvement in biometric identification performance when compared to an identical model that does not use such losses. The HMID model uses raw images instead of silhouettes and is able to perform robust recognition on images collected at range and altitude as many anthropometric properties are reasonably invariant to clothing, view and range. We show results on the USF dataset as well as the BRIAR dataset which includes probes with both clothing and view changes. Our approach (using body model losses) shows a significant improvement in Rank20 accuracy and True Accuracy Rate on the BRIAR evaluation dataset.

We propose a method to generate optimal natural language for block placement directives generated by a machine’s planner during human-agent interactions in the blocks world. A non user-friendly machine directive, e.g., move(ObjId, toPos), is transformed into visually and contextually grounded referring expressions that are much easier for the user to comprehend. We describe an algorithm that progressively and generatively transforms the machine’s directive in ECI (Elementary Composable Ideas)-space, generating many alternative versions of the directive. We then define a cost function to evaluate the ease of comprehension of these alternatives and select the best option. The parameters for this cost function were derived empirically from a user study that measured utterance-to-action timings.

We conducted a small scale user study to understand user experiences with two different forms of dialogue interfaces - text-based and voice-based - to interact with a virtual agent in a Blocks World environment while perform tower building tasks. The participants also had the option of using deictic gestures in addition to the speech modality. We identify common types of errors/issues that led to communication failures and share our observations about how users reacted to these issues. We also present survey data that reflects users' evaluations of the dialogue interfaces and their interactions with the virtual agent.

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12-18% over prior methods in high-density videos.

We present an adaptive data-driven algorithm for interactive crowd simulation. Our approach combines realistic trajectory behaviors extracted from videos with synthetic multi-agent algorithms to generate plausible simulations. We use statistical techniques to compute the movement patterns and motion dynamics from noisy 2D trajectories extracted from crowd videos. These learned pedestrian dynamic characteristics are used to generate collision-free trajectories of virtual pedestrians in slightly different environments or situations. The overall approach is robust and can generate perceptually realistic crowd movements at interactive rates in dynamic environments. We also present results from preliminary user studies that evaluate the trajectory behaviors generated by our algorithm.

We present an adaptive data-driven algorithm for interactive crowd simulation. Our approach combines realistic trajectory behaviors extracted from videos with synthetic multi-agent algorithms to generate plausible simulations. We use statistical techniques to compute the movement patterns and motion dynamics from noisy 2D trajectories extracted from crowd videos. These learned pedestrian dynamic characteristics are used to generate collision-free trajectories of virtual pedestrians in slightly different environments or situations. The overall approach is robust and can generate perceptually realistic crowd movements at interactive rates in dynamic environments. We also present results from preliminary user studies that evaluate the trajectory behaviors generated by our algorithm.

We present an interactive approach for analyzing crowd videos and generating content for multimedia applications. Our formulation combines online tracking algorithms from computer vision, non-linear pedestrian motion models from computer graphics, and machine learning techniques to automatically compute the trajectory-level pedestrian behaviors for each agent in the video. These learned behaviors are used to detect anomalous behaviors, perform crowd replication, augment crowd videos with virtual agents, and segment the motion of pedestrians. We demonstrate the performance of these tasks using indoor and outdoor crowd video benchmarks consisting of tens of human agents, moreover, our algorithm takes less than a tenth of a second per frame on a multi-core PC. The overall approach can handle dense and heterogeneous crowd behaviors and is useful for realtime crowd scene analysis applications.

AsthmaGuide is a smartphone and cloud based asthma system in which a smart phone is used as a hub for collecting a comprehensive collection of information. The data, including data over time, is then displayed in a cloud web application for both patients and healthcare providers to view. AsthmaGuide also provides an advice and alarm infrastructure based on the collected data and parameters set by healthcare providers. With these components, AsthmaGuide provides a comprehensive ecosystem that allows patients to be involved in their own health and also allows doctors to provide more effective day to day care. Using real asthma patient wheezing sounds we develop a new combination of classifiers that is 96% accurate at automatically detecting wheezing. This abstract provides an overview of the design and implementation of AsthmaGuide and provides empirical evidence that AsthmaGuide is 3% - 11% more accurate in detecting wheezing sounds than standard techniques.

Understanding and modeling how a crowd behaves in a wide variety of situations is an important problem in many areas. For example, during the planning stages, city, traffic, and evacuation engineers use crowd behavior modeling to predict usage patterns and to do safety analysis of their designs. Several research areas benefit from realistic simulation of crowds such as augmented reality, animation, games, virtual therapy, and virtual training. Not only is a realistic rendering of a virtual environment required for these applications, but also a realistic simulation of virtual humans is essential to providing an immersive experience for the users.

We present an approach to model dynamic, data-driven source and listener directivity for interactive wave-based sound propagation in virtual environments and computer games. Our directional source representation is expressed as a linear combination of elementary spherical harmonic (SH) sources. In the preprocessing stage, we precompute and encode the propagated sound fields due to each SH source. At runtime, we perform the SH decomposition of the varying source directivity interactively and compute the total sound field at the listener position as a weighted sum of precomputed SH sound fields. We propose a novel plane-wave decomposition approach based on higher-order derivatives of the sound field that enables dynamic HRTF-based listener directivity at runtime. We provide a generic framework to incorporate our source and listener directivity in any offline or online frequency-domain wave-based sound propagation algorithm. We have integrated our sound propagation system in Valve's Source game engine and use it to demonstrate realistic acoustic effects such as sound amplification, diffraction low-passing, scattering, localization, externalization, and spatial sound, generated by wave-based propagation of directional sources and listener in complex scenarios. We also present results from our preliminary user study.

We propose a new technique to simulate dynamic patterns of crowd behaviors using stress modeling. Our model accounts for permanent, stable disposition and the dynamic nature of human behaviors that change in response to the situation. The resulting approach accounts for changes in behavior in response to external stressors based on well-known theories in psychology. We combine this model with recent techniques on personality modeling for multi-agent simulations to capture a wide variety of behavioral changes and stressors. The overall formulation allows different stressors, expressed as functions of space and time, including time pressure, positional stressors, area stressors and inter-personal stressors. This model can be used to simulate dynamic crowd behaviors at interactive rates, including walking at variable speeds, breaking lane-formation over time, and cutting through a normal flow. We also perform qualitative and quantitative comparisons between our simulation results and real-world observations.

We introduce a method to pack Ptex per-face texture data that is both space-efficient and hardware-friendly. Recently presented real-time implementations of Ptex have been wasteful with space and required a storage cost many times higher than the size of the original texture data. Our method packs multiple levels of Ptex data together, and requires only around 8% increase in storage for our test textures. Additionally, because of efficient data packing, our method wastes less space than a typical texture atlas, which requires buffer regions to be added between the separate charts within the texture.

Abstract We propose a method for view‐dependent simplification of articulated‐body dynamics, which enables an automatic trade‐off between visual precision and computational efficiency. We begin by discussing the problem of simplifying the simulation based on visual criteria, and show that it raises a number of challenging questions. We then focus on articulated‐body dynamics simulation, and propose a semi‐predictive approach which relies on a combination of exact, a priori error metrics computations, and visibility estimations. We suggest several variants of semi‐predictive metrics based on hierarchical data structures and the use of graphics hardware, and discuss their relative merits in terms of computational efficiency and precision. Finally, we present several benchmarks and demonstrate how our view‐dependent articulated‐body dynamics method allows an animator (or a physics engine) to finely tune the visual quality and obtain potentially significant speed‐ups during interactive or off‐line simulations. Copyright © 2008 John Wiley & Sons, Ltd.