Motivating children to learn is a major challenge in education. One way to inspire motivation to learn is through immersion. We combine the immersive potential of augmented reality (AR), narrative, and large language models (LLMs) to bridge fantasy with reality in a mobile application, Moon Story, that teaches elementary schoolers astronomy and environmental science. Our system also builds upon learning theories such as culturally-relevant pedagogy. Using our application, a child embarks on a journey inspired by Chinese mythology, engages in real-world AR activities, and converses with a fictional character powered by an LLM. We conducted a controlled experiment (N = 50) with two conditions: one using an LLM and one that was hard-coded. Both conditions resulted in learning gains, high engagement levels, and increased science learning motivation. Participants in the LLM condition also wrote more relevant answers. Finally, participants of both Chinese and non-Chinese heritage found the culturally-based narrative compelling.

Navigating autonomous vehicles efficiently across unstructured and off-road terrains remains a formidable challenge, often requiring intricate mapping or multi-step pipelines. However, these conventional approaches struggle to adapt to dynamic environments. This paper presents TerrainSense, an end-to-end framework that overcomes these limitations. By utilizing a transformers, TerrainSense detects lane semantics and topology from camera images, enabling mapless path planning without the reliance on highly detailed maps. The efficacy of TerrainSense was rigorously assessed on six diverse datasets, evaluating its efficacy in detection, segmentation, and path prediction using various metrics. Notably, it outperforms the other state-of-the-art methods by 9.32% in precisely predicting the path with 18.28% faster inference time.

In many reinforcement learning problems, parameters of the model may vary with its phase while the agent attempts to learn through its interaction with the environment. For example, an autonomous car's reward on selecting a path may depend on traffic conditions at the time of the day or the transition dynamics of a drone may depend on the current wind direction. Many such processes exhibit a cyclic phase-structure and could be represented with a control policy parameterized over a circular or cyclic phase space. Attempting to model such phase variations with a standard data-driven approach (e.g. deep networks) without explicitly modeling the phase of the model can be challenging. Ambiguities may arise as the optimal action for a given state can vary depending on the phase. To better model cyclic environments, we propose phase-parameterized policies and value function approximators that explicitly enforce a cyclic structure to the policy or value space. We apply our phase-parameterized reinforcement learning approach to both feed-forward and recurrent deep networks in the context of trajectory optimization and locomotion problems. Our experiments show that our proposed approach has superior modeling performance than traditional function approximators in cyclic environments.

The word spotting approach is extremely useful for searching and annotating documents for which robust recognizers are unavailable. Traditionally, hand-designed features were used to represent the word images for spotting. In this paper, we learn a data-driven representation for word-images from Convolutional Neural Networks (CNNs). Previous approaches that learn deep neural networks for a particular task/dataset are difficult to design and train for generic word spotting. Instead, by “adapting” a CNN trained for a different problem, we show tremendous speedup in the training phase. Our experiments show that features extracted from an adapted-CNN handsomely outperform hand-designed features on both spotting and recognition tasks for printed (English and Telugu) and handwritten (IAM) document collections.

In this paper we present the techniques used for the University of Montréal's team submissions to the 2013 Emotion Recognition in the Wild Challenge. The challenge is to classify the emotions expressed by the primary human subject in short video clips extracted from feature length movies. This involves the analysis of video clips of acted scenes lasting approximately one-two seconds, including the audio track which may contain human voices as well as background music. Our approach combines multiple deep neural networks for different data modalities, including: (1) a deep convolutional neural network for the analysis of facial expressions within video frames; (2) a deep belief net to capture audio information; (3) a deep autoencoder to model the spatio-temporal information produced by the human actions depicted within the entire scene; and (4) a shallow network architecture focused on extracted features of the mouth of the primary human subject in the scene. We discuss each of these techniques, their performance characteristics and different strategies to aggregate their predictions. Our best single model was a convolutional neural network trained to predict emotions from static frames using two large data sets, the Toronto Face Database and our own set of faces images harvested from Google image search, followed by a per frame aggregation strategy that used the challenge training data. This yielded a test set accuracy of 35.58%. Using our best strategy for aggregating our top performing models into a single predictor we were able to produce an accuracy of 41.03% on the challenge test set. These compare favorably to the challenge baseline test set accuracy of 27.56%.