In this work, we study one-shot video object localization problem that aims to localize instances of unseen objects in the target video using a single query image of the object. Toward addressing this challenging problem, we extend a popular and successful object detection method, namely DETR (Detection Transformer), and introduce a novel approach –query-guided detection transformer for videos (QDETRv). A distinctive feature of QDETRv is its capacity to exploit information from the query image and spatio-temporal context of the target video, which significantly aids in precisely pinpointing the desired object in the video. We incorporate cross-attention mechanisms that capture temporal relationships across adjacent frames to handle the dynamic context in videos effectively. Further, to ensure strong initialization for QDETRv, we also introduce a novel unsupervised pretraining technique tailored to videos. This involves training our model on synthetic object trajectories with an analogous objective as the query-guided localization task. During this pretraining phase, we incorporate recurrent object queries and loss functions that encourage accurate patch feature reconstruction. These additions enable better temporal understanding and robust representation learning. Our experiments show that the proposed model significantly outperforms the competitive baselines on two public benchmarks, VidOR and ImageNet-VidVRD, extended for one-shot open-set localization tasks.

With over 5000 satellites and 50000 debris currently in low-earth orbit, assessing collision risk between these entities is a problem of growing importance. With about 2 tracking points per day reported by Celestrak for each tracked entity, intermediate orbit states are “propagated” from these measurements using perturbation calculations of which SGP4 is the one in dominant use. Perturbation methods can accumulate errors of 10s of km over a week, and many attempts to use established machine learning techniques, time series analysis and deep neural networks (DNN) to predict based on past measurements have been made so far. We focus on the problem of predicting the orbit of an entity with higher accuracy over 7 days into the future in order to enable better collision risk assessment and ample time to make corrective maneuvers. Towards this, we show that training a physics informed Neural Ordinary Differential Equation (NeuraIODE) over a few measurements can help predict the near future with reduced propagation error compared to SGP4 in current use for this purpose, much further into the future than known ML based methods. We discuss training costs, and highlight challenges with scaling up this approach for the large number of debris and satellites in orbit today.

Aritra Raut, Sriparna Saha, Anutosh Maitra, Roshni Ramnani. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Intent detection is one of the crucial Natural Language Understanding(NLU) tasks studied extensively. Misclassification of intents impacts the overall performance of the conversational systems as natural language understanding is the first mean of interaction between a user and a virtual agent. The traditional approach of intent detection is limited only to textual features of user utterances and overlooks other semantic features. The sentiment and emotional state of the speaker are two such semantic features that have essential impacts on intent detection, as they implicitly express user intention conveyed through the user’s message. Depending on the context, these features can help the dialogue agent respond to the same intent with varying degrees of sentiment and emotion. Thus, investigating the role of sentiment and emotion on intent detection is a matter of great interest. The current work investigates the impact of utilizing sentiment and emotion information on intent detection tasks and proposes emotion and sentiment aided intent detection models. We also investigate the impact of sentiment and emotion using three different multitasking frameworks and present a joint model that utilizes the co-relation information across these tasks to correctly identify all these NLU aspects (intent, sentiment, and emotion). The obtained experimental results by the proposed models outperform several baselines and state-of-the-art intent detection models on multiple datasets by a significant margin of 1.8% - 3%, demonstrating the significant role of sentiment and emotion features in intent detection1.

Movies reflect society and also hold power to transform opinions. Social biases and stereotypes present in movies can cause extensive damage due to their reach. These biases are not always found to be the need of storyline but can creep in as the author’s bias. Movie production houses would prefer to ascertain that the bias present in a script is the story’s demand. Today, when deep learning models can give human-level accuracy in multiple tasks, having an AI solution to identify the biases present in the script at the writing stage can help them avoid the inconvenience of stalled release, lawsuits, etc. Since AI solutions are data intensive and there exists no domain specific data to address the problem of biases in scripts, we introduce a new dataset of movie scripts that are annotated for identity bias. The dataset contains dialogue turns annotated for (i) bias labels for seven categories, viz., gender, race/ethnicity, religion, age, occupation, LGBTQ, and other, which contains biases like body shaming, personality bias, etc. (ii) labels for sensitivity, stereotype, sentiment, emotion, emotion intensity, (iii) all labels annotated with context awareness, (iv) target groups and reason for bias labels and (v) expert-driven group-validation process for high quality annotations. We also report various baseline performances for bias identification and category detection on our dataset.

Building conversation agents requires considerable manual effort in creating training data for intents / entities as well as mapping out extensive conversation flows.In this demonstration, we present ICM (Intent and Conversation Mining), a tool which can make the BOT build and update process much faster.ICM can be used to analyze existing conversation logs and help a bot designer to cluster, visualize and analyze customer intents; train custom intent models; and also to map and optimize conversation flows.The tool can be used for first time deployment or subsequent conversational flow updates in chatbots.

In this paper, we propose an unsupervised approach for knowledge graph (KG) creation from conversational data. We make use of intent classification and slot-filling, the two important components of any dialogue agent, exploit their interconnectedness, and finally construct a KG. We build a supervised intent classifier to extract the intent classes, and then on top of this we run our occlusion based slot-information extraction algorithm. Our algorithm is able to make use of supervised training of intent classifiers for extracting the relevant slot-information in an unsupervised way. To test the effectiveness of our system, we perform both automatic and manual evaluation of our intent-classifier and slot-filling system on three dialog datasets. Finally, we construct a knowledge graph from the dialogue conversation using an algorithm that makes use of our occlusion based slot-information extraction module. Empirical evaluation shows that our occlusion based method is able to successfully extract slot information from conversations, resulting in a high-quality KG.

Developing an adequate and human-like virtual agent has been one of the primary applications of artificial intelligence. In the last few years, task-oriented dialogue systems have gained huge popularity because of their upsurging relevance and positive outcomes. In real-world, users may not always have a predefined and rigid task goal beforehand; they upgrade/downgrade/change their goal component dynamically depending upon their utility value and agent's serving capability. However, existing virtual agents fail to incorporate this dynamic behavior, leading to either unsuccessful task completion or an ungratified user experience. The paper presents an end to end multimodal dialogue system for dynamic and co-operative goal setting, which incorporates i) a multi-modal semantic state representation in policy learning to deal with multi-modal inputs, ii) a goal manager module in a traditional dialogue manager for handling dynamic and goal unavailability scenarios effectively, iii) an accumulative reward (task/persona/sentiment) for task success, personalized persuasion and user-adaptive behavior, respectively. The obtained experimental results and the comparisons with baselines firmly establish the need and efficacy of the proposed system.

We present a goal oriented virtual agent for task planning, inference and user personalization. The dialog system of the agent models the activities in the form of a task graph. Coordination and planning of these tasks are done through associated constraints and interdependencies. The system uses the interaction history of an individual and a basic user preference model to create a personalized travel planning experience, as an example. Sophisticated natural language understanding techniques are used to identify multiple intents from user, and partial user utterances. In this demonstration we show how the virtual agent can be used to help the user organize his/her day by performing a multitude of related activities including scheduling meetings, retrieving flight itineraries, making routing decisions by analyzing traffic patterns, retrieving hotel accommodation options and suggesting restaurants by integrating with multiple external systems as well as a decision support engine and knowledge graph.

We present a method for the automatic extraction of glossary terms from unconstrained natural language requirements. The glossary terms are identified in two steps - a) compute units (which are candidates for glossary terms) b) disambiguate between the mutually exclusive units to identify terms. We introduce novel linguistic techniques to identify process nouns, abstract nouns and auxiliary verbs. The identification of units also handles co-ordinating conjunctions and adjectival modifiers. This requires solving co-ordination ambiguity and adjectival modifier ambiguity. The identification of terms among the units adapts an in-document statistical metric. We present an evaluation of our method over a real-life set of software requirements' documents and compare our results with that of a base algorithm. The intricate linguistic classification and the tackling of ambiguity result in superior performance of our approach over the base algorithm.