Large language models (LLMs) are widely used in various tasks and applications.However, despite their wide capabilities, they are shown to lack cultural alignment (Ryan et al., 2024;AlKhamissi et al., 2024) and produce biased generations (Naous et al., 2024) due to a lack of cultural knowledge and competence.Evaluation of LLMs for cultural awareness and alignment is particularly challenging due to the lack of proper evaluation metrics and unavailability of culturally grounded datasets representing the vast complexity of cultures at the regional and sub-regional levels.Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives.To address this issue, we introduce a novel CSI dataset for Indian culture, belonging to 17 cultural facets.The dataset comprises ∼8k cultural concepts from 36 sub-regions.To measure the cultural competence of LLMs on a cultural text adaptation task, we evaluate the adaptations using the CSIs created, LLM as Judge, and human evaluations from diverse socio-demographic region.Furthermore, we perform quantitative analysis demonstrating selective sub-regional coverage and surface-level adaptations across all considered LLMs.

Drones excel at capturing aerial-view images, especially in human unreachable areas. Automatically interpreting and describing these images enables decision-making easier without the need to review the images extensively. Traditional image captioning models struggle with aerial imagery due to diverse orientations, perspectives, and unclear objects. Integrating the capabilities of Large Vision Language Models (LVLMs) with drone images can improve description utility, benefiting strategic missions like surveillance, search and rescue, etc. However, the lack of image-caption datasets for drone imagery poses a significant challenge for training and evaluating drone image captioning. To address this gap, we contribute the first Aerial-view Image Captioning dataset (AeroCaps), containing four captions per image. Another major hurdle for the task is the hallucinatory nature of LVLMs. To this end, we perform the first extensive analysis of hallucinations on aerial imagery by two SOTA LVLMs - LLaVA and InstructBLIP on our proposed dataset and VisDrone. We explore the reasons behind such hallucinations. We release the LVLM-generated image captions along with our hallucination-labelled annotations as the Labelled Illusion Dataset (LID) for further research. Additionally, we review how effective advanced LLMs like GPT-4 are in evaluating the degree of hallucinations made by other LVLMs like LLaVA.

The standard language modeling (LM) loss by itself has been shown to be inadequate for effective dialogue modeling.As a result, various training approaches, such as auxiliary loss functions and leveraging human feedback, are being adopted to enrich open-domain dialogue systems.One such auxiliary loss function is Bag-of-Words (BoW) loss, defined as the crossentropy loss for predicting all the words/tokens of the next utterance.In this work, we propose a novel auxiliary loss named Bag-of-Keywords (BoK) loss to capture the central thought of the response through keyword prediction and leverage it to enhance the generation of meaningful and interpretable responses in open-domain dialogue systems.BoK loss upgrades the BoW loss by predicting only the keywords or critical words/tokens of the next utterance, intending to estimate the core idea rather than the entire response.We incorporate BoK loss in both encoder-decoder (T5) and decoder-only (Di-aloGPT) architecture and train the models to minimize the weighted sum of BoK and LM (BoK-LM) loss.We perform our experiments on two popular open-domain dialogue datasets, DailyDialog and Persona-Chat.We show that the inclusion of BoK loss improves the dialogue generation of backbone models while also enabling post-hoc interpretability.We also study the effectiveness of BoK-LM loss as a reference-free metric and observe comparable performance to the state-of-the-art metrics on various dialogue evaluation datasets.

Text Style Transfer (TST) involves transforming a source sentence with a given style label to an output with another target style meanwhile preserving content and fluency.We look at a fill-in-the-blanks approach (also referred to as prototype editing), where the source sentence is stripped off all style-containing words and filled in with suitable words.This closely resembles a Masked Language Model (MLM) objective, with the added initial step of masking only relevant style words rather than BERT's random masking.We show this simple MLM, trained to reconstruct style-masked sentences back into their original style, can even transfer style by making this MLM "Style-Aware".This simply involves appending the source sentence with a target style special token.The Style-Aware MLM (SA-MLM) now also accounts for the direction of style transfer and enables style transfer by simply manipulating these special tokens.To learn this n-word to n-word style reconstruction task, we use a single transformer encoder block with 8 heads, 2 layers and no auto-regressive decoder, making it non-generational.We empirically show that this lightweight encoder trained on a simple reconstruction task compares with elaborately engineered state-of-the-art TST models for even complex styles like Discourse or flow of logic, i.e.Contradiction to Entailment and vice-versa.Additionally, we introduce a more accurate attention-based style-masking step and a novel "attention-surplus" method to determine the position of masks from any arbitrary attribution model in O(1) time.Finally, we show that the SA-MLM arises naturally by considering a probabilistic framework for style transfer.

Many real-world applications are associated with collection of events with timestamps, known as time-to-event data. Earthquake occurrences, social networks, and user activity logs can be represented as a sequence of discrete events observed in continuous time. Temporal point process serves as an essential tool for modeling such time-to-event data in continuous time space. Despite having massive amounts of event sequence data from various domains like social media, healthcare etc., real world application of temporal point process faces two major challenges: 1) it is not generalizable to predict events from unseen event sequences in dynamic environment 2) they are not capable of thriving in continually evolving environment with minimal supervision while retaining previously learnt knowledge. To tackle these issues, we propose HyperHawkes, a hypernetwork based temporal point process framework which is capable of modeling time of event occurrence for unseen sequences and consequently, zero-shot learning for time-to-event modeling. We also develop a hypernetwork based continually learning temporal point process for continuous modeling of time-to-event sequences with minimal forgetting. HyperHawkes augments the temporal point process with zero-shot modeling and continual learning capabilities. We demonstrate the application of the proposed framework through our experiments on real-world datasets. Our results show the efficacy of the proposed approach in terms of predicting future events under zero-shot regime for unseen event sequences. We also show that the proposed model is able to learn the time-to-event sequences continually while retaining information from previous event sequences, mitigating catastrophic forgetting in neural temporal point process.

In dialogue systems, automatically evaluating machine-generated responses is critical and challenging.Despite the tremendous progress in dialogue generation research, its evaluation heavily depends on human judgments.The standard word-overlapping based evaluation metrics are ineffective for dialogues.As a result, most of the recently proposed metrics are model-based and reference-free, which learn to score different aspects of a conversation.However, understanding each aspect requires a separate model, which makes them computationally expensive.To this end, we propose Dial-M, a Masking-based reference-free framework for Dialogue evaluation.The main idea is to mask the keywords of the current utterance and predict them, given the dialogue history and various conditions (like knowledge, persona, etc.), thereby making the evaluation framework simple and easily extensible for multiple datasets.Regardless of its simplicity, Dial-M achieves comparable performance to state-ofthe-art metrics on several dialogue evaluation datasets.We also discuss the interpretability of our proposed metric along with error analysis.

Identification and categorization of social media posts generated during disasters are crucial to reduce the suffering of the affected people. However, the lack of labeled data is a significant bottleneck in learning an effective categorization system for a disaster. This motivates us to study the problem as unsupervised domain adaptation (UDA) between a previous disaster with labeled data (source) and a current disaster (target). However, if the amount of labeled data available is limited, it restricts the learning capabilities of the model. To handle this challenge, we use limited labeled data along with abundantly available unlabeled data, generated during a source disaster to propose a novel two-part graph neural network (GNN). The first part extracts domain-agnostic global information by constructing a token-level graph across domains and the second part preserves local instance-level semantics. In our experiments, we show that the proposed method outperforms state-of-the-art techniques by 2.74% weighted F1 score on average on two standard public datasets in the area of disaster management. We also report experimental results for granular actionable multilabel classification datasets in disaster domain for the first time, on which we outperform BERT by 3.00% on average w.r.t. weighted F1. Additionally, we show that our approach can retain performance when minimal labeled data are available.

Sharan Narasimhan, Suvodip Dey, Maunendra Desarkar. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Semantic characterization of the Point-of-Interest (POI) plays an important role for modeling location-based social networks and various related applications like POI recommendation, link prediction etc. However, semantic categories are not available for many POIs which makes this characterization difficult. Semantic annotation aims to predict such missing categories of POIs. Existing approaches learn a representation of POIs using graph neural networks to predict semantic categories. However, LBSNs involve complex and higher order mobility dynamics. These higher order relations can be captured effectively by employing hypergraphs. Moreover, visits to POIs can be attributed to various reasons like temporal characteristics, spatial context etc. Hence, we propose a Multi-view Hypergraph Convolution Network (Multi-HGCN) where we learn POI representations by considering multiple hypergraphs across multiple views of the data. We build a comprehensive model to learn the POI representation capturing temporal, spatial and trajectory-based patterns among POIs by employing hypergraphs. We use hypergraph convolution to learn better POI representation by using spectral properties of hypergraph. Experiments conducted on three real-world datasets show that the proposed approach outperforms the state-of-the-art approaches.

Task oriented virtual assistants or dialogue systems are being popular for different domains such as restaurant booking, weather update, flight booking etc. The efforts are supported by availability of large scale annotated conversational datasets for such domains. However, the same is not true for transport domain dialogue systems. Moreover, for such systems to be useful, they should be able to handle natural queries submitted by users. For countries like India where most of the people communicate in regional languages, it is important to have such systems support the regional languages. The existing datasets for transport domain are mostly monolingual in nature and support only English language. For countries like India, where people tend to speak multiple languages and have code-mixed conversations the existing systems and the datasets won’t be of much use. To the best of our knowledge, there is no multilingual code-mixed dataset available for designing public transport related conversation systems. In this paper, we propose a code-mixed English-Hindi dataset to accelerate the development of transport domain conversational systems suitable for countries like India. Our dataset has multiple intents like: route finding, bus/train/cab finding, nearby place search, traffic alert queries, out of domain queries. We also provide initial baseline results for user intent identification using existing state of the art models on our dataset and a prototype to show the usability of the work.

Dialogue State Tracking (DST) is a sub-task of task-based dialogue systems where the user intention is tracked through a set of (domain, slot, slot-value) triplets.Existing DST models can be difficult to extend for new datasets with larger domains/slots mainly due to either of the two reasons-i) prediction of domain-slot as a pair, and ii) dependency of model parameters on the number of slots and domains.In this work, we propose to address these issues using a Hierarchical DST (Hi-DST) model.At a given turn, the model first detects a change in domain followed by domain prediction if required.Then it decides suitable action for each slot in the predicted domains and finds their value accordingly.The model parameters of Hi-DST are independent of the number of domains/slots.Due to the hierarchical modeling, it achieves O(|M | + |N |) belief state prediction for a single turn where M and N are the set of unique domains and slots respectively.We argue that the hierarchical structure helps in the model explainability and makes it easily extensible to new datasets.Experiments on the MultiWOZ dataset show that our proposed model achieves comparable joint accuracy performance to state-of-the-art DST models.

Abstract Scientific documents generally contain multiple mathematical expressions in them. Detecting inline mathematical expressions are one of the most important and challenging tasks in scientific text mining. Recent works that detect inline mathematical expressions in scientific documents have looked at the problem from an image processing perspective. There is little work that has targeted the problem from NLP perspective. Towards this, we define a few features and applied Conditional Random Fields (CRF) to detect inline mathematical expressions in scientific documents. Apart from this feature based approach, we also propose a hybrid algorithm that combines Bidirectional Long Short Term Memory networks (Bi‐LSTM) and feature‐based approach for this task. Experimental results suggest that this proposed hybrid method outperforms several baselines in the literature and also individual methods in the hybrid approach.

The task of generating incorrect options for multiple-choice questions is termed as distractor generation problem. The task requires high cognitive skills and is extremely challenging to automate. Existing neural approaches for the task leverage encoder-decoder architecture to generate long distractors. However, in this process two critical points are ignored - firstly, many methods use Jaccard similarity over a pool of candidate distractors to sample the distractors. This often makes the generated distractors too obvious or not relevant to the question context. Secondly, some approaches did not consider the answer in the model, which caused the generated distractors to be either answer-revealing or semantically equivalent to the answer.

The prevalence of location-based social networks (LBSNs) has eased the understanding of human mobility patterns. However, categories which act as semantic characterization of the location, might be missing for some check-ins and can adversely affect modelling the mobility dynamics of users. At the same time, mobility patterns provide cues on the missing semantic categories. In this paper, we simultaneously address the problem of semantic annotation of locations and location adoption dynamics of users. We propose our model HAP-SAP, a latent spatio-temporal multivariate Hawkes process, which considers latent semantic category influences, and temporal and spatial mobility patterns of users. The inferred semantic categories can supplement our model on predicting the next check-in events by users. Our experiments on real datasets demonstrate the effectiveness of the proposed model for the semantic annotation and location adoption modelling tasks.

We propose IR2V EC , a Concise and Scalable encoding infrastructure to represent programs as a distributed embedding in continuous space. This distributed embedding is obtained by combining representation learning methods with flow information to capture the syntax as well as the semantics of the input programs. As our infrastructure is based on the Intermediate Representation (IR) of the source code, obtained embeddings are both language and machine independent. The entities of the IR are modeled as relationships, and their representations are learned to form a seed embedding vocabulary . Using this infrastructure, we propose two incremental encodings: Symbolic and Flow-Aware . Symbolic encodings are obtained from the seed embedding vocabulary , and Flow-Aware encodings are obtained by augmenting the Symbolic encodings with the flow information. We show the effectiveness of our methodology on two optimization tasks (Heterogeneous device mapping and Thread coarsening). Our way of representing the programs enables us to use non-sequential models resulting in orders of magnitude of faster training time. Both the encodings generated by IR2V EC outperform the existing methods in both the tasks, even while using simple machine learning models. In particular, our results improve or match the state-of-the-art speedup in 11/14 benchmark-suites in the device mapping task across two platforms and 53/68 benchmarks in the thread coarsening task across four different platforms. When compared to the other methods, our embeddings are more scalable , is non-data-hungry , and has better Out-Of-Vocabulary (OOV) characteristics .

Word embedding techniques in literature are mostly based on Bag of Words models where words that co-occur with each other are considered to be related. However, it is not necessary for similar or related words to occur in the same context window. In this paper, we propose a new approach to combine different types of resources for training word embeddings. The lexical resources used in this work are Dependency Parse Tree and WordNet. Apart from the co-occurrence information, the use of these additional resources helps us in including the semantic and syntactic information from the text in learning the word representations. The learned representations are evaluated on multiple evaluation tasks like Semantic Textual Similarity, Word Similarity. Results of the experimental analyses highlight the usefulness of the proposed methodology.

Micro-blogging sites provide a wealth of resources during disaster events in the form of short texts. Correct classification of these text data into various actionable classes can be of great help in shaping the means to rescue people in disaster-affected places. The process of classification of these text data poses a challenging problem because the texts are usually short and very noisy and finding good features that can distinguish these texts into different classes is time consuming, tedious and often requires a lot of domain knowledge. We propose a deep learning based model to classify tweets into different actionable classes such as resource need and availability, activities of various NGO etc. Our model requires no domain knowledge and can be used in any disaster scenario with little to no modification.

There has been a massive rise in the use of Community Question and Answering (CQA) forums to get solutions to various technical and non-technical queries. One common problem faced in CQA is the small number of experts, which leaves many questions unanswered. This paper addresses the challenging problem of predicting the best answerer for a new question and thereby recommending the best expert for the same. Although there are work in the literature that aim to find possible answerers for questions posted in CQA, very few algorithms exist for finding the best answerer whose answer will satisfy the information need of the original Poster. For finding answerers, existing approaches mostly use features based on content and tags associated with the questions. There are few approaches that additionally consider the users' history. In this paper, we propose an approach that considers a comprehensive set of features including but not limited to text representation, tag based similarity as well as multiple user-based features that target users' availability, agility as well as expertise for predicting the best answerer for a given question. We also include features that give incentives to users who answer less but more important questions over those who answer a lot of questions of less importance. A learning to rank algorithm is used to find the weight of each feature. Experiments conducted on a real dataset from Stack Exchange show the efficacy of the proposed method in terms of multiple evaluation metrics for accuracy, robustness and real time performance.

As the social networking sites get more popular, spammers target these sites to spread spam posts. Twitter is one of the most popular online social networking sites where users communicate and interact on various topics. Most of the current spam filtering methods in Twitter focus on detecting the spammers and blocking them. However, spammers can create a new account and start posting new spam tweets again. So there is a need for robust spam detection techniques to detect the spam at tweet level. These types of techniques can prevent the spam in real time. To detect the spam at tweet level, often features are defined, and appropriate machine learning algorithms are applied in the literature. Recently, deep learning methods are showing fruitful results on several natural language processing tasks. We want to use the potential benefits of these two types of methods for our problem. Toward this, we propose an ensemble approach for spam detection at tweet level. We develop various deep learning models based on convolutional neural networks (CNNs). Five CNNs and one feature-based model are used in the ensemble. Each CNN uses different word embeddings (Glove, Word2vec) to train the model. The feature-based model uses content-based, user-based, and n-gram features. Our approach combines both deep learning and traditional feature-based models using a multilayer neural network which acts as a meta-classifier. We evaluate our method on two data sets, one data set is balanced, and another one is imbalanced. The experimental results show that our proposed method outperforms the existing methods.

The phenomenal growth in the healthcare data has inspired us in investigating robust and scalable models for data mining. For classification problems Information Gain(IG) based Decision Tree is one of the popular choices. However, depending upon the nature of the dataset, IG based Decision Tree may not always perform well as it prefers the attribute with more number of distinct values as the splitting attribute. Healthcare datasets generally have many attributes and each attribute generally has many distinct values. In this paper, we have tried to focus on this characteristics of the datasets while analysing the performance of our proposed approach which is a variant of Decision Tree model and uses the concept of Correlation Ratio(CR). Unlike IG based approach, this CR based approach has no biasness towards the attribute with more number of distinct values. We have applied our model on some benchmark healthcare datasets to show the effectiveness of the proposed technique.

With more and more news sources publishing news articles online, news recommendation systems are becoming increasingly popular in recent days. However, if a news recommender shows to the user many news articles that cover same story or talk about same persons or events, then the recommendations might become monotonous for the user. A possible way to handle this scenario is to diversify the recommended news articles so that the articles in the recommendation list are not too similar to each other. Existing research works in literature discuss how to use user-specific relevance of news articles and diversification to produce the final recommendations. To be able to compute user-specific relevance of articles, several recommender systems suggest that users have a login, or they set cookies to track page visits of the users. Often users are unsure whether the information collected via these means will be used for the sole purpose of news recommendation. Privacy of the identity and page access patterns is a concern for many users. In this work, we consider the problem where diverse recommendations are to be given for users, but no past data about users is collected or stored by the system. We use only article popularity to estimate the relevance. We identify few aspects of news articles on which diversification can be performed. These aspects are incorporated in an optimization framework. An approximation algorithm is used to generate the recommendations. Experimental analysis shows that in scenarios where no individual user data is stored or accessible by the recommender system, diversification results in higher click rates and increase in the number of satisfied users. Performances of the multiple diversification aspects and approaches are also analyzed.

The field of context and intelligence, as a topic of pervasive computing, has been gaining considerable momentum. Typically, context-aware intelligence is applied to understand the situation of the users and their behavior with the objective of providing adaptive services that are closely associated with that context. In this work, we have taken an orthogonal approach wherein we attempt to aggregate knowledge and cognition, of the user, on a given topic to build models out of them. The model thus created is analyzed to derive inferences about the user, where the analysis is performed on a graph model comprising topics based information obtained by mining domain specific personal data sources and from certain facts on which the user has expressed fair level of belief. We have explored the possibility of deriving beneficial information by provisioning an appropriate representation of knowledge as belief-graph with specific orientation in healthcare and call this model as Med-Tree. Subject to privacy conditions, we open up the belief-graph model to establish objective based social connections that gets contextually bound. As a next step, such contextually bound ad-hoc networks are subjected to advanced querying process resulting in useful information extraction and inferences. Leveraging on user's knowledge or the belief-graph, the proposed Med-Tree could help derive benefits towards better personal healthcare and disease management.

In the Recommendation Problem, it is often important to find a set of items similar to a particular item or a group of items. This problem of finding similar items for the recommendation task may also be viewed as a link prediction problem in a network, where the items can be treated as the nodes. The strength of the edge connecting two items represents the similarity between the items. In this context, a central challenge is to suitably define an appropriate dissimilarity function between the items. For content based recommender systems, the dissimilarity function should take into account the individual attributes of the items. The same attribute may have different importances in different parts of the underlying network. We focus on the problem of learning a suitable dissimilarity function between items and address it by formulating it as a constrained optimization problem which captures the local weightages of the attributes in different regions of the graph. The constraints are imposed in such a way that the non-connected nodes show higher value of dissimilarity than the connected nodes. The local tuning of the weights learns the optimal value of weights in various parts of the network: from the portions having rich graph information to the portions having only content information. Detailed experimentation shows the superiority of the proposed algorithm over the Adamic Adar metric as well as logistic regression methodology.

Collaborative filtering is a widely used technique for rating prediction in recommender systems. Memory based collaborative filtering algorithms assign weights to the users to capture similarities between them. The weighted average of similar users' ratings for the test item is output as prediction. We propose a memory based algorithm that is markedly different from the existing approaches. We use preference relations instead of absolute ratings for similarity calculations, as preference relations between items are generally more consistent than ratings across like-minded users. Each user's ratings are viewed as a preference graph. Similarity weights are learned using an iterative method motivated by online learning. These weights are used to create an aggregate preference graph. Ratings are inferred to maximally agree with this aggregate graph. The use of preference relations allows the rating of an item to be influenced by other items, which is not the case in the weighted-average approaches of the existing techniques. This is very effective when the data is sparse, specially for the items rated by few users. Our experiments show that the our method outperforms other methods in the sparse regions. However, for dense regions, sometimes our results are comparable to the competing approaches, and sometimes worse.