Recent advancements in foundation models have improved autonomous tool usage and reasoning, but their capabilities in map-based reasoning remain underexplored. To address this, we introduce MapEval, a benchmark designed to assess foundation models across three distinct tasks—textual, API-based, and visual reasoning—through 700 multiple-choice questions spanning 180 cities and 54 countries, covering spatial relationships, navigation, travel planning, and real-world map interactions. Unlike prior benchmarks that focus on simple location queries, MapEval requires models to handle long-context reasoning, API interactions and visual map analysis, making it the most comprehensive evaluation framework for geospatial AI. On evaluation of 30 foundation models, including Claude-3.5-Sonnet, GPT-4o, Gemini-1.5-Pro, none surpasses 67% accuracy, with open-source models performing significantly worse and all models lagging over 20% behind human performance. These results expose critical gaps in spatial inference, as models struggle with distances, directions, route planning, and place-specific reasoning, highlighting the need for better geospatial AI to bridge the gap between foundation models and real-world navigation.

As AI grows more powerful, it will increasingly shape how we understand the world. But with this influence comes the risk of amplifying misinformation and deepening social divides—especially on consequential topics where factual accuracy directly impacts well-being. Scalable Oversight aims to ensure AI systems remain truthful even when their capabilities exceed those of their evaluators. Yet when humans serve as evaluators, their own beliefs and biases can impair judgment. We study whether AI debate can guide biased judges toward the truth by having two AI systems debate opposing sides of controversial factuality claims on COVID-19 and climate change where people hold strong prior beliefs. We conduct two studies. Study I recruits human judges with either mainstream or skeptical beliefs who evaluate claims through two protocols: debate (interaction with two AI advisors arguing opposing sides) or consultancy (interaction with a single AI advisor). Study II uses AI judges with and without human-like personas to evaluate the same protocols. In Study I, debate consistently improves human judgment accuracy and confidence calibration, outperforming consultancy by 4-10\% across COVID-19 and climate change claims. The improvement is most significant for judges with mainstream beliefs (up to +15.2\% accuracy on COVID-19 claims), though debate also helps skeptical judges who initially misjudge claims move toward accurate views (+4.7\% accuracy). In Study II, AI judges with human-like personas achieve even higher accuracy (78.5\%) than human judges (70.1\%) and default AI judges without personas (69.8\%), suggesting their potential for supervising frontier AI models. These findings highlight AI debate as a promising path toward scalable, bias-resilient oversight in contested domains.

Medical Visual Question Answering (VQA) presents challenges due to the complexity of imaging data and the need for precise, context-aware responses. Traditional VQA models often struggle in clinical settings, limiting their utility in decision-making. This study proposes DR-VQA, a fine-tuned BLIP (Bootstrapped Language-Image Pretraining) model for medical VQA specifically designed for diabetic retinopathy (DR) based on fundus imaging, leveraging both visual and textual data to generate accurate diagnostic answers. To enhance semantic relevance, BERT-based similarity evaluation is integrated. Using a diabetic retinopathy dataset, the model achieves a validation BERT similarity (BERTsim) score of 0.94 and a test score of 0.95 on 450 samples, demonstrating strong alignment with expert annotations. These results highlight the model's potential to assist clinicians by improving diagnostic accuracy and efficiency. The proposed approach can streamline medical workflows, reduce clinician workload, and enhance patient outcomes. Future work will focus on expanding datasets and refining the model for broader medical applications. We believe our approach will support to enhance the patient care as well democratization on AI technology for community.

Code synthesis, which requires a deep understanding of complex natural language (NL) problem descriptions, generation of code instructions for complex algorithms and data structures, and the successful execution of comprehensive unit tests, presents a significant challenge.Thus, while large language models (LLMs) demonstrate impressive proficiency in natural language processing (NLP), their performance in code generation tasks remains limited.In this paper, we introduce a new approach to code generation tasks leveraging the multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers.Our framework, MapCoder, consists of four LLM agents specifically designed to emulate the stages of this cycle: recalling relevant examples, planning, code generation, and debugging.After conducting thorough experiments, with multiple LLMs ablations and analyses across eight challenging competitive problem-solving and program synthesis benchmarks-MapCoder showcases remarkable code generation capabilities, achieving their new state-of-the-art (pass@1) results-(HumanEval 93.9%, MBPP 83.1%, APPS 22.0%, CodeContests 28.5%, and xCodeEval 45.3%).Moreover, our method consistently delivers superior performance across various programming languages and varying problem difficulties.

Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, Shafiq Joty. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Data-driven storytelling is a powerful method for conveying insights by combining narrative techniques with visualizations and text. These stories integrate visual aids, such as highlighted bars and lines in charts, along with textual annotations explaining insights. However, creating such stories requires a deep understanding of the data and meticulous narrative planning, often necessitating human intervention, which can be time-consuming and mentally taxing. While Large Language Models (LLMs) excel in various NLP tasks, their ability to generate coherent and comprehensive data stories remains underexplored. In this work, we introduce a novel task for data story generation and a benchmark containing 1,449 stories from diverse sources. To address the challenges of crafting coherent data stories, we propose a multi-agent framework employing two LLM agents designed to replicate the human storytelling process: one for understanding and describing the data (Reflection), generating the outline, and narration, and another for verification at each intermediary step. While our agentic framework generally outperforms non-agentic counterparts in both model-based and human evaluations, the results also reveal unique challenges in data story generation.

Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Prior studies in privacy policies frame the question answering (QA) task as identifying the most relevant text segment or a list of sentences from a policy document given a user query.Existing labeled datasets are heavily imbalanced (only a few relevant segments), limiting the QA performance in this domain.In this paper, we develop a data augmentation framework based on ensembling retriever models that captures the relevant text segments from unlabeled policy documents and expand the positive examples in the training set.In addition, to improve the diversity and quality of the augmented data, we leverage multiple pre-trained language models (LMs) and cascade them with noise reduction filter models.Using our augmented data on the PrivacyQA benchmark, we elevate the existing baseline by a large margin (10% F1) and achieve a new state-of-the-art F1 score of 50%.Our ablation studies provide further insights into the effectiveness of our approach.

Transfer learning that adapts a model trained on data-rich sources to low-resource targets has been widely applied in natural language processing (NLP).However, when training a transfer model over multiple sources, not every source is equally useful for the target.To better transfer a model, it is essential to understand the values of the sources.In this paper, we develop SEAL-Shap, an efficient source valuation framework for quantifying the usefulness of the sources (e.g., domains/languages) in transfer learning based on the Shapley value method.Experiments and comprehensive analyses on both cross-domain and cross-lingual transfers demonstrate that our framework is not only effective in choosing useful transfer sources but also the source values match the intuitive source-target similarity.

Md Rizwan Parvez, Tolga Bolukbasi, Kai-Wei Chang, Venkatesh Saligrama. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Text in many domains involves a significant amount of named entities.Predicting the entity names is often challenging for a language model as they appear less frequent on the training corpus.In this paper, we propose a novel and effective approach to building a discriminative language model which can learn the entity names by leveraging their entity type information.We also introduce two benchmark datasets based on recipes and Java programming codes, on which we evaluate the proposed model.Experimental results show that our model achieves 52.2% better perplexity in recipe generation and 22.06% on code generation than the stateof-the-art language models.

Street crime is a prevalent problem in developing countries like Bangladesh. Though this problem has been identified long before, no visible remedy or action can be seen to overcome or combat these street crimes in one of the most populated mega-cities, Dhaka, Bangladesh, of the world. In this paper, we propose a novel spatio-temporal street crime prediction model that exploits the historical street crime data of Dhaka city to predict the possibility of a crime in a particular region at a specific time. Our model captures both space and time proximity of past crimes while predicting a future crime. Experimental evaluation shows that our spatiotemporal prediction model can predict a future crime with 79.24% sensitivity and 68.2% specificity. As a proof of concept we develop an Android application that alerts a user about the possibility of different crimes in a place at a particular time.