This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless-M4T using Word Error Rate (WER), along with a detailed examination of the most frequent wrong words and error types including insertions, deletions, and substitutions. Our analysis is conducted using two types of datasets, read speech and conversational speech. Notably, we present the first conversational speech dataset designed for benchmarking Urdu ASR models. We find that seamless-large outperforms other ASR models on the read speech dataset, while whisper-large performs best on the conversational speech dataset. Furthermore, this evaluation highlights the complexities of assessing ASR models for low-resource languages like Urdu using quantitative metrics alone and emphasizes the need for a robust Urdu text normalization system. Our findings contribute valuable insights for developing robust ASR systems for low-resource languages like Urdu.

Active learning (AL) techniques reduce labeling costs for training neural machine translation (NMT) models by selecting smaller representative subsets from unlabeled data for annotation. Diversity sampling techniques select heterogeneous instances, while uncertainty sampling methods select instances with the highest model uncertainty. Both approaches have limitations - diversity methods may extract varied but trivial examples, while uncertainty sampling can yield repetitive, uninformative instances. To bridge this gap, we propose Hybrid Uncertainty and Diversity Sampling (HUDS), an AL strategy for domain adaptation in NMT that combines uncertainty and diversity for sentence selection. HUDS computes uncertainty scores for unlabeled sentences and subsequently stratifies them. It then clusters sentence embeddings within each stratum and computes diversity scores by distance to the centroid. A weighted hybrid score that combines uncertainty and diversity is then used to select the top instances for annotation in each AL iteration. Experiments on multi-domain German-English and French-English datasets demonstrate the better performance of HUDS over other strong AL baselines. We analyze the sentence selection with HUDS and show that it prioritizes diverse instances having high model uncertainty for annotation in early AL iterations.

With the widespread adoption of Large Language Models (LLMs) across various applications, it is imperative to ensure their fairness across all user communities.However, most LLMs are trained and evaluated on Western centric data, with little attention paid to lowresource languages and regional contexts.To address this gap, we introduce PakBBQ, a culturally and regionally adapted extension of the original Bias Benchmark for Question Answering (BBQ) dataset.PakBBQ comprises over 214 templates, 17180 QA pairs across 8 categories in both English and Urdu, covering eight bias dimensions including age, disability, appearance, gender, socio-economic status, religious, regional affiliation, and language formality that are relevant in Pakistan.We evaluate multiple multilingual LLMs under both ambiguous and explicitly disambiguated contexts, as well as negative versus non negative question framings.Our experiments reveal (i) an average accuracy gain of 12% with disambiguation, (ii) consistently stronger counter bias behaviors in Urdu than in English, and (iii) marked framing effects that reduce stereotypical responses when questions are posed negatively.These findings highlight the importance of contextualized benchmarks and simple prompt engineering strategies for bias mitigation in low resource settings.

The growing prevalence of text reuse and plagiarism in various fields has led to an urgent need for reliable computational methods for detection. However, current commercial plagiarism detection systems are ineffective in identifying paraphrased cases of text reuse, highlighting the need for improvement. Previous research on paraphrased text reuse and plagiarism detection has mainly focused on English, European, Persian, and Arabic languages, and very few studies have been reported on the under-resourced Urdu language. This study aims to overcome this research gap by using a Deep Neural Network (DNN) based architecture and pre-trained Large Language Models (LLMs) for the task of Urdu paraphrased text reuse and plagiarism detection. The architecture called Deep Text Reuse and Paraphrased Plagiarism Detection (D-TRaPPD), relies on LLMs for input and utilizes CNN and LSTM to extract essential textual features. Moreover, we have proposed and evaluated two D-TRaPPD variants, Word Embeddings-D-TRaPPD (WE-D-TRaPPD) and Sentence Embeddings-D-TRaPPD (SE-D-TRaPPD), using two gold standard document-level corpora containing both real and simulated cases of Urdu paraphrased text reuse and plagiarism. The results demonstrate the effectiveness of the D-TRaPPD architecture, with SE-D-TRaPPD achieving the highest $$F_1$$ scores of 91.77 for real cases and 95.15 for simulated cases. Furthermore, the results highlight the superiority of our approaches over the state-of-the-art methods for Urdu paraphrased text reuse and plagiarism detection.

Plagiarism, the unauthorized reuse of text, fueled by the ease of access to online content, is a pressing concern for academia, publishers, and authors. Paraphrasing, a common tactic in textual plagiarism, compounds the problem further. The automatic detection of paraphrased plagiarism in text documents is a fundamental task in Natural Language Processing (NLP), crucial for maintaining academic integrity and authenticity. This article presents an extensive investigation into Urdu sentential paraphrased plagiarism detection leveraging advanced Deep Neural Networks (DNNs) and Large Language Models (LLMs). The study builds upon the foundational work and proposes modifications to the Deep Text Reuse and Paraphrased Plagiarism Detection (D-TRaPPD) architecture to incorporate state-of-the-art pre-trained LLMs. The proposed approach, SELLM-D-TRaPPD, integrates various language models, including contextualized sentence embedding-based LLMs, language-agnostic and multilingual transformer-based LLMs, and multilingual knowledge-distilled transformer-based LLMs. We evaluated these models against three benchmark Urdu sentential paraphrase corpora—Urdu Sentential Paraphrase Corpus, Urdu Short Text Reuse Corpus, and Semi-automatic Urdu Sentential Paraphrase Corpus. The results demonstrate the effectiveness of SELLM-D-TRaPPD with LLMs, achieving F1 scores of 92.09%, 96.70%, and 98.23%, respectively. A comparative analysis with existing state-of-the-art methods shows significant performance improvements, establishing SELLM-D-TRaPPD as the new leading approach for Urdu sentential paraphrased plagiarism detection. These findings highlight the value of leveraging advanced neural network architectures and pre-trained LLMs in improving the accuracy and effectiveness of paraphrased plagiarism detection in Urdu, addressing a crucial gap in Urdu NLP research.

This paper introduces UQA, a novel dataset for question answering and text comprehension in Urdu, a low-resource language with over 70 million native speakers. UQA is generated by translating the Stanford Question Answering Dataset (SQuAD2.0), a large-scale English QA dataset, using a technique called EATS (Enclose to Anchor, Translate, Seek), which preserves the answer spans in the translated context paragraphs. The paper describes the process of selecting and evaluating the best translation model among two candidates: Google Translator and Seamless M4T. The paper also benchmarks several state-of-the-art multilingual QA models on UQA, including mBERT, XLM-RoBERTa, and mT5, and reports promising results. For XLM-RoBERTa-XL, we have an F1 score of 85.99 and 74.56 EM. UQA is a valuable resource for developing and testing multilingual NLP systems for Urdu and for enhancing the cross-lingual transferability of existing models. Further, the paper demonstrates the effectiveness of EATS for creating high-quality datasets for other languages and domains. The UQA dataset and the code are publicly available at www.github.com/sameearif/UQA

Recent studies have highlighted the challenges low-literacy users face with complex user interfaces, often preventing them from utilizing essential smartphone applications for an improved quality of life. This paper introduces the EvolveUI design approach that diverges from conventional interface design by evolving in complexity alongside a user’s growing proficiency. Initially presenting a single navigation point to simplify interaction, EvolveUI systematically expands, introducing more features and navigation points as users become more adept at interacting with the interface. We build upon previous adaptive user interface research by uniquely focusing on expanding functionality based on user proficiency, offering a tailored experience that aligns with individual learning curves. By conceptualizing the interface as a dynamically expanding hierarchy, starting as a minimalist “tree” and unfolding into a more complex structure, EvolveUI facilitates a more accessible and engaging user experience. Our study compares this evolving interface approach with conventional designs through usability tests on a mobile health application, demonstrating EvolveUI’s potential to enhance technology accessibility for low-literacy users and suggesting new directions for inclusive design practices.

In Low- and Middle-Income Countries (LMICs), poor health outcomes come from a high burden of disease, a shortage of healthcare professionals, and inefficient health information exchange leading to substantial economic losses. In this paper, we highlight critical gaps in healthcare delivery in Pakistan and propose solutions to improve patient outcomes in resource-constrained environments. We have built Darcheeni, an AI-driven healthcare framework that leverages artificial intelligence to assist and supplement physicians, streamline healthcare processes, and prioritize patient-centered care. Darcheeni analyzes doctor-patient interactions in real-time, integrates lab and imaging data, generates and distributes care plans customized to the patient’s needs, and sends them directly to patients’ smartphones. We also discuss the challenges and limitations associated with sustainable AI integration by centering our learnings from the pilot deployment of Darcheeni. By focusing on Pakistan as a case study, this work offers practical insights and strategies for deploying AI-driven technologies sustainably in similar resource-constrained environments and contributes to the broader discourse on the role of AI in global health improvement.

Abstract Automatic paraphrase detection is the task of measuring the semantic overlap between two given texts. A major hurdle in the development and evaluation of paraphrase detection approaches, particularly for South Asian languages like Urdu, is the inadequacy of standard evaluation resources. The very few available paraphrased corpora for these languages are manually created. As a result, they are constrained to smaller sizes and are not very feasible to evaluate mainstream data-driven and deep neural networks (DNNs)-based approaches. Consequently, there is a need to develop semi- or fully automated corpus generation approaches for the resource-scarce languages. There is currently no semi- or fully automatically generated sentence-level Urdu paraphrase corpus. Moreover, no study is available to localize and compare approaches for Urdu paraphrase detection that focus on various mainstream deep neural architectures and pretrained language models. This research study addresses this problem by presenting a semi-automatic pipeline for generating paraphrased corpora for Urdu. It also presents a corpus that is generated using the proposed approach. This corpus contains 3147 semi-automatically extracted Urdu sentence pairs that are manually tagged as paraphrased (854) and non-paraphrased (2293). Finally, this paper proposes two novel approaches based on DNNs for the task of paraphrase detection in Urdu text. These are Word Embeddings n-gram Overlap (henceforth called WENGO), and a modified approach, Deep Text Reuse and Paraphrase Plagiarism Detection (henceforth called D-TRAPPD). Both of these approaches have been evaluated on two related tasks: (i) paraphrase detection, and (ii) text reuse and plagiarism detection. The results from these evaluations revealed that D-TRAPPD ( $F_1 = 96.80$ for paraphrase detection and $F_1 = 88.90$ for text reuse and plagiarism detection) outperformed WENGO ( $F_1 = 81.64$ for paraphrase detection and $F_1 = 61.19$ for text reuse and plagiarism detection) as well as other state-of-the-art approaches for these two tasks. The corpus, models, and our implementations have been made available as free to download for the research community.

The COVID-19 pandemic has necessitated rapid top-down dissemination of reliable and actionable information. This presents unique challenges in engaging low-literate communities that live in poverty and lack access to the Internet. We describe the design and deployment of a voice-based social media platform, accessible over simple phones, for actively engaging such communities in Pakistan with reliable COVID information. We developed three strategies to overcome users’ hesitation, mistrust, and skepticism in engaging with COVID content. Users were: (1) encouraged to listen to reliable COVID advisory, (2) incentivized to share authentic content with others, and (3) prompted to critically think about COVID-related information behaviors. Using a mixed-methods evaluation, we show that users approached with all three strategies had a significantly higher engagement with COVID content compared to others. We discuss how new designs of social media can enable users to engage with and propagate authentic information.

With the advancement in natural-sounding speech production models, it is becoming important to develop models that can detect spoofed audios. Synthesized speech models do not explicitly account for all factors affecting speech production, such as the shape, size and structure of a speaker's vocal tract. In this paper, we hypothesize that due to practical limitations of audio corpora (including size, distribution, and balance of variables like gender, age, and accents), there exist certain phonemes that synthesized models are not able to replicate as well as the human articulation system and such phonemes differ in their spectral characteristics from bonafide speech. To discover such phonemes and quantify their effectiveness in distinguishing between spoofed and bonafide speech, we use a deep learning model with self-attention, and analyze the attention weights of the trained model. We use the ASVSpoof2019 dataset for our analysis and find that the attention mechanism picks most on fricatives: /S/,/SH/, nasals: /M/,/N/, vowels: /Y/, and stops: /D/. Furthermore, we obtain 7.54% EER on train and 11.98% on dev data when using only the top-16 most attended phonemes from input audio, better than when any other phoneme classes are used.

Crowdsourcing enables the completion of large-scale and hard-to-automate tasks, while allowing people to earn money. However, 3.6 billion people – a workforce comprising 46.4% of the world population – who could benefit most from this source of income lack the access and literacy to use computers, smartphones, and the internet. In this paper we present Karamad, a voice-based crowdsourcing platform that allows workers in low-resource regions to complete crowd work using low-end phones and receive payments as mobile airtime balance. We explore the usefulness, scalability, and sustainability of Karamad in Pakistan through a 6-month deployment. Without any advertising, training, or airtime subsidies, Karamad organically engaged 725 workers who completed 3,939 tasks (involving 43,006 components) including translations, dataset generation, and surveys on demographics, accessibility, disability, health, employment, and literacy. Collectively, the workers produced a valuable service market for potential customers and included female, unemployed, non-literate, and blind users.

We examine the opportunities and challenges in designing for maternal health in low-income, low-resource communities in patriarchal and religious contexts. Pakistan faces a crisis in maternal health with a maternal mortality ratio of 178 deaths per 100,000 live births, as compared to the developed-country average of just 12 deaths per 100,000. Through a 6-month long qualitative, empirical study we examine the prevalent beliefs and practices around maternal health in Pakistan, the access women have to health-care, the existing religious practices that influence them and the agency they exert in their own health-care decision making. We reveal the rampant misinformation among mothers and health workers, house-hold power dynamics that impact maternal health and the deep link between maternal health and religious beliefs. We also show how current maternal health care interventions fit poorly into this context and discuss alternate design recommendations for meeting the maternal health needs of these women.

Information and Communication Technology interventions have the potential to improve outcomes in health and other development sectors in low-income settings. Large-scale impact, however, remains the central challenge for the HCI4D community as significant and diverse resources are typically required to scale such interventions beyond the pilot stage. In contrast, voice-based entertainment services accessible over simple phones, designed for similarly low-income, low-literate populations manage to scale 'virally' to tens of thousands of users with little to no advertising cost. Our study compares the outcomes of using voice-based entertainment to spread a maternal-health hotline against conventional advertisement channels including paper flyers, posters, radio, TV, social media and robocalls. Through an 11-week deployment in Pakistan where the hotline reached 21,770 users over 32,625 calls, we find that the entertainment service outperformed other channels on all popular user acquisition metrics, with the exception of robocalls, which lead in terms of spread.

This paper presents the first attempt at Automatic Text Simplification (ATS) for Urdu, the language of 170 million people worldwide. Being a low-resource language in terms of standard linguistic resources, recent text simplification approaches that rely on manually crafted simplified corpora or lexicons such as WordNet are not applicable to Urdu. Urdu is a morphologically rich language that requires unique considerations such as proper handling of inflectional case and honorifics. We present an unsupervised method for lexical simplification of complex Urdu text. Our method only requires plain Urdu text and makes use of word embeddings together with a set of morphological features to generate simplifications. Our system achieves a BLEU score of 80.15 and SARI score of 42.02 upon automatic evaluation on manually crafted simplified corpora. We also report results for human evaluations for correctness, grammaticality, meaning-preservation and simplicity of the output. Our code and corpus are publicly available to make our results reproducible.

Information dissemination using automated phone calls allows reaching low-literate and tech-naive populations. Open challenges include rapid verification of expected knowledge gaps in the community, dissemination of specific information to address these gaps, and follow-up measurement of knowledge retention. We report Sawaal, a voice-based telephone service that uses audio-quizzes to address these challenges. Sawaal allows its open community of users to post and attempt multiple-choice questions and to vote and comment on them. Sawaal spreads virally as users challenge friends to quiz competitions. Administrator-posted questions allow confirming specific knowledge gaps, spreading correct information and measuring knowledge retention via rephrased, repeated questions. In 14 weeks and with no advertisement, Sawaal reached 3,433 users (120,119 calls) in Pakistan, who contributed 13,276 questions that were attempted 455,158 times by 2,027 users. Knowledge retention remained significant for up to two weeks. Surveys revealed that 71% of the mostly low-literate, young, male users were blind.

HCI4D researchers and practitioners have leveraged voice forums to enable people with literacy, socioeconomic, and connectivity barriers to access, report, and share information. Although voice forums have received impassioned usage from low-income, low-literate, rural, tribal, and disabled communities in diverse HCI4D contexts, the participation of women in these services is almost non-existent. In this paper, we investigate the reasons for the low participation of women in social media voice forums by examining the use of Sangeet Swara in India and Baang in Pakistan by marginalized women and men. Our mixed-methods approach spanning content analysis of audio posts, quantitative analysis of interactions between users, and qualitative interviews with users indicate gender inequity due to deep-rooted patriarchal values. We found that women on these forums faced systemic discrimination and encountered abusive content, flirts, threats, and harassment. We discuss design recommendations to create social media voice forums that foster gender equity in use of these services.

Rotating Savings and Credit Association (ROSCA) is a mechanism of informal collaborative savings that is widely used across the globe. Despite its popularity and prevalence, it is not well-studied from HCI and CSCW perspectives. The global increase in mobile penetration has created opportunities to serve the unbanked using mobile-based Digital Financial Services (DFS) for greater financial inclusion but there have not been any DFS-based interventions around ROSCAs. In this paper, we report a qualitative study involving 80 individuals to understand the dynamics of ROSCAs and opportunities for their digitization in the Pakistani context. We also present a smartphone-based Digital ROSCA platform designed on top of a simulated mobile money system. The platform was designed to be inclusive towards low-literate users. We present qualitative findings of its evaluation with 15 users (3 individual ROSCA groups). We find that digitization has the potential to support and strengthen traditional ROSCAs by mitigating issues like record-keeping, delayed payments, collection, distribution, and safety of money. It also allows the creation of payment history for individuals that can be used to score their financial credibility.

Speech is more natural than text for a large part of the world including hard-to-reach populations (low-literate, poor, tech-novice, visually-impaired, marginalized) and oral cultures. Voice-based services over simple mobile phones are effective means to provide orality-driven social connectivity to such populations. We present Baang, a versatile and inclusive voice-based social platform that allows audio content creation and sharing among its open community of users. Within 8 months, Baang spread virally to 10,721 users (69% of them blind) who participated in 269,468 calls and shared their thoughts via 44,178 audio-posts, 343,542 votes, 124,389 audio-comments and 94,864 shares. We show that the ability to vote, comment and share leads to viral spread, deeper engagement, longer retention and emergence of true dialog among participants. Beyond connectivity, Baang provides its users with a voice and a social identity as well as means to share information and get community support.

State-of-the-art Natural Language Processing algorithms rely heavily on efficient word segmentation. Urdu is amongst languages for which word segmentation is a complex task as it exhibits space omission as well as space insertion issues. This is partly due to the Arabic script which although cursive in nature, consists of characters that have inherent joining and non-joining attributes regardless of word boundary. This paper presents a word segmentation system for Urdu which uses a Conditional Random Field sequence modeler with orthographic, linguistic and morphological features. Our proposed model automatically learns to predict white space as word boundary as well as Zero Width Non-Joiner (ZWNJ) as sub-word boundary. Using a manually annotated corpus, our model achieves F1 score of 0.97 for word boundary identification and 0.85 for sub-word boundary identification tasks. We have made our code and corpus publicly available to make our results reproducible.

This paper presents results of a field study of an interactive voice response (IVR) system developed for the agricultural community of Punjab, Pakistan. We studied the information requirements and the user demographics to develop a basic IVR system which disseminates agro-information such as weather forecast, pesticide and fertilizer information etc. In terms of usability and information extraction, simple menu-based navigation was, relatively, easily understood and used. The usage was evaluated and results show that such a system is a viable option to deliver agro-information.

We explore how development-related, voice-based, information services could organically spread among low-literate masses in the developing world. We report lessons learned from a remote deployment of "Polly" in India (from the US) to spread job-related information. Polly is an entertainment driven, voice-based service, available over simple phones that is aimed at familiarizing people with speech interfaces and mass-dissemination of development related information to low-literate users. In 2012, Polly had become viral in Pakistan and successfully spread recorded newspaper job ads to thousands of mobile phone users. Remotely deployed in India, Polly did not take off immediately as it did in Pakistan. Instead, it initially entered a six-month long phase of fluctuating, intermittent activity. We experimented with various forms of seeding and it eventually transitioned into a viral phase, with sustained transmission that continued for five months but without (exponential) growth. Finally, interface adjustments in response to user feedback enabling plain-voice asynchronous voice-messaging resulted in an abrupt exponential and viral growth amassing 10,349 phone calls by 1,613 users over a span of seven days. Of these, 299 users also transitioned to the job service. User feedback and surveys suggest possible reasons for each phase. We study the challenges of remote deployment and the interplay of user interface; language of the system; seeding mechanisms and active response to user feedback towards the uptake of the service. We also report a detailed comparison of viral spread in the two countries.

We explore how telephone-based services might be mass adopted by low-literate users in the developing world. We focus on speech and push-button dialog systems requiring neither literacy nor training. Building on the success of Polly, a simple telephone-based voice manipulation and forwarding system that was first tested in 2011, we report on its first large-scale sustained deployment. In 24/7 operation in Pakistan since May 9, 2012, as of mid-September Polly has spread to 85,000 users, engaging them in 495,000 interactions, and is continuing to spread to 1,000 new people daily. It has also attracted 27,000 people to a job search service, who in turn listened 279,000 times to job ads and forwarded them 22,000 times to their friends. We report users' activity over time and across demographics, analyze user behavior within several randomized controlled trials, and describe lessons learned regarding spread, scalability and sustainability of telephone-based speech-based services.

Entertainment has recently been shown to be a powerful motivator for mastering new technologies. We therefore set out to use viral entertainment to introduce telephone-based, speech-based services to low-literate people in developing countries. We describe Polly, a simple voice manipulation and forwarding system that went viral in Pakistan last year. Seeded once by 32 low-skilled office workers in a Pakistani university, in 3 weeks Polly amassed 2,032 users and 10,629 interactions. From analyzing the traffic and its content, it is evident that Polly has been used extensively for entertainment and social contact, but it has also been put to an unintended use as a voicemail and group messaging facility. This demonstrated the potential for speech based services, and the pent-up demand for entertainment, among our target population. Also of note, Polly's viral spread crossed gender and age boundaries and even established itself in a female population. However, it appears to have not crossed socioeconomic boundaries.