We present NERBench-Chhattisgarh, a gold-standard Named Entity Recognition (NER) dataset covering seven under-resourced languages spoken in Central India: Baigani, Chhattisgarhi, Surgujia, Sadri, Kudukh, Halbi, and Gondi. Addressing the digital divide for tribal languages, our corpus comprises 166,444 annotated tokens across 8,391 sentences, spanning both Indo-Aryan and Dravidian language families. The dataset features high lexical sparsity and a "nature-centric" ontology of 22 entity types based on the CLIA Phase-II schema, capturing culturally specific entities often absent in standard benchmarks. To establish a benchmark for language variety-aware information access, we evaluate three multilingual encoders: mBERT, XLM-RoBERTa, and IndicBERT, using two adaptation strategies: direct parameter-efficient fine-tuning (LoRA) and a Chhattisgarhi-Pivot Adaptive Pre-training (CPAP) approach. Our results show a morphological barrier: while pivot-based adaptation facilitates transfer for distant languages through script alignment, it induces negative transfer in morphologically complex agglutinative languages like Gondi. We publicly release the dataset, code, and adapted model checkpoints. https://github.com/Rajesh-NLP/NER-Chhattisgarh to support future research in inclusive Information Retrieval.

Sarcasm identification in textual data is the most captivating area of research in the current research trends. It is a challenging task for humans as well as for the computer. In this article, we have tried to identify sarcasm in the Hindi newspaper headlines of two of the most-read Hindi newspapers in India, namely Hindustan and Dainik Jagran. Initially, we collected 88,518 Hindi newspaper headlines and identified 1,945 headlines to be sarcastic, which we have considered for the present study. The headlines taken into consideration belong to the political domain and were published during some of the recent Legislative Assembly Elections of 2020, 2021, and 2022. Various machine learning and deep learning techniques have been used to develop the baseline models. It justifies the assumption that sarcastic text does not always bear a negative sentiment. It may bear a positive sentiment depending on the context. The present article aims at the creation of a dataset consisting of 1,945 Hindi newspaper headlines, training and testing machine learning and deep learning models, namely Extra Trees Classifier, Random Forest Classifier, XGBClassifier, fasttext-stackedTCN, and mBERT-stackedTCN for sarcasm identification on the dataset and comparing the results obtained by the models after the experiment. Out of all the choosen models, the Random Forest Classifier performs better with \(F_1\) score of 92.11 before data augmentation and 90.68 after data augmentation.

One of the primary tasks in the early stages of data mining involves the identification of entities from biomedical corpora. Traditional approaches relying on robust feature engineering face challenges when learning from available (un-)annotated data using data-driven models like deep learning-based architectures. Despite leveraging large corpora and advanced deep learning models, domain generalization remains an issue. Attention mechanisms are effective in capturing longer sentence dependencies and extracting semantic and syntactic information from limited annotated datasets. To address out-of-vocabulary challenges in biomedical text, the PCA-CLS (Position and Contextual Attention with CNN-LSTM-Softmax) model combines global self-attention and character-level convolutional neural network techniques. The model's performance is evaluated on eight distinct biomedical domain datasets encompassing entities such as genes, drugs, diseases, and species. The PCA-CLS model outperforms several state-of-the-art models, achieving notable F-scores, including 88.19% on BC2GM, 85.44% on JNLPBA, 90.80% on BC5CDR-chemical, 87.07% on BC5CDR-disease, 89.18% on BC4CHEMD, 88.81% on NCBI, and 91.59% on the s800 dataset.

Machine translation (MT) systems have been built using numerous different techniques for bridging the language barriers. These techniques are broadly categorized into approaches like Statistical Machine Translation (SMT) and Neural Machine Translation (NMT). End-to-end NMT systems significantly outperform SMT in translation quality on many language pairs, especially those with the adequate parallel corpus. We report comparative experiments on baseline MT systems for Assamese to other Indo-Aryan languages (in both translation directions) using the traditional Phrase-Based SMT as well as some more successful NMT architectures, namely basic sequence-to-sequence model with attention, Transformer, and finetuned Transformer. The results are evaluated using the most prominent and popular standard automatic metric BLEU (BiLingual Evaluation Understudy), as well as other well-known metrics for exploring the performance of different baseline MT systems, since this is the first such work involving Assamese. The evaluation scores are compared for SMT and NMT models for the effectiveness of bi-directional language pairs involving Assamese and other Indo-Aryan languages (Bangla, Gujarati, Hindi, Marathi, Odia, Sinhalese, and Urdu). The highest BLEU scores obtained are for Assamese to Sinhalese for SMT (35.63) and the Assamese to Bangla for NMT systems (seq2seq is 50.92, Transformer is 50.01, and finetuned Transformer is 50.19). We also try to relate the results with the language characteristics, distances, family trees, domains, data sizes, and sentence lengths. We find that the effect of the domain is the most important factor affecting the results for the given data domains and sizes. We compare our results with the only existing MT system for Assamese (Bing Translator) and also with pairs involving Hindi.

Corpus preparation for low-resource languages and for development of human language technology to analyze or computationally process them is a laborious task, primarily due to the unavailability of expert linguists who are native speakers of these languages and also due to the time and resources required. Bhojpuri, Magahi, and Maithili, languages of the Purvanchal region of India (in the north-eastern parts), are low-resource languages belonging to the Indo-Aryan (or Indic) family. They are closely related to Hindi, which is a relatively high-resource language, which is why we compare them with Hindi. We collected corpora for these three languages from various sources and cleaned them to the extent possible, without changing the data in them. The text belongs to different domains and genres. We calculated some basic statistical measures for these corpora at character, word, syllable, and morpheme levels. These corpora were also annotated with parts-of-speech (POS) and chunk tags. The basic statistical measures were both absolute and relative and were expected to indicate linguistic properties, such as morphological, lexical, phonological, and syntactic complexities (or richness). The results were compared with a standard Hindi corpus. For most of the measures, we tried to match the corpus size across the languages to avoid the effect of corpus size, but in some cases it turned out that using the full corpus was better, even if sizes were very different. Although the results are not very clear, we tried to draw some conclusions about the languages and the corpora. For POS tagging and chunking, the BIS tagset was used to manually annotate the data. The POS-tagged data sizes are 16,067, 14,669, and 12,310 sentences, respectively, for Bhojpuri, Magahi, and Maithili. The sizes for chunking are 9,695 and 1,954 sentences for Bhojpuri and Maithili, respectively. The inter-annotator agreement for these annotations, using Cohen’s Kappa, was 0.92, 0.64, and 0.74, respectively, for the three languages. These (annotated) corpora have been used for developing preliminary automated tools, which include POS tagger, Chunker, and Language Identifier. We have also developed the Bilingual dictionary (Purvanchal languages to Hindi) and a Synset (that can be integrated later in the Indo-WordNet) as additional resources. The main contribution of the work is the creation of basic resources for facilitating further language processing research for these languages, providing some quantitative measures about them and their similarities among themselves and with Hindi. For similarities, we use a somewhat novel measure of language similarity based on an n-gram-based language identification algorithm. An additional contribution is providing baselines for three basic NLP applications (POS tagging, chunking, and language identification) for these closely related languages.