Indian poetry, known for its linguistic complexity and deep cultural resonance, has a rich and varied heritage spanning thousands of years. However, its layered meanings, cultural allusions, and sophisticated grammatical constructions often pose challenges for comprehension, especially for non-native speakers or readers unfamiliar with its context and language. Despite its cultural significance, existing works on poetry have largely overlooked Indian language poems. In this paper, we propose the Translation and Image Generation (TAI) framework, leveraging Large Language Models (LLMs) and Latent Diffusion Models through appropriate prompt tuning. Our framework supports the United Nations Sustainable Development Goals of Quality Education (SDG 4) and Reduced Inequalities (SDG 10), by enhancing the accessibility of culturally rich Indian-language poetry to a global audience. It includes (1) a translation module that uses an Odds Ratio Preference Alignment Algorithm to accurately translate morphologically rich poetry into English; (2) an image generation module that employs a semantic graph to capture tokens, dependencies, and semantic relationships between metaphors and their meanings, to create visually meaningful representations of Indian poems. Our comprehensive experimental evaluation, including both human and quantitative assessments, demonstrates the superiority of TAI Diffusion in poem image generation tasks, outperforming strong baselines. To further address the scarcity of resources for Indian-language poetry, we introduce the Morphologically Rich Indian Language Poems MorphoVerse Dataset, comprising 1,570 poems across 21 low-resource Indian languages. By addressing the gap in poetry translation and visual comprehension, this work aims to broaden accessibility and enrich the reader’s experience.

Existing approaches to complaint analysis largely rely on unimodal, short-form content such as tweets or product reviews. This work advances the field by leveraging multimodal, multi-turn customer support dialogues—where users often share both textual complaints and visual evidence (e.g., screenshots, product photos)—to enable fine-grained classification of complaint aspects and severity. We introduce VALOR, a Validation-Aware Learner with Expert Routing, tailored for this multimodal setting. It employs a multi-expert reasoning setup using large-scale generative models with Chain-of-Thought (CoT) prompting for nuanced decision-making. To ensure coherence between modalities, a semantic alignment score is computed and integrated into the final classification through a meta-fusion strategy. In alignment with the United Nations Sustainable Development Goals (UN SDGs), the proposed framework supports SDG 9 (Industry, Innovation and Infrastructure) by advancing AI-driven tools for robust, scalable, and context-aware service infrastructure. Further, by enabling structured analysis of complaint narratives and visual context, it contributes to SDG 12 (Responsible Consumption and Production) by promoting more responsive product design and improved accountability in consumer services. We evaluate VALOR on a curated multimodal complaint dataset annotated with fine-grained aspect and severity labels, showing that it consistently outperforms baseline models, especially in complex complaint scenarios where information is distributed across text and images. This study underscores the value of multimodal interaction and expert validation in practical complaint understanding systems.

Vision-Language Models (VLMs) are increasingly used in clinical diagnostics, yet their robustness to adversarial attacks remains largely unexplored, posing serious risks.Existing medical attacks focus on secondary objectives such as model stealing or adversarial finetuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect.To address this, we propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible.The method injects coordinated perturbations into non-diagnostic background regions and employs an attention-distraction mechanism to shift the model's focus away from pathological areas.Extensive evaluations across six medical imaging modalities show that MedFocusLeak achieves state-of-the-art performance, generating misleading yet realistic diagnostic outputs across diverse VLMs.We further introduce a unified evaluation framework with novel metrics that jointly capture attack success and image fidelity, revealing a critical weakness in the reasoning capabilities of modern clinical VLMs.The code associated with this project is available at MedFocusLeak.

India's linguistic landscape, spanning 22 scheduled languages and hundreds of marginalized dialects, has driven rapid growth in NLP datasets, benchmarks, and pretrained models.However, no dedicated survey consolidates resources developed specifically for Indian languages.Existing reviews either focus on a few high-resource languages or subsume Indian languages within broader multilingual settings, limiting coverage of low-resource and culturally diverse varieties.To address this gap, we present the first unified survey of Indian NLP resources, covering 200+ datasets, 50+ benchmarks, and 100+ models, tools, and systems across text, speech, multimodal, and culturally grounded tasks.We organize resources by linguistic phenomena, domains, and modalities; analyze trends in annotation, evaluation, and model design; and identify persistent challenges such as data sparsity, uneven language coverage, script diversity, and limited cultural and domain generalization.This survey offers a consolidated foundation for equitable, culturally grounded, and scalable NLP research in the Indian linguistic ecosystem.

Eric Onyame, Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen, Chirag Agarwal. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Conversational recommender systems (CRSs) operate under incremental preference revelation, requiring recommendation decisions under uncertainty.While recent LLM-based approaches achieve strong performance on proxy metrics such as Recall@K and BLEU, they often fail to deliver high-quality, user-aligned recommendations in practice, as they optimize intermediate objectives like retrieval accuracy or fluent generation rather than recommendation quality itself.We propose HARPO (Hierarchical Agentic Reasoning with Preference Optimization), an agentic framework that reframes conversational recommendation as a structured decision-making process optimized for multi-dimensional recommendation quality.HARPO integrates (i) hierarchical preference learning that decomposes recommendation quality into interpretable dimensions (relevance, diversity, satisfaction, and engagement) with context-dependent weighting; (ii) deliberative tree-search reasoning guided by a learned value network evaluating candidate paths on predicted quality; and (iii) domainagnostic reasoning abstractions through Virtual Tool Operations and multi-agent refinement.We evaluate HARPO on ReDial, IN-SPIRED, and MUSE, demonstrating consistent improvements over strong baselines on recommendation-centric metrics while maintaining competitive response quality.Our code is available at https://harpo-bench.github.io.

Radiology impression generation involves producing concise, clinically meaningful summaries from detailed imaging findings such as CT and MRI scans, serving as a critical aid in diagnosis and treatment planning. However, recent studies highlight a severe shortage of radiologists, particularly in low and middle-income countries, where there is fewer than one radiologist per 100,000 people, making timely expert interpretation a significant challenge. While advancements in AI, especially large language models (LLMs), offer promising potential to automate this task, current systems often suffer from hallucinations, omissions of key clinical details, and a lack of linguistic clarity, thereby raising serious concerns about their safety and reliability in real-world clinical settings. In this work, we attempted to address this issue by introducing RADO , a novel framework for radiology impression generation that integrates safety, faithfulness, and linguistic refinement rewards for preference optimization. To support robust evaluation, we introduce RIB , a real-world benchmark dataset curated and annotated by radiologists, spanning 1,429 annotated CT and MRI findings and impressions across 27 study types. RADO enforces critical safety and factuality constraints via carefully designed reward models and achieves state-of-the-art performance across multiple automatic and human evaluation metrics. Our framework significantly outperforms existing baselines, demonstrating improved factual consistency, reduced omissions, and higher clinical relevance, thus advancing the safety and reliability of generative AI in high-stakes medical applications. The code and dataset associated with the work are made available at RADO .

Misinformation poses a growing threat across media ecosystems, yet research on Hindi, one of the world's most widely spoken languages, remains limited. We introduce a novel multimodal Hindi dataset of 6,544 article–image pairs to advance misinformation detection. Unlike existing English-centric and predominantly unimodal datasets, ours integrates text, images, and affective signals while being carefully cleaned of veracity cues to avoid artefact-driven inflation. Each sample is annotated with sentiment and emotions, making this the first Hindi resource with multimodal and affective dimensions. Through extensive experiments using IndicBART, IndicBERT, mBERT, and Vision Transformer models, we demonstrate the effectiveness of text–image fusion and affective features across multiple configurations. We also analyze the readability characteristics of genuine and misleading articles, providing insights into the linguistic patterns of Hindi misinformation. This dataset establishes a robust benchmark for multimodal misinformation detection and lays essential groundwork for research in Hindi and other low-resource languages.

Financial grievances on social media increasingly feature multimodal content, combining text and images to express complex concerns. Previous text-only approaches often struggle to capture mixed emotional phenomena within posts, such as praise for credit card services alongside dissatisfaction with branch wait times. Prior research has relied on binary classifications, overlooking root aspect-specific complaints (e.g., categorizing card services as noncomplaint while branch services are complaint-related). To address the scarcity of multimodal data resources for this task, we developed the financial multimodal complaint (FMC) dataset, comprising over 6,000 tweets across nine financial domains, capturing cross-modal complaint dynamics. To demonstrate the practical use of the FMC dataset, we designed F-ACE, a multimodal, aspect-based, explainable complaint identification model that integrates BLIP-2, Sentence Transformer, and ResNet-152 via an intermodal fusion (IMF) framework, followed by a BART-Large-based causal extractor for nuanced aspect classification. Furthermore, we evaluate the proposed model’s comprehension and adaptability through the utilization of various domain-specific and cross-domain datasets alongside state-of-the-art models. Our contribution sets a preferable alternative for advancing research in aspect-aware, multimodal complaint analysis, enabling a deeper understanding of financial grievances in the digital age.

In this study, we investigate the role of visual content in accurately predicting movie genres. By extracting keyframes from Hindi, Bengali, Malayalam, and Telugu language-based Indian movie trailers in the Flickscore dataset, we analyse visual elements using visual language model (VLM) and Large Language Models (LLMs). Our approach focuses on the FAMOS aspects (focus, action, mood, object, setting) to understand the movie’s theme, summary, and genre. This method eliminates the need to manually prepare textual metadata, thus reducing the time and effort required for genre identification. The visual features captured from the keyframes are leveraged to predict genres more efficiently, offering a reliable way to automate this process. Additionally, we integrate the visual information into a content-based filtering (CBF) system to predict user preferences. Our study highlights the effectiveness of using visual features, which significantly enhances the performance of recommendation systems (RS) by improving accuracy in predicting user preferences based on movie genres. We demonstrate that by analysing the implicit content in movie frames, we can achieve better insights than traditional metadata-driven approaches. Overall, our findings emphasise that visual content holds valuable information for understanding movie genres and can play a critical role in user preference understanding.

Despite progress in comment-aware multimodal and multilingual summarization for English and Chinese, research in Indian languages remains limited. This study addresses this gap by introducing COSMMIC, a pioneering comment-sensitive multimodal, multilingual dataset featuring nine major Indian languages. COSMMIC comprises 4,959 article-image pairs and 24,484 reader comments, with ground-truth summaries available in all included languages. Our approach enhances summaries by integrating reader insights and feedback. We explore summarization and headline generation across four configurations: (1) using article text alone, (2) incorporating user comments, (3) utilizing images, and (4) combining text, comments, and images. To assess the dataset’s effectiveness, we employ state-of-the-art language models such as LLama3 and GPT-4. We conduct a comprehensive study to evaluate different component combinations, including identifying supportive comments, filtering out noise using a dedicated comment classifier using IndicBERT, and extracting valuable insights from images with a multilingual CLIP-based classifier. This helps determine the most effective configurations for natural language generation (NLG) tasks. Unlike many existing datasets that are either text-only or lack user comments in multimodal settings, COSMMIC uniquely integrates text, images, and user feedback. This holistic approach bridges gaps in Indian language resources, advancing NLP research and fostering inclusivity.

Akash Ghosh, Aparna Garimella, Pritika Ramu, Sambaran Bandyopadhyay, Sriparna Saha. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

While there exists a lot of work on explainable complaint mining, articulating user concerns through text or video remains a significant challenge, often leaving issues unresolved. Users frequently struggle to express their complaints clearly in text but can easily upload videos depicting product defects (e.g., vague text such as 'worst product' paired with a 5-second video depicting a broken headphone with the right earcup). This paper formulates a new task in the field of complaint mining to aid the common users' need to write an expressive complaint, which is Complaint Description from Videos (CoD-V) (e.g., to help the above user articulate her complaint about the defective right earcup). To this end, we introduce ComVID, a video complaint dataset containing 1,175 complaint videos and the corresponding descriptions, also annotated with the emotional state of the complainer. Additionally, we present a new complaint retention (CR) evaluation metric that discriminates proposed (CoD-V) task against standard video summary generation and description task. To strengthen this initiative, we introduce a multimodal Retrieval-Augmented Generation (RAG) embedded VideoLLaMA2-7b model, designed to generate complaints while accounting for the user's emotional state. We conduct a comprehensive evaluation of several Video Language Models on several tasks (pre-trained and fine-tuned versions) with a range of established evaluation metrics, including METEOR, perplexity, and the Coleman-Liau readability score, among others. Our study lays the foundation for a new research direction to provide a platform for users to express complaints through video. Dataset and resources are available at: https://github.com/sarmistha-D/CoD-V.

The task of text-to-image generation has encountered significant challenges when applied to literary works, especially poetry. Poems are a distinct form of literature, with meanings that frequently transcend beyond the literal words. To address this shortcoming, we propose a PoemToPixel framework designed to generate images that visually represent the inherent meanings of poems. Our approach incorporates the concept of prompt tuning in our image generation framework to ensure that the resulting images closely align with the poetic content. In addition, we propose the PoeKey algorithm, which extracts three key elements in the form of emotions, visual elements, and themes from poems to form instructions which are subsequently provided to a diffusion model for generating corresponding images. Furthermore, to expand the diversity of the poetry dataset across different genres and ages, we introduce MiniPo, a novel multimodal dataset comprising 1001 children’s poems and images. Leveraging this dataset alongside PoemSum, we conducted both quantitative and qualitative evaluations of image generation using our PoemToPixel framework. This paper demonstrates the effectiveness of our approach and offers a fresh perspective on generating images from literary sources. The code and dataset used in this work are publicly available.

The exponential technological breakthrough of the FinTech industry has significantly enhanced user engagement through sophisticated advisory chatbots. However, large-scale fine-tuning of LLMs can occasionally yield unprofessional or flippant remarks, such as “With that money, you’re going to change the world,” which, though factually correct, can be contextually inappropriate and erode user trust. The scarcity of domain-specific datasets has led previous studies to focus on isolated components, such as reasoning-aware frameworks or the enhancement of human-like response generation. To address this research gap, we present Fin-Solution 2.O, an advanced solution that 1) introduces the multi-turn financial conversational dataset, Fin-Vault, and 2) incorporates a unified model, Fin-Ally, which integrates commonsense reasoning, politeness, and human-like conversational dynamics. Fin-Ally is powered by COMET-BART-embedded commonsense context and optimized with a Direct Preference Optimization (DPO) mechanism to generate human-aligned responses. The novel Fin-Vault dataset, consisting of 1,417 annotated multi-turn dialogues, enables Fin-Ally to extend beyond basic account management to provide personalized budgeting, real-time expense tracking, and automated financial planning. Our comprehensive results demonstrate that incorporating commonsense context enables language models to generate more refined, textually precise, and professionally grounded financial guidance, positioning this approach as a next-generation AI solution for the FinTech sector.

Recent advancements in text-to-image diffusion models have achieved remarkable success in generating realistic and diverse visual content. A critical factor in this process is the model’s ability to accurately interpret textual prompts. However, these models often struggle with creative expressions, particularly those involving complex, abstract, or highly descriptive language. In this work, we introduce a novel training-free approach tailored to improve image generation for a unique form of creative language: poetic verse, which frequently features layered, abstract, and dual meanings. Our proposed PoemTale Diffusion approach aims to minimise the information that is lost during poetic text-to-image conversion by integrating a multi stage prompt refinement loop into Language Models to enhance the interpretability of poetic texts. To support this, we adapt existing state-of-the-art diffusion models by modifying their self-attention mechanisms with a consistent self-attention technique to generate multiple consistent images, which are then collectively used to convey the poem’s meaning. Moreover, to encourage research in the field of poetry, we introduce the P4I (PoemForImage) dataset, consisting of 1,111 poems sourced from multiple online and offline resources. We engaged a panel of poetry experts for qualitative assessments. The results from both human and quantitative evaluations validate the efficacy of our method and contribute a novel perspective to poem-to-image generation with enhanced information capture in the generated images.

With the increasing use of Retrieval-Augmented Generation (RAG), strong retrieval models have become more important than ever.In healthcare, multimodal retrieval models that combine information from both text and images offer major advantages for many downstream tasks such as question answering, cross-modal retrieval, and multimodal summarization, since medical data often includes both formats.However, there is currently no standard benchmark to evaluate how well these models perform in medical settings.To address this gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark.M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over 1.2 Million text documents and 164K multimodal queries, all collected under approved licenses.We evaluate leading multimodal retrieval models on this benchmark to explore the challenges specific to different medical specialities and to understand their impact on retrieval performance.By releasing M3Retrieve, we aim to enable systematic evaluation, foster model innovation, and accelerate research toward building more capable and reliable multimodal retrieval systems for medical applications.

Poetry is an expressive form of art that invites multiple interpretations, as readers often bring their own emotions, experiences, and cultural backgrounds into their understanding of a poem.Recognizing this, we aim to generate images for poems and improve these images in a zero-shot setting, enabling audiences to modify images as per their requirements.To achieve this, we introduce a novel Weighted Prompt Manipulation (WPM) technique, which systematically modifies attention weights and text embeddings within diffusion models.By dynamically adjusting the importance of specific words, WPM enhances or suppresses their influence in the final generated image, leading to semantically richer and more contextually accurate visualizations.Our approach exploits diffusion models and large language models (LLMs) such as GPT in conjunction with existing poetry datasets, ensuring a comprehensive and structured methodology for improved image generation in the literary domain.To the best of our knowledge, this is the first attempt at integrating weighted prompt manipulation for enhancing imagery in poetic language.Resources related to data and codes are available here: DIY

Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka, Nemil Shah, Abhilekh Borah, Vanshika Shah, Nishant Mishra, Sriparna Saha. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Punit Kumar Singh, Nishant Kumar, Akash Ghosh, Kunal Pasad, Khushi Soni, Manisha Jaishwal, Sriparna Saha, Syukron Abu Ishaq Alfarozi, Asres Temam Abagissa, Kitsuchart Pasupa, Haiqin Yang, Jose G Moreno. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Recent advancements in Multimodal Large Language Models (MLLMs), coupled with the progress of reinforcement learning, have substantially enhanced reasoning and decision-making across modalities, including text, vision, audio, and video. This tutorial introduces the fundamental principles, methodologies, and practical applications of MLLM reasoning, with a particular emphasis on strengthening reasoning capabilities in multilingual and cross-domain settings. We further discuss the key challenges and limitations of current multimodal reasoning approaches, as well as future directions for advancing the field. By highlighting how MLLMs support enhanced reasoning and planning in cross-lingual and cross-domain contexts, this session aims to equip researchers and practitioners with the conceptual foundations and practical tools needed to effectively integrate MLLM reasoning into their work.

The dynamic propagation of social media has broadened the reach of financial advisory content through podcast videos, yet extracting insights from lengthy, multimodal segments (30-40 minutes) remains challenging. We introduce FASTER(Financial Advisory Summariser with Textual Embedded Relevant images), a modular framework that tackles three key challenges: (1) extracting modality-specific features, (2) producing optimized, concise summaries, and (3) aligning visual keyframes with associated textual points. FASTER employs BLIP-2 for semantic visual descriptions, OCR for textual patterns, and Whisper-based transcription with Speaker diarization as BOS features. A modified Direct Preference Optimization (DPO)-based loss function, equipped with BOS-specific fact-checking, ensures precision, relevance, and factual consistency against the human-aligned summary. A ranker-based retrieval mechanism further aligns keyframes with summarized content, enhancing interpretability and cross-modal coherence. To acknowledge data resource scarcity, we introduce Fin-APT, a dataset comprising 470 publicly accessible financial advisory pep-talk videos for robust multimodal research. Comprehensive cross-domain experiments confirm FASTER's strong performance, robustness, and generalizability when compared to Large Language Models (LLMs) and Vision-Language Models (VLMs). By establishing a new standard for multimodal summarization, FASTER makes financial advisory content more accessible and actionable, thereby opening new avenues for research.

Maintaining accurate and up-to-date information is a persistent challenge for large-scale knowledge repositories, where outdated content can compromise their value. In this paper, we present a Multitask learning framework that uses Large Language Models (LLMs) for two tasks: semantic update detection and semantic update necessity prediction. The update detection task identifies obsoleteness by comparing older and newer text versions, while the update necessity prediction task determines whether an update is required based on a given context. To support these tasks, we curate a specialized dataset from Wikipedia called SEMUPDATES, focusing on frequently updated articles. Our experiments with five LLMs across four datasets in zero-shot, few-shot, and fine-tuned settings demonstrate that fine-tuning significantly enhances performance. In the multitask learning setup, Qwen delivers the best overall performance, while Mistral achieves the highest accuracy on individual tasks when fine-tuned separately. However, the performance differences across models are not substantial, suggesting that multiple LLMs can be effectively adapted for content update automation. These findings highlight the potential of LLMs in detecting and predicting obsolescence, providing a scalable solution for maintaining the timeliness of digital knowledge repositories.

In today's competitive marketing landscape, effective complaint management is crucial for customer service and business success. Video complaints, integrating text and image content, offer invaluable insights by addressing customer grievances and delineating product benefits and drawbacks. However, comprehending nuanced complaint aspects within vast daily multimodal financial data remains a formidable challenge. Addressing this gap, we have curated a proprietary multimodal video complaint dataset comprising 433 publicly accessible instances. Each instance is meticulously annotated at the utterance level, encompassing five distinct categories of financial aspects and their associated complaint labels. To support this en-deavour, we introduce Solution 3.0, a model designed for multimodal aspect-based complaint identification task. So-lution 3.0 is tailored to perform three key tasks: 1) handling multimodal features (audio and video), 2) facilitating mul-tilabel aspect classification, and 3) conducting multitasking for aspect classifications and complaint identification parallelly. Solution 3.0 utilizes a CLIP-based dual frozen encoder with an integrated image segment encoder for global feature fusion, enhanced by contextual attention (ISEC) to improve accuracy and efficiency. Our proposed framework surpasses current multimodal baselines, exhibiting superior performance across nearly all metrics by opening new ways to strengthen appropriate customer care initiatives and effectively assisting individuals in resolving their problems.

In light of the escalating challenges posed by our modern lifestyle, cultivating a nuanced approach for addressing mental health issues becomes imperative. Navigating the complexities of daily life demands a thoughtful and comprehensive strategy to identify and address the diverse array of mental health issues that may manifest. The challenges in accurately identifying mental health expressions stem from their complex character of communication, which frequently shares linguistic patterns and similar expressive nuances as communicated by humans. However, we hypothesize that mental health conditions are closely associated with affective factors in particular feelings, moods, and emotions. These states define how we think, feel, and behave. Thus, in this article, we aim to explore and analyze the association of the affective states such as sentiment and emotion with mental health in the view of identifying mental health conditions accurately once the feelings and emotions of humans are understood. In this regard, this article investigates multi-task classification encompassing mental health disorder identification (MHDI), emotion recognition (ER), and sentiment analysis (SA) in non-clinical conversations where MHDI forms the primary task and ER-SA the auxiliary tasks boosting the identification of the primary one. To demonstrate our hypothesis, we propose Core Fusion Network (CFN), a variation of multi-tasking in light of the significance that sentiment and emotion plays in understanding mental health. This method adeptly considers private and shared features across tasks, significantly enhancing classification precision. For our study, we extend the recently released MotiVAte dataset containing dyadic conversations between support seekers and a virtual assistant imparting hope and motivation to enclose emotion and sentiment tags for each conversation in a semi-supervised manner. Our hypothesis is reinforced by an extensive ablation study with state-of-the-art multi-task models and the proposed Core Fusion Network (CFN), which exhibits increased accuracy of 89.12% for MHDI, 64.24% for ER, and 79.04% for SA in the tri-task variant as opposed to its corresponding uni-task and bi-task variants. These outcomes underscore the potential of multi-task learning in streamlining mental health classification by integrating emotional and sentiment dimensions.

This article proposes a meta-learning-based, few-shot image captioning framework based on an ensemble of object-relational transformer models and a self-distillation strategy. Unlike traditional approaches, the proposed framework can train models using non-paired images and captions. In each iteration, multiple base models are trained on distinct data samples, forming an ensemble that generates pseudo-captions with confidence-based weighting. An efficient pseudo-feature generation method based on gradient descent enables learning from non-paired captions. These pseudo-features and pseudo-captions are then used to train the base models in subsequent iterations. The obtained results show that using just 1% of the paired image-caption training data significantly improves performance and generates meaningful captions. The proposed framework demonstrates the potential to reduce reliance on large, paired datasets while maintaining high-quality image captioning performance. 1

Continual learning, alternatively referred to as incremental learning or lifelong learning, represents a learning paradigm enabling an agent to acquire new information without compromising its retention of previously learned knowledge. Continual learning research has resulted in a number of ways to prevent catastrophic forgetting in deep neural networks. Apparently, continuous learning of recurrent models has never been used to solve the image captioning problems in the Hindi language. This research presents a novel approach to employ continual learning in the context of generating image captions in Hindi. Given its widespread usage in South Asia and its official status in India, Hindi ranks as the third most spoken language worldwide. We look at the continual learning of Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) based models for image captioning in a systematic way. In continuous image captioning problems, we incorporate an attention-based method that explicitly addresses the transient aspect of vocabulary. Using MS-COCO datasets, we employ our methods to solve the incremental image captioning problem. Our findings show that the proposed method can learn five captioning tasks in consecutive order without forgetting the one it has already learned.

With the increasing availability of multimodal movie data, there is a growing interest in leveraging these data to improve movie recommendations. In the recent era, due to the increase in the number of users and movies on OTT platforms such as Amazon Prime, its services, including personalized movie recommendations, become challenging. This article proposes a novel approach$M^{2}RM^{2}UL$, which stands for multimodal movie recommendation with multitasking and movie–user learning. The initial phase involves preprocessing multimodal movie data to extract features encompassing visual and textual information. Subsequently, a multitasking architecture is employed, simultaneously undertaking classification and regression tasks to acquire user and movie representations. The learned representations are used to make personalized and accurate movie recommendations. Additionally, the Netflix Prize dataset has been augmented to include textual and visual features, rendering it multimodal. We conducted extensive experiments on three real-world multimodal movie datasets (Movielens-100K, MMTF-14K, and Netflix Prize) and compared our approach with several state-of-the-art movie recommendation algorithms. The experimental results illustrate that our approach outperforms the baseline methods in terms of recommendation accuracy and diversity. Furthermore, we demonstrate the effectiveness of our approach in different scenarios, such as cold-start and sparse data. Our empirical study provides strong evidence for the effectiveness of the proposed approach in multimodal movie recommendation.

In the era of modern healthcare, swiftly generating medical question summaries is crucial for informed and timely patient care. Despite the increasing complexity and volume of medical data, existing studies have focused solely on text-based summarization, neglecting the integration of visual information. Recognizing the untapped potential of combining textual queries with visual representations of medical conditions, we introduce the Multimodal Medical Question Summarization (MMQS) Dataset. This dataset, a major contribution of our work, pairs medical queries with visual aids, facilitating a richer and more nuanced understanding of patient needs. We also propose a framework, utilizing the power of Contrastive Language Image Pretraining(CLIP) and Large Language Models(LLMs), consisting of four modules that identify medical disorders, generate relevant context, filter medical concepts, and craft visually aware summaries. Our comprehensive framework harnesses the power of CLIP, a multimodal foundation model, and various general-purpose LLMs, comprising four main modules: the medical disorder identification module, the relevant context generation module, the context filtration module for distilling relevant medical concepts and knowledge, and finally, a general-purpose LLM to generate visually aware medical question summaries. Leveraging our MMQS dataset, we showcase how visual cues from images enhance the generation of medically nuanced summaries. This multimodal approach not only enhances the decision-making process in healthcare but also fosters a more nuanced understanding of patient queries, laying the groundwork for future research in personalized and responsive medical care. Disclaimer: The article features graphic medical imagery, a result of the subject's inherent requirements.

The advancement of Artificial Intelligence is pivotal in reshaping healthcare, enhancing diagnostic precision, and facilitating personalized treatment strategies.One major challenge for healthcare professionals is quickly navigating through long clinical documents to provide timely and effective solutions.Doctors often struggle to draw quick conclusions from these extensive documents.To address this issue and save time for healthcare professionals, an effective summarization model is essential.Most current models assume the data is only textbased.However, patients often include images of their medical conditions in clinical documents.To effectively summarize these multimodal documents, we introduce EDI-Summ, an innovative Image-Guided Encoder-Decoder Model.This model uses modality-aware contextual attention on the encoder and an image cross-attention mechanism on the decoder, enhancing the BART base model to create detailed visual-guided summaries.We have tested our model extensively on three multimodal clinical benchmarks involving multimodal question and dialogue summarization tasks.Our analysis demonstrates that EDI-Summ outperforms state-of-the-art large language and vision-aware models in these summarization tasks.

Prince Jha, Raghav Jain, Konika Mandal, Aman Chadha, Sriparna Saha, Pushpak Bhattacharyya. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

While considerable research has delved into detecting toxic content in text-based data, the realm of video content, particularly in languages other than English, has received less attention. Prior studies have primarily focused on creating automated tools to identify online toxic speech but have often overlooked the crucial next steps of mitigating its impact and discouraging future use. We can discourage social media users from sharing such material by automatically generating interventions that explain why certain content is inappropriate. To bridge this research gap, we propose an innovative task: generating interventions for toxic videos in code-mixed languages which go beyond existing methods focusing on text and images to combat online toxicity. We are introducing a Toxic Code-Mixed Intervention Video benchmark dataset (ToxCMI), comprising 1697 code-mixed toxic video utterances sourced from YouTube. Each utterance in this dataset has been meticulously annotated for toxicity and severity, accompanied by interventions provided in Hindi-English code-mixed languages. We have developed an advanced multimodal framework ToxVI, specifically designed for the task of generating Toxic Video appropriate Interventions, leveraging Large Language Models (LLMs), which comprises three modules - Modality module, Cross-Modal Synchronization module and Generation module. Our experiments demonstrate that integrating multiple modalities from the videos significantly enhances the performance of the proposed task and outperforms all the baselines by a significant margin.

Over the last few years, artificial intelligence-based clinical assistance has gained immense popularity and demand in telemedicine, including automatic disease diagnosis. Patients often describe their signs and symptoms to doctors using visual aids, which provide vital evidence for identifying a medical condition. In addition to learning from our experiences, we learn from well-established theories/ knowledge. With the motivation of leveraging visual cues and medical knowledge, we propose a transformer-based, knowledge-infused multi-modal medical dialogue generation (KI-MMDG) framework. In addition, we present a discourse-aware image identifier (DII) that recognizes signs and their severity by leveraging the current conversation context in addition to the image of the signs. We first curate an empathy and severity-aware multi-modal medical dialogue (ES-MMD) corpus in English, which is annotated with intent, symptoms, and visual signs with severity information. Experimental results show the superior performance of the proposed KI-MMDG model over uni-modal and non-knowledge infused generative models, demonstrating the importance of visual signs and knowledge infusion in symptom investigation and diagnosis. We also observed that the DII model surpasses the existing state-of-the-art model by 7.84%, indicating the crucial significance of dialogue context for identifying a sign image surfaced during conversations. The code and dataset are available at https://github.com/NLP-RL/KI-MMDG.

Sarcasm primarily involves saying something but “meaning the opposite” or “meaning something completely different” in order to convey a particular tone or mood. In both the above cases, the “meaning” is reflected by the communicative intention of the speaker, known as dialogue acts. In this paper, we seek to investigate a novel phenomenon of analyzing sarcasm in the context of dialogue acts with the hypothesis that the latter helps to understand the former better. Toward this aim, we extend the multi-modal MUStARD dataset to enclose dialogue acts for each dialogue. To demonstrate the utility of our hypothesis, we develop a dialogue act-aided multi-modal transformer network for sarcasm identification (MM-SARDAC), leveraging interrelation between these tasks. In addition, we introduce an order-infused, multi-modal infusion mechanism into our proposed model, which allows for a more intuitive combined modality representation by selectively focusing on relevant modalities in an ordered manner. Extensive empirical results indicate that dialogue act-aided sarcasm identification achieved better performance compared to performing sarcasm identification alone. The dataset and code are available at https://github.com/mohit2b/MM-SARDAC.

Prince Jha, Krishanu Maity, Raghav Jain, Apoorv Verma, Sriparna Saha, Pushpak Bhattacharyya. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Timeline summarization is the task of automatically generating concise overviews of documents that capture the key events and their progression on timelines. While this capability is useful for quickly comprehending event sequences without reading lengthy descriptions, timeline summarization remains a relatively underexplored area in recent years when compared to traditional document summarization task and their evolution. The advent of large language models (LLMs) has led some to presume summarization as a solved problem. However, timeline summarization poses unique challenges for LLMs. Our investigation is centered on evaluating the performance of LLMs, against state-of-the-art models in this field. We employed three different approaches: chunking, knowledge graph-based summarization, and TimeRanker. Each of these methods was systematically tested on three benchmark datasets for timeline summarization to assess their effectiveness in capturing and condensing key events and their evolution within timelines. Our findings reveal that while LLMs show promise, timeline summarization remains a complex task that is not yet fully resolved.

Social media platforms like Twitter act as a medium for communication among people, government agencies, NGOs, and other relief providing agencies in widespread humanitarian havoc during a disaster outbreak when other communication means might not be available. Various agencies leverage twitter's open and public features to get timely and reliable updates, thus support agencies in communicating with the people on rescue and provide immediate relief. As situational updates are mixed in millions of other tweets, an efficient system is required to extract and summarize these tweets. We have developed a novel framework that uses a deep learning-based classification model to separate the informational tweets from others and summarizes them in the current paper. Non-situational tweets mostly comprise sentiments like grief, anger, sorrow, etc. Motivated by this observation, we have solved sentiment classification and informative tweet selection tasks simultaneously using a multi-task learning (MTL) in a deep-learning framework. Our summarization approach generates clustering solutions using various existing approaches and then ensembles cluster solutions using generative modelling. A summary is formulated by extracting tweets from different clusters. The proposed approach's superior performance on four disaster-related events indicates the developed framework's efficiency over state-of-the-art techniques.

In the financial service sector, customer service is the most critical tool for long-term business growth. A financial complaint detection (CD) system could aid in the identification of shortcomings in product features and service delivery. This could further ensure faster resolution of customer complaints and thereby help retain existing clients and attract new ones. Prior research has prioritized only complaint identification and prediction of the corresponding severity levels; the first aim is to categorize a textual element as a complaint or a noncompliant. The other attempts to classify complaints into several severity levels based on the degree of risk the complainant is willing to endure. Identifying the reason or source of a complaint in a text is a significant but underexplored area in natural language processing study. We propose an explainable complaint cause identification approach with a dyadic attention mechanism at the sentence and word levels, enabling it to give varying amounts of emphasis to more and less important information. As the first subtask, the model simultaneously trains CD, sentiment detection, and emotion recognition tasks. Afterwards, we identify the complaint’s cause and its severity level. To do this, the causal span annotations for complaint tweets are added to an existing financial complaints corpus. The findings suggest that conventional computing techniques can be adapted to solve extremely relevant new problems, generating novel opportunities for research1.

In our digitally connected world, the influx of microblog data poses a formidable challenge in extracting relevant information amid a continuous stream of updates. This challenge intensifies during crises, where the demand for timely and relevant information is crucial. Current summarization techniques often struggle with the intricacies of microblog data in such situations. To address this, our research explores crisis-related microblogs, recognizing the crucial role of multimedia content, such as images, in offering a comprehensive perspective. In response to these challenges, we introduce a multimodal extractive-abstractive summarization model. Leveraging a fusion of TF-IDF scoring and bigram filtering, coupled with the effectiveness of three distinct models—BIGBIRD, CLIP, and bootstrapping language-image pre-training (BLIP)—we aim to overcome the limitations of traditional extractive and text-only approaches. Our model is designed and evaluated on a newly curated Twitter dataset featuring 12 494 tweets and 3090 images across eight crisis events, each accompanied by gold-standard summaries. The experimental findings showcase the remarkable efficacy of our model, surpassing current benchmarks by a notable margin of 16% and 17%. This confirms our model's strength and its relevance in crisis scenarios with the crucial interplay of text and multimedia. Notably, our research contributes to multimodal, abstractive microblog summarization, addressing a key gap in the literature. It is also a valuable tool for swift information extraction in time-sensitive situations.

The escalating prevalence of online cyberbullying and trolling across various social media platforms has becomes a pressing concern. Extensive research demonstrates the detrimental impact of cyberbullying on the mental well-being of its victims. Given the sheer volume of online content, manual identification of cyberbullying instances proves unfeasible, necessitating the development of automated cyberbullying detection methods. This challenge has attracted considerable attention within the natural language processing (NLP) community, owing to advancements in machine learning techniques. However, most of the methods fail to provide reasoning for their decisions which warrants the use of interpretable models that can explain the model’s output in real-time. Interpretable models rather than black-box models with high performance are becoming popular adhering the “right to explanations” laws. Motivated by this, we create a cyberbullying corpusBullyExplainin code-mixed language, where a post has been annotated with four labels, i.e., bully, sentiment, target, and rationales (explainability). Current work addresses the task of explainable cyberbully detection and proposes a unified generative framework,BullyGenby redefining this multitask problem as a text-to-text generation task. Our framework is capable of not only detecting whether the text is a cyberbully or not but also provides reasoning by predicting rationale, target group and sentiment of the text. Experimental results illustrate the efficacy of our proposed model by outperforming the state-of-the-art and several baselines by a significant margin and conclude that text-to-text generation model could be a good alternative for multitask classification problems.

Social media platforms are a double-edged sword: on the one hand, they enable the dissemination of information; but on the other hand, they also provide an avenue for spreading online abuse and harassment, such as hate speech. While significant research efforts are being devoted to detecting online hate speech in the English language, little attention has been paid to the Thai language. In this study, we created a benchmark dataset, calledHateThaiSent, which labels each post with both hate speech and sentiment information. To detect hate speech, we created a multitask model that uses a dual-channel deep learning approach based on FastText and BERT embeddings, with an added capsule network. One channel utilizes pretrained FastText embeddings while the other uses embeddings from the BERT language model. We aimed to answer two research questions: (Q1) Does incorporating sentiment information improves the performance of hate speech detection (HD) in the Thai language? (Q2) What is the comparative effectiveness of two different approaches for sentiment-aware HD in the Thai language: feature engineering versus multitasking? Our proposed approach outperformed other baselines and state-of-the-art models on theHateThaiSentdataset, with overall accuracy/macro-F1 values of 89.67%/89.79%, and 80.92%/80.97% for hate speech and sentiment detection tasks, respectively. We concluded that multitasking is more effective than feature engineering in enhancing the performance of the main task (HD).

Cyberbullying is a malady of social media, and its automatic detection is critically important considering its virulence, velocity of spreading, and the scale of the havoc it can wreak. However, the problem is challenging due to its disguised behavior, noise in the content, and, in recent times, introduction of code-mixing. In this work, we propose MTBullyGNN a novel graph neural network (GNN)-based multitask (MT) framework that solves sentiment-aided cyberbullying detection (CD) from code-mixed language. The GNN helps detect unlabelled or noisy label nodes (sentences) accurately by aggregating information from similarly labeled nodes. To connect nodes, we apply cosine similarity between sentences and create a single text graph for a benchmark code-mixed cyberbullying corpus, BullySent. Experimental results illustrate that MTBullyGNN outperforms the state-of-the-art (SOTA) methods for both the single (CD) and MT (CD and sentiment) settings by up to 4.46% and 4.92% in classification accuracy, respectively. Furthermore, another benchmark Hindi–English code-mixed single-task dataset has also been considered to illustrate the robustness of our proposed model. The code will be made publicly available in the camera-ready version.

Recent years have witnessed tremendous growth in the publication of research articles as well as in the rise of new research topics. Articles get published in a streaming manner and therefore retrieving and recommending trending topics continuously by updating the trend of topics with time will be beneficial for young researchers. The proposed topic recommendation system is a clustering-based approach that utilizes an autoencoder framework for the generation of clusters. The autoencoder framework considers articles as input in a multiview framework and produces latent data in lower-dimensional space using graph attention-based encoder and decoder networks. The latent data is then partitioned with respect to its different views and finally, a single consensus overlapping partitioning is produced by satisfying all the views. Since the publication of articles is a continuous process, to update the trending topics continuously, the clustering-based approach is kept on and applied iteratively in a sliding window manner. The generated clusters are analyzed to extract the trending topics and recommendations for future scope. An article can belong to multiple topics and the proposed method is developed by considering such criteria and therefore tested with the modified version of the multilabel scientific article data ArXiv, named CSML.ArXiv. The superiority of the proposed method can be observed from its comparisons with existing methods, and a few baseline methods with respect to cluster formation, topic extraction, and trending topic evaluation.

During any natural disaster or unfortunate accident, both civilians and responders need information on an urgent basis. In such events, microblogging sites particularly Twitter plays an important role in providing real-time information. The raw form of microblog tweets is prodigiously informative but massive in size. The end-users and data analysts have to go through millions of tweets before extraction of any information. To ease the process and extract only relevant information, artificial intelligence (AI)-based techniques can be incorporated to generate summaries from the incoming information. Moreover, tweets keep on arriving continuously in a streaming manner, and therefore in ideal cases, the summaries also need to be updated continuously. In this work, we have proposed a clustering-based summary generation approach that takes multiviewed representations of data and utilizes a new variant of generative adversarial network (GAN) named triple-GAN to perform clustering. Triple-GAN consists of three networks, a generator, a discriminator, and a separator. Maintaining equilibrium among these networks requires proper parameter tuning which makes training of GAN difficult. In the literature, GAN-based techniques have been extensively applied to image datasets. In the proposed method, we have explored the usage of GAN for text data in an unsupervised manner and the analysis of the training of GAN has also been reported. The developed method opens up a new direction in utilizing GAN for solving clustering problem of text data. The proposed method is applied to two versions of four disaster-based microblog datasets and obtained results are compared with many existing and a few baseline methods. The comparative study illustrates the superiority and efficacy of the developed method.

The automatic detection of financial complaints (FINCORP) can benefit businesses and online merchants. Compared with manually tagged complaints, they can use this information to monitor and address issues and effectively route them to appropriate teams. This can also promote greater transparency and accountability when dealing with consumer financial products and services, strengthening the firm’s brand value. In linguistic studies, complaints have been classified into severity categories based on the level of risk the complainant is prepared to accept. Furthermore, since emotions influence every speech act, an individual’s emotional state considerably impacts the complaint expression. In this article, we introduce a FINCORP resource, a collection of annotated complaints arising between financial institutions and consumers expressed in English on Twitter. The dataset has been enriched with the associated emotion, sentiment, and complaint severity classes. The dataset comprises 3149 complaint and 3133 noncomplaint instances spanning over ten domains (e.g., credit cards, mortgages). For a comprehensive evaluation of our dataset, we develop a multitask framework for complaint detection and severity classification with emotion recognition (ER) and sentiment classification as the additional tasks and compare it with several existing baselines. The corpus and code are available here:https://github.com/RohanBh23/FINCORP.

The prevalence of complaints submitted online and the sheer volume of information made available by social media platforms highlight the need for automated complaint analysis tools. In linguistic studies, complaints have been classified according to how much of personal risk the complainant is willing to take. This is crucial information for understanding the motivations of complainants and how people come up with reasonable means of reparation. Few attempts have been made to use the existing multimodal complaint model, which focuses on improving the textual mode with the help of images, to find specific visual features that help identify complaints. Our aim is to find a solution to this problem. In order to detect complaints and the severity level associated with them in a multitask setting, we propose Multimodal framEwork for Complaint and Severity-level detectIon (MECSI), a novel multimodal framework that uses local and global attributes (in both modalities) and relates them to the textual context. To do this, we add severity-level annotation to the newly released CESAMARD dataset, which is a compilation of reviews and images of products listed on the Amazon website. The experimental findings confirmed the superiority of our proposed model over the state-of-the-art model and other strong rival baselines, proving the efficacy of our proposed framework.

In breast cancer treatment, accurately predicting how long a patient will survive is crucial for decision-making. This information guides treatment choices and supports patients' psychological recovery. To address this challenge, we introduce a novel predictive model to forecast breast cancer prognosis by leveraging diverse data sources, including clinical records, copy number variation, gene expressions, DNA methylation, microRNA (miRSeq) sequencing, and whole slide image data from the TCGA Database. The methodology incorporates graph contrastive learning with cross-modality attention (CAGCL), considering all possible combinations of the six distinct data modalities. Feature embeddings are enhanced through graph contrastive learning, which identifies subtle differences and similarities among samples. Further, to learn the complementary nature of information across multiple data modalities, a cross-attention framework is proposed and applied to the graph contrastive learning-based extracted features from various data sources for breast cancer survival prediction. It performs a binary classification to anticipate the likelihood of short- and long-term breast cancer survivors, delineated by a five-year threshold. The proposed model (CAGCL) showcases superior performance compared to baseline models and other state-of-the-art models. The model attains an accuracy of 0.932, a sensitivity of 0.954, a precision of 0.958, an F1 score of 0.956, and an AUC of 0.948, underscoring its effectiveness in predicting breast cancer survival.

Complaining is an illocutionary act in which the speaker communicates his/her dissatisfaction with a set of circumstances and holds the hearer (the complainee) answerable, directly or indirectly.Considering breakthroughs in machine learning approaches, the complaint detection task has piqued the interest of the natural language processing (NLP) community.Most of the earlier studies failed to justify their findings, necessitating the adoption of interpretable models that can explain the model's output in real-time.We introduce an explainable complaint dataset, X-CI, the first benchmark dataset for explainable complaint detection.Each instance in the X-CI dataset is annotated with five labels: complaint label, emotion label, polarity label, complaint severity level, and rationale (explainability), i.e., the causal span explaining the reason for the complaint/noncomplaint label.We address the task of explainable complaint detection and propose a commonsense-aware unified generative framework by reframing the multitask problem as a text-to-text generation task.Our framework can predict the complaint cause, severity level, emotion, and polarity of the text in addition to detecting whether it is a complaint or not.We further establish the advantages of our proposed model on various evaluation metrics over the state-of-the-art models and other baselines when applied to the X-CI dataset in both full and few-shot settings 1 .

In the modern era, the rapid expansion of social media and the proliferation of the internet community has led to a multi-fold increase in the richness and range of views and outlooks expressed by readers and viewers. To obtain valuable insights from this vast sea of opinions, we present an inventive and holistic procedure for multi-modal abstractive summarization with comment sensitivity. Our proposed model utilizes both textual and visual modalities and examines the remarks provided by the readers to produce summaries that apprehend the significant points and opinions made by them. Our model features a transformer-based encoder that seamlessly processes both news articles and comments, merging them before transmitting the amalgamated information to the decoder. Additionally, the core segment of our architecture consists of an attention-based merging technique which is trained adversarially by means of a generator and discriminator to bridge the semantic gap between comments and articles. We have used a Bi-LSTM-based branch for image pointer generation. We assess our model on the reader-aware multi-document summarization (RA-MDS) dataset which contains news articles, their summaries, and related comments. We have extended the dataset by adding images pertaining to news articles in the corpus to increase the richness and diversity of the dataset. Our comprehensive experiments reveal that our model outperforms similar pre-trained models and baselines across two of the four evaluated metrics, showcasing its superior performance.

Large language models (LLMs), when scaled from millions to billions of parameters, have been demonstrated to exhibit the so-called 'emergence' effect, in that they are not only able to produce semantically correct and coherent text, but are also able to adapt themselves surprisingly well with small changes in contexts supplied as inputs (commonly called prompts). Despite producing semantically coherent and potentially relevant text for a given context, LLMs are vulnerable to yield incorrect information. This misinformation generation, or the so-called hallucination problem of an LLM, gets worse when an adversary manipulates the prompts to their own advantage, e.g., generating false propaganda to disrupt communal harmony, generating false information to trap consumers with target consumables etc. Not only does the consumption of an LLM-generated hallucinated content by humans pose societal threats, such misinformation, when used as prompts, may lead to detrimental effects for in-context learning (also known as few-shot prompt learning). With reference to the above-mentioned problems of LLM usage, we argue that it is necessary to foster research on topics related to not only identifying misinformation from LLM-generated content, but also to mitigate the propagation effects of this generated misinformation on downstream predictive tasks thus leading to more robust and effective leveraging in-context learning.

Automated complaint analysis is vital for generating critical insights, which in turn enhance customer satisfaction, product quality, and overall business performance. Nevertheless, conventional methods frequently fail to capture the nuances of aspect-level complaints and inadequately utilize external knowledge, thus creating a gap in effective complaint detection and analysis. In response to this issue, we proactively explore the role of external knowledge and multimodality in this domain. This leads to the development of MGasD (Multimodal Generative framework for aspect-based complaint and sentiment Detection), a multimodal knowledge-infused unified framework. MGasD diverges from traditional methods by reframing the complaint detection problem as a multimodal text-to-text generation task. Significantly, our research includes the development of a novel aspect-level dataset. Annotated for both complaint and sentiment categories across diverse domains such as books, electronics, edibles, fashion, and miscellaneous, this dataset provides a comprehensive platform for the concurrent study of complaints and sentiment. This resource facilitates a more robust understanding of consumer feedback. Our proposed methodology establishes a benchmark performance in the novel aspect-based complaint and sentiment detection tasks based on extensive evaluation. We also demonstrate that our model consistently outperforms all other baselines and state-of-the-art models in both full and few-shot settings (The dataset and code are available at:https://github.com/appy1608/CIKM2023).

With the advancement of telemedicine, both researchers and medical practitioners are working hand-in-hand to develop various techniques to automate various medical operations, such as diagnosis report generation. In this paper, we first present a multi-modal clinical conversation summary generation task that takes a clinician-patient interaction (both textual and visual information) and generates a succinct synopsis of the conversation. We propose a knowledge-infused, multi-modal, multi-tasking medical domain identification and clinical conversation summary generation (MM-CliConSummation) framework. It leverages an adapter to infuse knowledge and visual features and unify the fused feature vector using a gated mechanism. Furthermore, we developed a multi-modal, multi-intent clinical conversation summarization corpus annotated with intent, symptom, and summary. The extensive set of experiments, both quantitatively and qualitatively, led to the following findings: (a) critical significance of visuals, (b) more precise and medical entity preserving summary with additional knowledge infusion, and (c) a correlation between medical department identification and clinical synopsis generation. Furthermore, the dataset and source code are available at https://github.com/NLP-RL/MM-CliConSummation

Significant developments in techniques such as encoder-decoder models have enabled us to represent information comprising multiple modalities.This information can further enhance many downstream tasks in the field of information retrieval and natural language processing; however, improvements in multi-modal techniques and their performance evaluation require large-scale multi-modal data which offers sufficient diversity.Multi-lingual modeling for a variety of tasks like multi-modal summarization, text generation, and translation leverages information derived from highquality multi-lingual annotated data.In this work, we present the current largest multilingual multi-modal summarization dataset (M3LS), and it consists of over a million instances of document-image pairs along with a professionally annotated multi-modal summary for each pair.It is derived from news articles published by British Broadcasting Corporation(BBC) over a decade and spans 20 languages, targeting diversity across five language roots, it is also the largest summarization dataset for 13 languages and consists of cross-lingual summarization data for 2 languages.We formally define the multi-lingual multi-modal summarization task utilizing our dataset and report baseline scores from various state-of-the-art summarization techniques in a multi-lingual setting.We also compare it with many similar datasets to analyze the uniqueness and difficulty of M3LS.

Advancements in Conversational Natural Language Processing (NLP) have the potential to address critical social challenges, particularly in achieving the United Nations’ Sustainable Development Goal of quality education. However, the application of NLP in the educational domain, especially language learning, has been limited due to the inherent complexities of the field and the scarcity of available datasets. In this paper, we introduce T-VAKS (Tutoring Virtual Agent with Knowledge Selection), a novel language tutoring multimodal Virtual Agent (VA) designed to assist students in learning a new language, thereby promoting AI for Social Good. T-VAKS aims to bridge the gap between NLP and the educational domain, enabling more effective language tutoring through intelligent virtual agents. Our approach employs an information theory-based knowledge selection module built on top of a multimodal seq2seq generative model, facilitating the generation of appropriate, informative, and contextually relevant tutor responses. The knowledge selection module in turn consists of two sub-modules: (i) knowledge relevance estimation, and (ii) knowledge focusing framework. We evaluate the performance of our proposed end-to-end dialog system against various baseline models and the most recent state-of-the-art models, using multiple evaluation metrics. The results demonstrate that T-VAKS outperforms competing models, highlighting the potential of our approach in enhancing language learning through the use of conversational NLP and virtual agents, ultimately contributing to addressing social challenges and promoting well-being.

Medical Question Answering (MedQA) is one of the most popular and significant tasks in developing healthcare assistants. When humans extract an answer to a question from a document, they first (a) understand the question itself in detail and (b) utilize relevant knowledge/experiences to determine the answer segments. In multi-span question answering, it becomes increasingly important to comprehend the query accurately and possess relevant knowledge, as the interrelationship among different answer segments is essential for achieving completeness. Motivated by this, we first propose a transformer-based query semantic and knowledge (QueSemKnow) guided multi-span question-answering model. The proposed QueSemKnow works in a two-phased manner; in the first stage, a multi-task model is proposed to extract query semantics: (i) intent identification and (ii) question type prediction. In the second stage, QueSemKnow selects a relevant subset of the knowledge graph as the underlying context/document and extracts answers depending on the semantic information extracted from the first stage and context. We build a multi-task query semantic extraction model for query intent and query type identification to investigate the co-relation among these tasks. Furthermore, we created a semantically aware medical question-answering corpus named QueSeMSpan MedQA wherein each question is annotated with its corresponding semantic information. The proposed model outperforms several baselines and existing state-of-the-art models by a large margin on multiple datasets, which firmly demonstrates the effectiveness of the human-inspired multi-span question-answering methodology.

Temporal reasoning represents a vital component of human communication and understanding, yet remains an underexplored area within the context of Large Language Models (LLMs).Despite LLMs demonstrating significant proficiency in a range of tasks, a comprehensive, large-scale analysis of their temporal reasoning capabilities is missing.Our paper addresses this gap, presenting the first extensive benchmarking of LLMs on temporal reasoning tasks.We critically evaluate 8 different LLMs across 6 datasets using 3 distinct prompting strategies.Additionally, we broaden the scope of our evaluation by including in our analysis 2 Code Generation LMs.Beyond broad benchmarking of models and prompts, we also conduct a finegrained investigation of performance across different categories of temporal tasks.We further analyze the LLMs on varying temporal aspects, offering insights into their proficiency in understanding and predicting the continuity, sequence, and progression of events over time.Our findings reveal a nuanced depiction of the capabilities and limitations of the models within temporal reasoning, offering a comprehensive reference for future research in this pivotal domain.

With the rise of social media and online communication, the issue of cyberbullying has gained significant prominence.While extensive research is being conducted to develop more effective models for detecting cyberbullying in monolingual languages, a significant gap exists in understanding code-mixed languages and the need for explainability in this context.To address this gap, we have introduced a novel benchmark dataset named Bully-Explain for explainable cyberbullying detection in code-mixed language.In this dataset, each post is meticulously annotated with four labels: bully, sentiment, target, and rationales, indicating the specific phrases responsible for identifying the post as a bully.Our current research presents an innovative unified generative framework, GenEx, which reimagines the multitask problem as a text-to-text generation task.Our proposed approach demonstrates its superiority across various evaluation metrics when applied to the BullyExplain dataset, surpassing other baseline models and current state-of-the-art approaches. 1

Automatic detection of consumers' complaints about items or services they buy can be critical for organizations and online merchants.Previous studies on complaint identification are limited to text.Images along with the reviews can provide cues to identify complaints better, thus emphasizing the importance of incorporating multi-modal inputs into the process.Generally, the customer's emotional state significantly impacts the complaint expression; thus, the effect of emotion and sentiment on complaint identification must also be investigated.Furthermore, different organizations are usually not allowed to share their privacy-sensitive records due to data security and privacy concerns.Due to these issues, traditional models find it hard to understand and identify complaint patterns, particularly in the financial and healthcare sectors.In this work, we extend a benchmark multi-modal complaint dataset, a collection of reviews and images of the products posted on the e-commerce website Amazon.We propose a federated meta-learning-based multimodal multi-task framework for identifying complaints considering emotion recognition and sentiment analysis as two auxiliary tasks.Experimental results indicate that the proposed approach outperforms the baselines and the state-of-the-art approaches in centralized and federated meta-learning settings 1 .

Aritra Raut, Sriparna Saha, Anutosh Maitra, Roshni Ramnani. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Apoorva Singh, Raghav Jain, Sriparna Saha. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Financial industries rely on a variety of data to understand an organization's financial health and performance, and analysis of financial statements is used for decision-making and understanding business activities. Financial statements often have complex, unstructured formats, making it difficult to extract useful information for decision-making. Organizing and refining these data is crucial for effective analysis. Previous research in the field of table detection has primarily centred around object detection methods, with limited exploration of methods for cell-wise information extraction by identifying row and column entities. In order to address this research gap, this paper proposes a novel dataset called TERED (Tabular Entity Relationship Establishment Dataset) to train a model to identify relationships among elements in large financial tables, such as statements and balance sheets, using computer vision techniques which have yet to be fully explored in financial analysis. The dataset contains more than 10,000 tabular data in scanned image and pdf formats and is divided into 12 classes. We trained CLF-RCNN, a Contrastive Learning based Faster RCNN model which in turn is a state-of-the-art object detection model on this dataset and achieved an F1 score of 93 % for table detection and 74% for identifying tabular entities and relationships. Additionally, we introduced a new loss term influenced by contrastive learning that improves prediction performances with our developed algorithm to return the sequential order of the unordered predictions.

Accessibility to the internet has led to a massive increase in the usage of social networking and online communication apps over the past decade. With so much content available online, these platforms suffer from governance and monitoring problems making the users susceptible to online cyberbullying and trolling. Recently, this trolling and hate comes majorly from memes that combine text and image modalities. Many studies show how cyberbullying can harm the mental well-being of the affected individuals. Previous studies also tried to study the role of sentiment, emotions, and sarcasm in identifying hateful memes setting up the meme detection task as a multi-modal, multitask problem. In the past, meme detection models were discriminative, but more recently generative models have been used to solve non-generation tasks such as aspect-based sentiment analysis and span detection. Motivated by this, in this work, we propose a unified Multimodal Generative framework, MGex by reframing the multitasking problem of detecting cyberbullying, sentiment, emotion, and sarcasm as a multimodal text-to-text generation problem. For this purpose, we use MultiBully dataset which provides annotation for all these labels. Here, we evaluate and contrast our proposed generative framework with several multitasking baselines and state-of-the-art models. Disclaimer: The article contains offensive text and profanity. This is owing to the nature of the work and does not reflect any opinion or stand of the authors.

Due to the growth in demand for brief and pertinent multimedia material over the past few years, multimodal summarization has attracted a lot of study interest. Recently Transformers have been widely used for various sequence processing tasks due to their fast parallel processing ability compared to LSTMs. Although Multimodal Summarization (MS) has tractioned much research interest of late, a research gap exists in producing topic-relevant multimodal summaries. Since any summary deals with concise information, it should carry the essence of the topic from which it was derived. Further, due to the lack of alignment information among the images and the inter-modal segments, MS systems also face difficulty choosing appropriate pictorial summaries. To study these research questions, we propose a Multitask learning-based Multimodal Pointer Generator Transformer (MPGT), which utilizes the topic information of the samples to produce multimodal summaries. We also augment the popular MSMO dataset for this study with similar “On-Topic” and “Off-Topic” images. Our results show that inter-modal attention among images helps achieve better alignment in the visual modality and improves image precision scores. Our analysis also provides discussions on how we can further enhance topic-relevant MS systems.

An online recommendation system is one of the desires of digital e-commerce sectors and the OTT platforms like Amazon Prime, Netflix, SonyLiv, etc. In recent times, with an increase in the interaction of users with the different e-commerce platforms and then analyzing their liking-disliking essence, the recommendation system tries to predict the preference of the user for recommending new items that may capture his attention. In the current study, a multi-task-based architecture is designed to solve the multi-modal movie recommendation problem. Here our hypothesis is that solving two related tasks, namely (a) genre classification of movies and (b) rating identification for a user-movie pair, helps in generating good quality movie embeddings in an end-to-end setting without using a rating vector. For generating the representation of movies, unlike the state-of-the-art techniques, feature vectors extracted from multiple modalities like textual summary, audio and video information present in the movie trailers, and meta-data information are fused together. For representing the user, average representations of movies that are liked by the user are considered. Different multitasking models, fully shared (FS), shared-private (SP), and adversarial shared-private (ASP) feature models are developed for solving the above-mentioned two tasks simultaneously, genre classification, and user-movie rating prediction. For experimental purposes, MMTF-14K: a multifaceted movie trailer feature dataset was extended by incorporating textual features and meta-data information, and a multi-modal version of the MovieLens-100K dataset is used. Results of different multitasking models are shown in terms of RMSE and different rank-based metrics. The proposed multi-task model along with the adversarial training outperforms the state-of-the-art models when applied to the MMTF-14K and multi-modal version of MovieLens-100K datasets.

Valuable feedback can be found in customer reviews about a product or service, but it can be difficult to identify specific complaints and their underlying causes from the text. While various methods have been used to detect and analyze complaints, there has been little research on examining complaints at the aspect level and the reasons behind these complaints. To address this, we added rationale annotation for aspect-based complaint classes to a publicly available benchmark multimodal complaint dataset (CESAMARD) covering five domains (books, electronics, edibles, fashion, and miscellaneous). Current methods treat these tasks as classification problems and do not use external knowledge. Our study proposes a knowledge-infused, hierarchical, multimodal generative approach for aspect-based complaint and rationale detection that reframes the multitasking problem as a multimodal text-to-text generation task. Our approach achieved benchmark performance in the aspect-based complaint and rationale detection task through an extensive evaluation. We demonstrated that our model consistently outperformed all other baselines and state-of-the-art models in both full and few-shot settings. Our study contributes to the development of more accurate and efficient methods for extracting valuable insights from customer reviews to improve products and service 1

Figurative language is an essential component of human communication, and detecting sarcasm in text has become a challenging yet highly popular task in natural language processing. As humans, we rely on a combination of visual and auditory cues, such as facial expressions and tone of voice, to comprehend a message. Our brains are implicitly trained to integrate information from multiple senses to form a complete understanding of the message being conveyed, a process known as multi-sensory integration. The combination of different modalities not only provides additional information but also amplifies the information conveyed by each modality in relation to the others. Thus, the infusion order of different modalities also plays a significant role in multimodal processing. In this paper, we investigate the impact of different modality infusion orders for identifying sarcasm in dialogues. We propose a modality order-driven module integrated into a transformer network, MO-Sarcation that fuses modalities in an ordered manner. Our model outperforms several state-of-the-art models by 1-3% across various metrics, demonstrating the crucial role of modality order in sarcasm detection. The obtained improvements and detailed analysis show that audio tone should be infused with textual content, followed by visual information to identify sarcasm efficiently. The code and dataset are available at https://github.com/mohit2b/MO-Sarcation.

Image captioning frameworks usually employ an encoder-decoder paradigm, with the encoder receiving abstract image feature vectors as input and the decoder for language modeling. Nowadays, most prominent architectures employ features from region proposals derived from object detection modules. In this work, we propose a novel architecture for image captioning. We employ the object detection module integrated with transformer architecture as an encoder and GPT-2 (Generative Pre-trained Transformer) as a decoder. The encoder utilizes the information of the spatial relationships among detected objects. We introduce a unique methodology for image caption generation in Hindi, which is widely spoken in South Asia and India and is the world’s third most spoken language as well as India’s official language. In terms of BLEU scores, the proposed approach’s performance is comparable to those of other baselines, and the results illustrate that the proposed approach outperforms the other baselines. The efficacy of the proposed approach in generating correct captions is further determined by human assessment in terms of adequacy and fluency.

Dense image captioning is a task that requires generating localized captions in natural language for multiple regions of an image. This task leverages its functionalities from both computer vision for recognizing regions in an image and natural language processing for generating captions. Numerous works have been carried out on dense image captioning for resource-rich languages like English; however, resource-poor languages like Hindi are not explored. Hindi is one of India’s official languages and is the third most spoken language in the world. This article proposes a dense image captioning model to describe different segments of an image by generating more than one caption in the Hindi language. For localized image recognition and language modeling, we employ Faster R-CNN and Long Short-Term Memory (LSTM), respectively. Apart from this, we conduct various experiments using gated recurrent units (GRUs) and attention mechanism. By manually translating the well-known Visual Genome dataset from English to Hindi, a dataset has been created for dense image captioning in Hindi. The experiments conducted on the newly constructed Hindi dense image captioning dataset illustrate the efficacy of the proposed method over the state-of-the-art methods.

In sequence-to-sequence modeling tasks, such as image captioning, machine translation, and visual question answering, encoder-decoder architectures are state of the art. An encoder, convolutional neural network (CNN) encodes input images into fixed dimensional vector representation in the image captioning task, whereas a decoder, a recurrent neural network, performs language modeling and generates the target descriptions. Recent CNNs use the same operation over every pixel; however, all the image pixels are not equally important. To address this, the proposed method uses a dynamic convolution-based encoder for image encoding or feature extraction, Long-Short-Term-Memory as a decoder for language modeling, and X-Linear attention to make the system robust. Encoders, attentions, and decoders are important aspects of the image captioning task; therefore, we experiment with various encoders, decoders, and attention mechanisms. Most of the works for image captioning have been carried out for the English language in the existing literature. We propose a novel approach for caption generation from images in Hindi. Hindi, widely spoken in South Asia and India, is the fourth most-spoken language globally; it is India’s official language. The proposed method utilizes dynamic convolution operation on the encoder side to obtain a better image encoding quality. The Hindi image captioning dataset is manually created by translating the popular MSCOCO dataset from English to Hindi. In terms of BLEU scores, the performance of the proposed method is compared with other baselines, and the results obtained show that the proposed method outperforms different baselines. Manual human assessment in terms of adequacy and fluency of the captions generated further determines the efficacy of the proposed method in generating good-quality captions.

The advancement of medical research in the field of cancer prognosis and diagnosis using various modalities has put oncologists under tremendous stress. The complexity and heterogeneity involved in multiple modalities and their significantly varied clinical outcomes make it difficult to analyze the disease and provide the correct treatment. Breast cancer is the major concern among all cancers worldwide, specifically for females. To help oncologists and cancer patients, research for breast cancer survival estimation has been proposed. It ranges from complex deep neural networks to simple and interpretable architectures. We propose a utility kernel for a support vector machine (SVM) in this article. It is a simple yet powerful function, which performs better than other popular machine learning algorithms and deep neural networks in the task of breast cancer survival prediction using the TCGA-BRCA dataset. This study validates the proposed utility kernel using four different modalities (gene expression, copy number variation, clinical, and histopathological tissue images) and their multi-modal combinations. The SVM based on our utility kernel empirically proves its efficacy by achieving the highest value on various performance measures, whereas advanced deep neural networks fail to train on small and highly imbalanced breast cancer data.

Nowadays, multiple sources of information about proteins are available such as protein sequences, 3D structures, Gene Ontology (GO), etc. Most of the works on protein-protein interaction (PPI) identification had utilized these information about proteins, mainly sequence-based, but individually. The new advances in deep learning techniques allow us to leverage multiple sources/modalities of proteins, which complement each other. Some recent works have shown that multi-modal PPI models perform better than uni-modal approaches. This paper aims to investigate whether the performance of multi-modal PPI models is always consistent or depends on other factors such as dataset distribution, algorithms used to learn features, etc. We have used three modalities for this study: Protein sequence, 3D structure, and GO. Various techniques, including deep learning algorithms, are employed to extract features from multiple sources of proteins. These feature vectors from different modalities are then integrated in several combinations (bi-modal and tri-modal) to predict PPI. To conduct this study, we have used Human and S. cerevisiae PPI datasets. The obtained results demonstrate the potentiality of a multi-modal approach and deep learning techniques in predicting protein interactions. However, the predictive capability of a model for PPI depends on feature extraction methods as well. Also, increasing the modality does not always ensure performance improvement. In this study, the PPI model integrating two modalities outperforms the designed uni-modal and tri-modal PPI models.

The knowledge of protein-protein interaction (PPI) helps us to understand proteins' functions, the causes and growth of several diseases, and can aid in designing new drugs. The majority of existing PPI research has relied mainly on sequence-based approaches. With the availability of multi-omics datasets (sequence, 3D structure) and advancements in deep learning techniques, it is feasible to develop a deep multi-modal framework that fuses the features learned from different sources of information to predict PPI. In this work, we propose a multi-modal approach utilizing protein sequence and 3D structure. To extract features from the 3D structure of proteins, we use a pre-trained vision transformer model that has been fine-tuned on the structural representation of proteins. The protein sequence is encoded into a feature vector using a pre-trained language model. The feature vectors extracted from the two modalities are fused and then fed to the neural network classifier to predict the protein interactions. To showcase the effectiveness of the proposed methodology, we conduct experiments on two popular PPI datasets, namely, the human dataset and the S. cerevisiae dataset. Our approach outperforms the existing methodologies to predict PPI, including multi-modal approaches. We also evaluate the contributions of each modality by designing uni-modal baselines. We perform experiments with three modalities as well, having gene ontology as the third modality.

The advent of the Internet is a boon to society. However, many of its banes cannot be undermined, cyberbullying being one of them. The emotional state and sentiment of a person have a significant influence on the intended content. The current work is the first attempt in investigating the role of sentiment and emotion information for identifying cyberbullying in the Indian scenario. From Twitter, a benchmark Hind–English code-mixed corpus called BullySentEmo has been developed as there is no dataset available labeled with bully, sentiment, and emotion. Moreover, emoji information available with tweet texts can provide better understanding of user intention. The developed dataset consists of both modalities, tweet text, and emoji. In India, the majority of communication on different social media platforms is based on Hindi and English and language switching is a common practice in digital communication. An attention-based multimodal, adversarial multitasking framework is proposed for cyberbully detection (CBD) considering two auxiliary tasks: sentiment analysis (SA) and emotion recognition (ER). Experimental results indicate that compared to unimodal and single-task variants, the proposed framework improves the performance of the main task, i.e., CBD, by 3.59% and 2.56% in terms of accuracy and F1-Score, respectively. Furthermore, two different benchmark datasets (Twitter dataset and Aggression dataset) have been considered to show the robustness of our proposed model.

In today’s scenario, a large number of scientific articles on various domains are being published everyday, resulting in a rapid change in the trends of research topics. Retrieving the trending topics, evaluating the trends and extracting the scope of topics could be beneficial to young researchers, which can be recommended for future scope. Publication of articles is a continuous process, and so is the evolution of topics as well as the scope. The dynamic behavior of topics can be handled by continuously updating the partitioning of incoming articles arriving in a streaming manner. This article proposes an online clustering approach utilizing the autoencoder, which is capable of handling the online stream of articles. The training is performed by considering the multiple views of articles and posing the topic modeling (TM) problem as a multiview (MV) clustering problem. In addition, we employ an evolutionary-based approach to the latent representation of data to automatically determine the number of clusters. To learn the nonlinear mapping appropriately and to generate clusters effectively, the model simultaneously optimizes the reconstruction loss and clustering loss in an MV framework. The developed method has experimented on ArXiv dataset to determine the trending topics, scope, and recommendation of topics. The superiority of the proposed method in generating clusters as well as in determining scope is shown by comparing the results with many existing and baseline methods.

Mental health disorder continues to be a grievous concern plaguing humans worldwide. The scarcity of mental health professionals (MHPs) has driven novel efforts lately to combat mental illness by developing automated systems capable of assisting MHPs. However, lack of high-quality conversational data due to privacy concerns remains a bottleneck toward its study and automation. Also, distinguishing and identifying various mental disorders is a challenging task as these are expressed using similar patterns in terms of language usage and overall sentiment polarity. The need for classifying different mental illnesses is rather intuitive as their true diagnosis is imperative for determining their path to recovery. As a first step toward creating automated virtual agent (VA), in this article, we curate a large-scale dataset for multiple mental illnesses comprising of dyadic conversations between a mentally ill user (typically a support seeker) and the VA (typically a peer supporter imparting hope and motivation), collected from a peer-to-peer support platform. We present a hierarchical attention-based deep neural network classifier for modeling conversations to detect different mental disorders as the dialog progresses. The proposed network is equipped with lexicon-based sentiment scores in order to prioritize certain contributing features. The proposed model attained an accuracy of 83.91% and outperformed several strong baselines.

Gender plays a crucial role in improving the performance quality of personalized systems. Privacy and anonymity allow users to hide their details. Based on the intuition that post contents of male and female users differ, we can predict the gender of the social media account holder via their corresponding posts. These posts can be multimodal (text + image) in nature. We investigate various emotion-assisted multimodal gender prediction models in this article. The developed models use gated recurrent units (GRUs) and ResNets for extracting features from the tweets and the images, respectively. The distribution of emotion categories is related to the gender of the target person. In response to these findings, this article describes the first attempt to use multimodal (image and text posted) information for gender prediction in a multitask setting with emotion recognition as an auxiliary task. The enriched PAN-2018 dataset with gender and emotion labels is used to train gender and emotion networks. Several models were developed to improve the gender prediction task by generating better emotion-aware features. The results show that the proposed multimodal emotion detection model outperforms single-modal (text and image)-based models and state-of-the-art systems on the benchmark PAN-2018 dataset.

With the advent of internet technologies, it has created different ways of writing anonymously, which has lead to criminal and malicious activities over social media platforms. Thus, the automatic authentication checking of the available contents is the need of the hour. Social media sites, such as Facebook, Twitter, and so on, are used heavily by the users for sharing of information about their day-to-day activities. The identity of the suspect user is matched against tweets written by the specific user in tweet-user verification process. Writing styles of different users differ from each other, due to unique word choices, emoji selection, sentence formation, and punctuation usage. We have developed a multimodal Siamese-based architecture, which uses attention between the text and emoji parts of the tweet for generating a combined representation for the tweet. Attention helps in selecting the relevant information from different modalities. Modality attention is used for fusing the two modalities (text and emoji). We have used a newly developed multimodal Twitter dataset for evaluating the performance of the proposed model. We achieved an average accuracy, precision, recall, and$F$-measure values of 68.50%, 78.52%, 69.47%, and 67.05%, respectively. The results show an increase of 2.14% in$F$-measure in comparison with the current state-of-the-art (SOTA) models for this dataset.

During network planning phase, optimal network planning implemented through efficient resource allocation and static traffic demand provisioning in IP-over-elastic optical network (IP-over-EON) is significantly challenging compared with the fixed-grid wavelength division multiplexing (WDM) network due to increased flexibility in IP-over-EON. Mathematical programming based optimization models used for this purpose may not provide solution for large networks due to large computational complexity. In this regard, a greedy heuristic may be used that intuitively selects traffic elements in sequence from static traffic demand matrix and provisions the traffic elements after necessary resource allocation. However, in general, such greedy heuristics offer suboptimal solutions, since appropriate traffic sequence offering the optimal performance is rarely selected. In this regard, we propose a reinforcement learning technique (in particular a Q-learning method), combined with an auxiliary graph (AG)-based energy efficient greedy method to be used for large network planning. The Q-learning method is used to decide the suitable sequence of traffic allocation such that the overall power consumption in the network reduces. In the proposed heuristic, each traffic from the given static traffic demand matrix is successively selected using the Q-learning method and provisioned using the AG-based greedy method.

The expression of displeasure on a consumer's behalf towards an organization, product, or event is denoted via the speech act known as complaint. Customers typically post reviews on retail websites and various social media platforms about the products or services they purchase, and the reviews may include complaints about the products or services. Automatic detection of consumers' complaints about items or services they buy can be critical for organizations and online merchants since they can use this insight to meet the customers' requirements, including handling and addressing the complaints. Previous studies on Complaint Identification (CI) are limited to text. Images posted with the reviews can provide cues to identify complaints better, thus emphasizing the importance of incorporating multi-modal inputs into the process. Furthermore, the customer's emotional state significantly impacts the complaint expression since emotions generally influence any speech act. As a result, the impact of emotion and sentiment on automatic complaint identification must also be investigated. One of the major contributions of this work is the creation of a new dataset- Complaint, Emotion, and Sentiment Annotated Multi-modal Amazon Reviews Dataset (CESAMARD), a collection of opinionated texts (reviews) and images of the products posted on the website of the retail giant Amazon. We present an attention-based multi-modal, adversarial multi-task deep neural network model for complaint detection to demonstrate the utility of the multi-modal dataset. Experimental results indicate that the multi-modality and multi-tasking complaint identification outperforms uni-modal and single-task variants.

Artificial Intelligence-based clinical decision support is gaining ever-growing popularity and demand in both the research and industry communities. One such manifestation is automatic disease diagnosis, which aims to assist clinicians in conducting symptom investigations and disease diagnoses. When we consult with doctors, we often report and describe our health conditions with visual aids. Moreover, many people are unacquainted with several symptoms and medical terms, such as mouth ulcer and skin growth. Therefore, visual form of symptom reporting is a necessity. Motivated by the efficacy of visual form of symptom reporting, we propose and build a novel end-to-end Multi-modal Disease Diagnosis Virtual Assistant (MDD-VA) using reinforcement learning technique. In conversation, users' responses are heavily influenced by the ongoing dialogue context, and multi-modal responses appear to be of no difference. We also propose and incorporate a Context-aware Symptom Image Identification module that leverages discourse context in addition to the symptom image for identifying symptoms effectively. Furthermore, we first curate a multi-modal conversational medical dialogue corpus in English that is annotated with intent, symptoms, and visual information. The proposed MDD-VA outperforms multiple uni-modal baselines in both automatic and human evaluation, which firmly establishes the critical role of symptom information provided by visuals . The dataset and code are available at https://github.com/NLP-RL/DrCanSee

Existing bug text reports are widely used by software engineers to assist them in understanding important components of individual defects and adjustments done to resolve the fault. However, bug reports are typically long and require significant effort to comprehend. Summarization of bug reports appears to be beneficial in this regard, covering essential and diverse information. We frame the Bug report summarization problem as a clustering-based optimization problem and solve it using a multi-view multi-objective optimization framework. To represent the bug reports, semantic and syntactic representations, which are regarded as separate views, are taken into account. Several cluster quality measures computed on partitionings obtained using distinct views are optimized simultaneously using a multi-objective optimization-based approach known as archived multi-objective simulated annealing. To determine the consensus between the partitionings generated using different views, an agreement index is computed, which is also optimized simultaneously along with other cluster quality measures. The proposed methodology automatically determines the number of clusters. The experiments are carried out using the two benchmark datasets (SDS and ADS) and evaluated using the well-known ROUGE, Precision, Recall, and F-measure evaluation metrics. The obtained results show that the proposed methodology outperforms state-of-the-art methods.

Existing extractive summarization systems suffer from the problem of identifying correct aspects while generating summary from multiple documents on a particular topic. Online access to documents allows readers to post comments on available documents/blogs which may be helpful in correctly identifying the important aspects from a document in a crowd-sourcing fashion. However, comments can be unstructured and boisterous, making them difficult to manage. The current paper builds an unsupervised summarization system utilizing documents and the corresponding comments by carefully selecting a subset of sentences from the document based on document-comment similarities. A binary multiobjective optimization technique exploring the search capability of differential evolution is utilized for simultaneously optimizing different aspects of summarization like diversity between sentences of the summary, user-attention and density-based score providing importance to the comments with respect to the documents, user-attention with a syntactic score between the summary phrases. Extensive analysis is performed by varying the combinations of different objective functions, on a dataset containing 45 topics belonging to different news themes as well as in a new French dataset having documents, summaries, and comments. The experimental result validates the importance of all the objective functions and superiority over other methods.

The advent of the Internet is a boon to society. However, many of its banes cannot be undermined, and cyberbullying is one of them. In this work, we have created a benchmark corpus for cyberbullying detection in code-mixed languages. In India, most communications on different social media platforms are based on Hindi and English languages, and language switching is a common practice in digital communication. To investigate how code-mixed data can be handled effectively, BERT language model and VecMap based bilingual embedding along with a two-channel convolutional neural network model, namely BERT+VecMap-CNN, have been used. The input to one channel is the BERT language model and that to the other is the bilingual word embedding based on VecMap. As a baseline, we used standard machine learning models, as well as deep neural network models like CNN and LSTM. Our proposed model outperforms the baselines with overall accuracy and F1-measure values of 81.12%, and 81.03%, respectively. Furthermore, a different benchmark code-mixed dataset has been considered to show the robustness of our proposed model.1

Due to the rapid spread of COVID-19 as a global pandemic, it has become increasingly critical to have fast, cheap, and reliable tools to assist physicians in diagnosing COVID19. Several automated systems using deep learning techniques have demonstrated promising results by analyzing Computed Tomography (CT-scan) or X-ray data to complement conventional diagnostic tools. In this paper, we aim to emphasize the role of point-of-care ultrasound imaging using deep learning as a tool to detect COVID-19 more prominently. Ultrasound imaging is non-invasive and widely available in medical facilities all over the world. This paper presents an ensemble technique based on Sugeno Fuzzy Integrals with convolutional neural networks (CNNs) as the base model. It classifies lung ultrasound (LUS) images of patients into COVID-19 and Non-COVID-19 categories. The lack of COVID-19 data makes it challenging to train a traditional CNN from scratch, so we have adapted a transfer learning approach instead of training the base classifiers VGG16, ResNet-50, and GoogLeNet. We apply the gained knowledge in the target domain of small lung ultrasound frames, considering the ImageNet dataset as the source domain. We have also adapted image pre-processing techniques to remove noises so that the model can only focus on specific features. Our proposed framework is evaluated on a publicly available dataset, achieving 96.7% accuracy. The proposed architecture outperforms the state-of-the-art method on the same dataset and proves to be a reliable COVID-19 detector.

Psychological stress is a feeling of mental stress and pressure. Within this context we try to understand the tweets which express psychological stress. Psychological stress detection is a complex task majorly because the stressed community contains mostly introverts. With the increase in people using social-media sites, the information related to mental health of the people extracted from their posts are increasing day-by-day. We use this information to design an AI enabled framework for automatic stress detection. However these data are noisy and complex, therefore deep learning based models are utilized for automatic extraction of features rather than manual extraction of features. Several deep learning based architectures including Multichannel CNN, CNN, GRU, Capsule network and BERT model are explored for solving this task of detecting tweets having mentions about mental stress. Experimental results on a standard Twitter dataset reveal that Multichannel CNN attains the best performance with accuracy of 97.5%, precision, recall and f-score values of 96.8%, 97.5% and 97.2%, respectively.

Plagiarism detection, and identifying the author(s) of scholarly articles are important for determining the actual credit of author(s). Authorship attribution of long texts mostly depends on manual feature engineering for the extraction of relevant features. In this paper, we have developed a capsule-based convolutional neural network (CNN) architecture, which takes term frequency-inverse document frequency (TF-IDF) vector as input, for solving the task of authorship attribution. The TF-IDF vectors are used to represent the articles. The generated TF-IDF vector is fed to a CNN framework, which uses a capsule for learning the spatial features. Our proposed approach reduces the extra effort of hand-crafted feature extraction. The developed model is tested on the created scholarly abstract-based dataset. From the obtained results, it can be concluded that TF-IDF based capsule-CNN method outperforms previous approaches which had utilized hand-engineered features with the neural architectures. Different multi-tasking-based models are also developed to analyze the effect of category information in the authorship prediction task. From the results, it is revealed that multi-tasking-based models yield comparative results as compared to the single-tasking-based system. To illustrate the relevance of our developed system, different qualitative analyses have also been carried out, which clearly reveals the effectiveness of utilizing category information, and capsule networks. The source codes for the proposed approach will be shared after the acceptance.

Intent detection is one of the crucial Natural Language Understanding(NLU) tasks studied extensively. Misclassification of intents impacts the overall performance of the conversational systems as natural language understanding is the first mean of interaction between a user and a virtual agent. The traditional approach of intent detection is limited only to textual features of user utterances and overlooks other semantic features. The sentiment and emotional state of the speaker are two such semantic features that have essential impacts on intent detection, as they implicitly express user intention conveyed through the user’s message. Depending on the context, these features can help the dialogue agent respond to the same intent with varying degrees of sentiment and emotion. Thus, investigating the role of sentiment and emotion on intent detection is a matter of great interest. The current work investigates the impact of utilizing sentiment and emotion information on intent detection tasks and proposes emotion and sentiment aided intent detection models. We also investigate the impact of sentiment and emotion using three different multitasking frameworks and present a joint model that utilizes the co-relation information across these tasks to correctly identify all these NLU aspects (intent, sentiment, and emotion). The obtained experimental results by the proposed models outperform several baselines and state-of-the-art intent detection models on multiple datasets by a significant margin of 1.8% - 3%, demonstrating the significant role of sentiment and emotion features in intent detection1.

Recent advances in deep learning typically, with the introduction of transformer based models has shown massive improvement and success in many Natural Language Processing (NLP) tasks. One such area which has leveraged immensely is conversational agents or chatbots in open-ended (chit-chat conversations) and task-specific (such as medical or legal dialogue bots etc.) domains. However, in the era of automation, there is still a dearth of works focused on one of the most relevant use cases, i.e., tutoring dialog systems that can help students learn new subjects or topics of their interest. Most of the previous works in this domain are either rule based systems which require a lot of manual efforts or are based on multiple choice type factual questions. In this paper, we propose EDICA (Educational Domain Infused Conversational Agent), a language tutoring Virtual Agent (VA). EDICA employs two mechanisms in order to converse fluently with a student/user over a question and assist them to learn a language: (i) Student/Tutor Intent Classification (SIC-TIC) framework to identify the intent of the student and decide the action of the VA, respectively, in the on-going conversation and (ii) Tutor Response Generation (TRG) framework to generate domain infused and intent/action conditioned tutor responses at every step of the conversation. The VA is able to provide hints, ask questions and correct student's reply by generating an appropriate, informative and relevant tutor response. We establish the superiority of our proposed approach on various evaluation metrics over other baselines and state of the art models.

The prediction of the protease cleavage site of an amino acid sequence of Human Immune Deficiency Virus (HIV-1) type 1 has various significant applications in finding novel drug targets, Protease Inhibitor study, and many more. HIV-1 protease cleavage site prediction studies have been carried out for decades using different feature extraction techniques by many researchers. Existing studies only focus on the simple representation of features, the latent feature space which is capable of reconstructing the data well was never explored. Motivated by this, we have utilized the concept of Stacked Autoencoder (SAE) for latent feature extraction from the con-catenated feature set which is constructed with three different types of extracted features namely structural, physicochemical and sequential features. The hidden nodes in deep layers of SAE manage the high level abstraction for dimensionality reduction and maintain the key information of the data simultaneously. Several classifiers are trained using this informative feature set and finally majority voting based classifier ensemble technique is applied for cleavage site prediction task. The proposed approach is applied on several benchmark data sets and average accuracy, precision, recall, and F-measure values of 0.95, 0.91, 0.83, 0.86, respectively, are attained. Comparison with respect to other existing studies also establishes the efficacy of the proposed technique.

With the surge of digitized entertainment systems in recent years, the element of personalised experience for users gives a competitive edge to entertainment businesses. Hence, the importance of recommendation systems, in particular for video or movie recommendation, is evident. However, majority of the recommendation systems are primarily learned in supervised fashion on empirical data like user ratings. This approach has certain short comings, specifically the cold start problem and the data sparsity problem. Moreover, existing recommendation systems do not utilize multiple information associated with videos/movies like their textual summary, meta-data information, audio and video signals etc. With the increase in the multi-modal information processing in different fields of artificial intelligence, we have proposed the task of multi-modal recommendation system in the current study. For representing the movie/video, feature vectors generated from text modality, meta-data modality, audio and video modalities are concatenated. A novel knowledge graph based approach is applied for generating feature vectors from text modality. Finally, a Siamese architecture based deep learning technique is proposed to perform regression over similarities using multi-modal user-item embeddings. As there was no data set available for solving the task of multi-modal recommendation, we have also enhanced the existing benchmark Movie-lens 100k dataset with text, video, audio information and utilized that for performing experimentation. Experimental results establish the efficacy of using multi-modal information for movie embedding generation

The worldwide breakout of the novel COVID-19 has resulted in one of the worst epidemics in modern times since World War II. Although various vaccinations are being produced, their efficacy remains a considerable hurdle. This is especially true when new virus strains emerge. The main challenge to combating this pandemic is diagnosing and isolating COVID-19 positive cases as early as possible. As a result, COVID-19 needs to be detected early and accurately to prevent its spread. This paper proposes a computer-aided automated COVID-19 detection tool based on Computed Tomography (CT-scan) images of lungs. The proposed approach applies an ensemble technique based on Sugeno Fuzzy Integrals with convolutional neural networks (CNNs) as the base model. The lack of COVID-19 data makes it challenging to train a standard CNN from scratch, so we use a transfer learning approach instead of training the base classifiers, VGG-16, InceptionResnetV2, and Xception. We apply the gained knowledge in the target domain of small CT-scan data, considering ImageNet dataset as the source domain. We have also adapted image pre-processing techniques to remove noises so that the model can only focus on specific features. Our proposed framework achieves 98.99% accuracy on a publicly available dataset and outperforms the existing state-of-the-art methods. Experimental results and comparative analysis with baselines establish the need and effectiveness of our proposed model.

With the rise of personalized online services, a huge opportunity for user profiling has developed. Gender plays a very important role for services that rely on information about a user's background. Although, due to anonymity and privacy, the gender information of a user is usually unavailable for other users. Social Networking sites have provided users with a lot of features to express their thoughts and emotions either using pictures or emojis or writing texts. Based on the idea that female and male users have some differences in their post and message contents, social media accounts can be analyzed using their textual posts for finding the user's gender. In this work, we explore different emotion-aided multi-modal gender prediction models. The basic intuition behind our proposed approach is to predict the gender of a user based on the emotional clues present in their multimodal posts, which includes texts as well as images. PAN 2018 dataset is enriched with emotion labels, for the experimentation. Different multi-tasking based architectures have been developed for gender prediction. Obtained results on the benchmark PAN-2018 dataset illustrate that the proposed multimodal emotion-aided system performs better than the single modal (with only text and only image) based models and the state of the art system too.

Automatic depression detection from multimodal signals assumes significance since the features from different modalities possesses contrasting but relevant information. Multimodal signal analysis involving the derivation of a joint representation, identification of correct alignment, and selection of appropriate fusing mechanisms is a challenging task. In this paper, we focus on an approach to detecting depression from multimodal signals using task-specific representation and late fusion. Our method uses a task-oriented embedding generated through a GPT2-medium language model that is fine-tuned on a conversational dataset. We present our experimental results benchmarking on Distress Analysis Interview Corpus - a Wizard of Oz (DAIC-WOZ) corpus. Our proposed task-oriented embedding on a simple BiLSTM network outperforms the previous text-based depression prediction methods. The results are further strengthened when audio and video modalities are fused with the text predictions on a late-fusion network.

The social NLP researchers and mental health practitioners have witnessed exponential growth in the field of mental health detection and analysis on social media. It has become important to identify the reason behind mental illness. In this context, we introduce a new dataset for Causal Analysis of Mental health in Social media posts (CAMS). We first introduce the annotation schema for this task of causal analysis. The causal analysis comprises of two types of annotations, viz, causal interpretation and causal categorization. We show the efficacy of our scheme in two ways: (i) crawling and annotating 3155 Reddit data and (ii) re-annotate the publicly available SDCNL dataset of 1896 instances for interpretable causal analysis. We further combine them as CAMS dataset and make it available along with the other source codes https://anonymous.4open.science/r/CAMS1/. Our experimental results show that the hybrid CNN-LSTM model gives the best performance over CAMS dataset.

Keyword extraction is an integral task for many downstream problems like clustering, recommendation, search and classification. Development and evaluation of keyword extraction techniques require an exhaustive dataset; however, currently, the community lacks large-scale multi-lingual datasets. In this paper, we present MAKED, a large-scale multi-lingual keyword extraction dataset comprising of 540K+ news articles from British Broadcasting Corporation News (BBC News) spanning 20 languages. It is the first keyword extraction dataset for 11 of these 20 languages. The quality of the dataset is examined by experimentation with several baselines. We believe that the proposed dataset will help advance the field of automatic keyword extraction given its size, diversity in terms of languages used, topics covered and time periods as well as its focus on under-studied languages.

Although a wide range of applications have been proposed in the field of multimodal natural language processing, very few works have been tackling multimodal relational lexical semantics. In this paper, we propose the first attempt to identify lexico-semantic relations with visual clues, which embody linguistic phenomena such as synonymy, co-hyponymy or hypernymy. While traditional methods take advantage of the paradigmatic approach or/and the distributional hypothesis, we hypothesize that visual information can supplement the textual information, relying on the apperceptum subcomponent of the semiotic textology linguistic theory. For that purpose, we automatically extend two gold-standard datasets with visual information, and develop different fusion techniques to combine textual and visual modalities following the patch-based strategy. Experimental results over the multimodal datasets show that the visual information can supplement the missing semantics of textual encodings with reliable performance improvements.

Rahul Kumar, Sandeep Mathias, Sriparna Saha, Pushpak Bhattacharyya. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Tulika Saha, Saichethan Reddy, Anindya Das, Sriparna Saha, Pushpak Bhattacharyya. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Detecting cyberbullying from memes is highly challenging, because of the presence of the implicit affective content which is also often sarcastic, and multi-modality (image + text). The current work is the first attempt, to the best of our knowledge, in investigating the role of sentiment, emotion and sarcasm in identifying cyberbullying from multi-modal memes in a code-mixed language setting. As a contribution, we have created a benchmark multi-modal meme dataset called MultiBully annotated with bully, sentiment, emotion and sarcasm labels collected from open-source Twitter and Reddit platforms. Moreover, the severity of the cyberbullying posts is also investigated by adding a harmfulness score to each of the memes. The created dataset consists of two modalities, text and image. Most of the texts in our dataset are in code-mixed form, which captures the seamless transitions between languages for multilingual users. Two different multimodal multitask frameworks (BERT+ResNET-Feedback and CLIP-CentralNet) have been proposed for cyberbullying detection (CD), the three auxiliary tasks being sentiment analysis (SA), emotion recognition (ER) and sarcasm detection (SAR). Experimental results indicate that compared to uni-modal and single-task variants, the proposed frameworks improve the performance of the main task, i.e., CD, by 3.18% and 3.10% in terms of accuracy and F1 score, respectively.

The scarcity of Mental Health Professionals (MHPs) available to assist patients underlines the need for developing automated systems to help MHPs combat the grievous mental illness called Major Depressive Disorder. In this paper, we develop a Virtual Assistant (VA) that serves as a first point of contact for users who are depressed or disheartened. In support based conversations, two primary components have been identified to produce positive outcomes,empathy andmotivation. While empathy necessitates acknowledging the feelings of the users with a desire to help, imparting hope and motivation uplifts the spirit of support seekers in distress. A combination of these aspects will ensure generalized positive outcome and beneficial alliance in mental health support. The VA, thus, should be capable of generating empathetic and motivational responses, continuously demonstrating positive sentiment by the VA. The end-to-end system employs two mechanisms in a pipe-lined manner : (i)Motivational Response Generator (MRG) : a sentiment driven Reinforcement Learning (RL) based motivational response generator; and (ii)Empathetic Rewriting Framework (ERF) : a transformer based model that rewrites the response from MRG to induce empathy. Experimental results indicate that our proposed VA outperforms several of its counterparts. To the best of our knowledge, this is the first work that seeks to incorporate these aspects together in an end-to-end system.

The three times increase of SonyLiv viewers during the Tokyo Olympic, the 10% hike of YouTube users during the isolation era of covid-pandemic, and the 19% growth in Netflix user count due to the fastest growth of OTT, etc. have made the digital platform’s mode all-time active and specific. The hourly increase of users’ interactions and the e-commerce platform’s desire of letting users engage on their sites are pushing researchers to shape the virtual digital web as user specific and revenue-oriented. This paper develops a deep learning-based approach for building a movie recommendation system with three main aspects: (a) using a knowledge graph to embed text and meta information of movies, (b) using multi-modal information of movies like audio, visual frames, text summary, meta data information to generate movie/user representations without directly using rating information; this multi-modal representation can help in coping up with cold-start problem of recommendation system (c) a graph attention network based approach for developing regression system. For meta encoding, we have built knowledge graph from the meta information of the movies directly. For movie-summary embedding, we extracted nouns, verbs, and object to build a knowledge graph with head-relation-tail relationships. A deep neural network, as well as Graph attention networks, are utilized for measuring performance in terms of RMSE score. The proposed system is tested on an extended MovieLens-100K data-set having multi-modal information. Experimental results establish that only rating-based embeddings in the current setup outperform the state-of-the-art techniques but usage of multi-modal information in embedding generation performs better than its single-modal counterparts.1.

Over the last few years, dozens of healthcare surveys have shown a shortage of doctors and an alarming doctor-population ratio. With the motivation of assisting doctors and utilizing their time efficiently, automatic disease diagnosis using artificial intelligence is experiencing an ever-growing demand and popularity. Humans are known by the company they keep; similarly, symptoms also exhibit the association property, i.e., one symptom may strongly suggest another symptom's existence/non-existence, and their association provides crucial information about the suffering condition. The work investigates the role of symptom association in symptom investigation and disease diagnosis process. We propose and build a virtual assistant called Association guided Symptom Investigation and Diagnosis Assistant (A-SIDA) using hierarchical reinforcement learning. The proposed A-SIDDA converses with patients and extracts signs and symptoms as per patients' chief complaints and ongoing dialogue context. We infused association-based recommendations and critic into the assistant, which reinforces the assistant for conducting context-aware, symptom-association guided symptom investigation. Following the symptom investigation, the assistant diagnoses a disease based on the extracted signs and symptoms. The assistant then diagnoses a disease based on the extracted signs and symptoms. In addition to diagnosis accuracy, the relevance of inspected symptoms is critical to the usefulness of a diagnosis framework. We also propose a novel evaluation metric called Investigation Relevance Score (IReS), which measures the relevance of symptoms inspected during symptom investigation. The obtained improvements (Diagnosis success rate-5.36%, Dialogue length-1.16, Match rate-2.19%, Disease classifier-6.36%, IReS-0.3501, and Human score-0.66) over state-of-the-art methods firmly establish the crucial role of symptom association that gets uncovered by the virtual agent. Furthermore, we found that the association guided symptom investigation greatly increases human satisfaction, owing to its seamless topic (symptom) transition.

Image captioning refers to the process of generating a textual description that describes objects and activities present in a given image. It connects two fields of artificial intelligence, computer vision, and natural language processing. Computer vision and natural language processing deal with image understanding and language modeling, respectively. In the existing literature, most of the works have been carried out for image captioning in the English language. This article presents a novel method for image captioning in the Hindi language using encoder–decoder based deep learning architecture with efficient channel attention. The key contribution of this work is the deployment of an efficient channel attention mechanism with bahdanau attention and a gated recurrent unit for developing an image captioning model in the Hindi language. Color images usually consist of three channels, namely red, green, and blue. The channel attention mechanism focuses on an image’s important channel while performing the convolution, which is basically to assign higher importance to specific channels over others. The channel attention mechanism has been shown to have great potential for improving the efficiency of deep convolution neural networks (CNNs). The proposed encoder–decoder architecture utilizes the recently introduced ECA-NET CNN to integrate the channel attention mechanism. Hindi is the fourth most spoken language globally, widely spoken in India and South Asia; it is India’s official language. By translating the well-known MSCOCO dataset from English to Hindi, a dataset for image captioning in Hindi is manually created. The efficiency of the proposed method is compared with other baselines in terms of Bilingual Evaluation Understudy (BLEU) scores, and the results obtained illustrate that the method proposed outperforms other baselines. The proposed method has attained improvements of 0.59%, 2.51%, 4.38%, and 3.30% in terms of BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores, respectively, with respect to the state-of-the-art. Qualities of the generated captions are further assessed manually in terms of adequacy and fluency to illustrate the proposed method’s efficacy.

Breast Cancer is a highly aggressive type of cancer generally formed in the cells of the breast. Despite significant advances in the treatment of primary breast cancer in the last decade, there is a dire need to attempt of an accurate predictive model for breast cancer prognosis prediction. Researchers from various disciplines are working together to develop methods to save people from this fatal disease. A good predictive model can help in correct prognosis prediction of breast cancer. This accurate prediction can have several benefits like detection of cancer in the early stage, spare patients from getting unnecessary treatment and medical expenses related to it. Previous works rely mostly on uni-modal data (selected gene expression)for predictive model design. In recent years, however, multi-modal cancer data sets have become available (gene expression, copy number alteration and clinical). Motivated by the enhancement of deep-learning based models, in the current study, we propose to use some deep-learning based predictive models in a stacked ensemble framework to improve the prognosis prediction of breast cancer from available multi-modal data sets. One of the unique advantages of the proposed approach lies in the architecture of the model. It is a two-stage model. Stage one uses a convolutional neural network for feature extraction, while stage two uses the extracted features as input to the stack-based ensemble model. The predictive performance evaluated using different performance measures shows that this model produces better result than already existing approaches. This model results in AUC value of 0.93 and accuracy of 90.2 percent at medium stringency level (Specificity = 95 percent and threshold = 0.45). Keras 2.2.1, along with Tensorflow 1.12, is used for implementing the source code of the model. The source code can be downloaded from Github: https://github.com/nikhilaryan92/BreastCancer.

In today's digital world, we are equipped with modern computer-based data collection sources and feature extraction methods. It enhances the availability of the multi-view data and corresponding researches. Multi-view prediction models form a mainstream research direction in the healthcare and bioinformatics domain. While these models are designed with the assumption that there is no missing data for any views, in the real world, certain views of the data are often not having the same number of samples, resulting in the incomplete multi-view dataset. The studies performed over these datasets are termed incomplete multi-view clustering or prediction. Here, we develop a two-stage generative incomplete multi-view prediction model named GIMPP to address the missing view problem of breast cancer prognosis prediction by explicitly generating the missing data. The first stage incorporates the multi-view encoder networks and the bi-modal attention scheme to learn common latent space representations by leveraging complementary knowledge between different views. The second stage generates missing view data using view-specific generative adversarial networks conditioned on the shared representations and encoded features given by other views. Experimental results on TCGA-BRCA and METABRIC datasets proves the usefulness of the developed method over the state-of-the-art methods.

An in-depth exploration of gene prognosis using different methodologies aids in understanding various biological regulations of genes in disease pathobiology and molecular functions. Interpreting gene functions at biological and molecular levels remains a daunting yet crucial task in domains such as drug design, personalized medicine, and next-generation diagnostics. Recent advancements in omics technologies have produced diverse heterogeneous genomic datasets like micro-array gene expression, miRNA expression, DNA sequence, 3D structures, which are significant resources for understanding the gene functions. In this paper, we propose a novel self-attention based deep multi-modal model, named DeePROG, for the prognosis of disease affected genes based on heterogeneous omics data. We use three NCBI datasets covering three modalities, namely gene expression profile, the underlying DNA sequence, and the 3D protein structures. To extract useful features from each modality, we develop several context-specific deep learning models. Besides, we develop three attention-based deep bi-modal architectures along with DeePROG to leverage the prognosis of the underlying biomedical data. We assess the performance of the models' in terms of computational assessment of function annotation (CAFA2) metrics. Moreover, we analyze the results in terms of receiver operating characteristics (ROC) curve in high-class imbalance data setting and perform statistical significance tests in terms of Welch's t-test. Experiment results show that DeePROG significantly outperforms baseline models across in terms of performance metrics. The source code and all preprocessed datasets used in this study are available at https://github.com/duttaprat/DeePROG.

Hyperparameter tuning, specifically tuning of learning rate, can often be a time-consuming process, especially when dealing with large data sets. A mathematical foundation in the choice of learning rate can minimize tuning efforts. We propose the application of a novel adaptive learning rate paradigm, guided by Lipschitz continuity of the loss functions (LipGene), to the task of Gene Expression Inference using shallow neural networks. We utilize Mean Absolute Error and Quantile loss separately for training. Our adaptive learning rate, which is dynamically computed for each epoch, is based on the principle of Lipschitz constant and requires no tuning. Experimentally, we prove that our proposed approach greatly surpasses conventional choices of learning rates in terms of both speed of convergence and generalizability. Advocating the principle of Parsimonious Computing, our method can reduce compute infrastructure required for training by using smaller networks with a minimal compromise on the prediction error.

MOTIVATION: To minimize the accelerating amount of time invested on the biomedical literature search, numerous approaches for automated knowledge extraction have been proposed. Relation extraction is one such task where semantic relations between the entities are identified from the free text. In the biomedical domain, extraction of regulatory pathways, metabolic processes, adverse drug reaction or disease models necessitates knowledge from the individual relations, for example, physical or regulatory interactions between genes, proteins, drugs, chemical, disease or phenotype. RESULTS: In this paper, we study the relation extraction task from three major biomedical and clinical tasks, namely drug-drug interaction, protein-protein interaction, and medical concept relation extraction. Towards this, we model the relation extraction problem in a multi-task learning (MTL)framework, and introduce for the first time the concept of structured self-attentive network complemented with the adversarial learning approach for the prediction of relationships from the biomedical and clinical text. The fundamental notion of MTL is to simultaneously learn multiple problems together by utilizing the concepts of the shared representation. Additionally, we also generate the highly efficient single task model which exploits the shortest dependency path embedding learned over the attentive gated recurrent unit to compare our proposed MTL models. The framework we propose significantly improves over all the baselines (deep learning techniques)and single-task models for predicting the relationships, without compromising on the performance of all the tasks.

Speech act classification determining the communicative intent of an utterance has been studied widely over the years as an independent task. This holds true for discussion in any for a, including social media platforms such as Twitter. However, the tweeter’s emotional state has a huge impact on its pragmatic content because communication is fundamentally characterized and mediated by direct emotions. Sentiment as a human behavior often has a strong relation to emotion, and one helps to understand the other better. We hypothesize that the association between emotion and sentiment will provide a clearer understanding of the tweeter’s state of mind, aiding the identification of tweet acts (speech acts in Twitter, TAs). As the first step, we create a new multimodal, emotion-TA,EmoTAdataset collected from the open-source Twitter dataset. To incorporate these multiple aspects, we propose a multitask ensemble adversarial learning framework for multimodal TA classification (TAC). In addition, we also incorporate a joint embedding network, with bidirectional constraints to capture and efficiently integrate the shared semantic relationships across modalities and learn generalized features across multiple tasks. Experimental results indicate that the proposed framework boosts the performance of the primary task, TAC, by benefiting from the two secondary tasks, i.e., sentiment and emotion analyses compared to its unimodal and single-task TAC variants.

Microblog summarization systems are gaining importance during natural disasters. A lot of tweets are posted along with multimedia content during the occurrence of any natural disaster event. Extracting relevant information/summary from these tweets is important for the smooth functioning of the rescue operation. Moreover, because of the limited size of the tweets, in many cases, tweets are associated with images. The current work is the first of its kind where both the image and the tweet text are utilized simultaneously to generate a summary from microblog data generated during a disaster event. Different aspects, such as syntactic similarity, the maximum length of the tweets, retweet score, and antiredundancy, are considered as objective functions and those are simultaneously optimized using a metaheuristic population-based evolutionary strategy to select a good set of tweets to form a good quality summary. In order to extract information from images, a dense captioning model is utilized and the dense captions are further utilized for calculating the antiredundancy measure. We employed word mover distance to capture the semantic similarity between two tweets. Due to the unavailability of the dataset for multimodal microblog summarization tasks in a disaster-event scenario, datasets are created and made openly available to the community. The obtained summarization results are evaluated using the well-known ROUGE measure.

Authorship attribution (AA) is an important task, as it identifies the author of a written text from a set of suspect authors. Different methodologies of anonymous writing have been discovered with the rising usage of social media. This anonymous writing leads to an increase in malicious and suspicious activities, and anonymity makes it difficult to find the suspect. AA helps to find the writer of a suspect text from a set of suspects. Different social media platforms, such as Twitter, Facebook, and Instagram, are used regularly by the users for sharing their daily life activities. Finding the writer of microtexts is considered the toughest task due to the shorter length of the suspect piece of text. We present a Capsule-based convolutional neural network (CNN) model over character$n$-grams for performing the AA task. Capsule with kervolutional neural networks (KNNs) has also been utilized for this task. We also present different analyses of our developed system, which improves the interpretability of our developed system. Heat-maps for different models illustrate the relevant text fragments for the prediction task. A standard Twitter data set is used for evaluating the performance of the developed systems. The experimental evaluation shows that capsule-based CNNs and capsule-based KNNs perform competitively and are able to outperform previous methods. The source codes and the supplementary file are available herehttps://github.com/chanchalIITP/AuthorIdentification.

Web page segmentation (WPS) aims to break a web page into different segments with coherent intra- and inter-semantics. By evidencing the morpho-dispositional semantics of a web page, WPS has traditionally been used to demarcate informative from non-informative content, but it has also evidenced its key role within the context of non-linear access to web information for visually impaired people. For that purpose, a great deal of ad hoc solutions have been proposed that rely on visual, logical, and/or text cues. However, such methodologies highly depend on manually tuned heuristics and are parameter-dependent. To overcome these drawbacks, principled frameworks have been proposed that provide the theoretical bases to achieve optimal solutions. However, existing methodologies only combine few discriminant features and do not define strategies to automatically select the optimal number of segments. In this article, we present a multi-objective clustering technique called MCS that relies on \( K \) -means, in which (1) visual, logical, and text cues are all combined in a early fusion manner and (2) an evolutionary process automatically discovers the optimal number of clusters (segments) as well as the correct positioning of seeds. As such, our proposal is parameter-free, combines many different modalities, does not depend on manually tuned heuristics, and can be run on any web page without any constraint. An exhaustive evaluation over two different tasks, where (1) the number of segments must be discovered or (2) the number of clusters is fixed with respect to the task at hand, shows that MCS drastically improves over most competitive and up-to-date algorithms for a wide variety of external and internal validation indices. In particular, results clearly evidence the impact of the visual and logical modalities towards segmentation performance.

Protein-protein interaction (PPI) plays essential roles in nearly all biological processes of living organisms. Identification of PPI helps in understanding the cellular pathways and complex structure of proteins. Most of the works on the prediction of PPI utilized one type of information, mainly sequence-based. With the recent development in deep learning technologies, capturing the diverse and sparse biological dataset's relevant features is possible. This paper proposes a framework that incorporates a multi-modal representation of proteins to predict the interaction between them. Current work utilizes 3D structure and Gene ontology (GO) information to generate the vector representations of proteins. The task of PPI is divided into two phases: feature generation and prediction. We use deep learning algorithms in both stages. We validate our approach on two datasets: Human and S. cerevisiae. The trained model achieves accuracy values of 97.94% and 95.33% on the human and S. cerevisiae test sets, respectively. The results obtained over these datasets illustrate the superiority of the proposed method as compared to state-of-the-art methods.

There has been significant progress in image captioning in recent years. The generation of video descriptions is still in its early stages; this is due to the complex nature of videos in comparison to images. Generating paragraph descriptions of a video is even more challenging. Amongst the main issues are temporal object dependencies and complex object-object relationships. Recently, many works are proposed on the generation of multi-sentence video descriptions. The majority of the proposed works are based on a two-step approach: 1) event proposals and 2) caption generation. While these approaches produce good results, they miss out on globally available information. Here we propose the use of global object proposals while generating the video captions. Experimental results on ActivityNet dataset illustrate that the use of global object proposals can produce more informative and correct captions. We also propose three scores to evaluate the object detection capacity of the generator. Comparison of captions generated by the proposed method and the state-of-the-art techniques proves the efficacy of the proposed method.

Hate speech has become a critical problem in all social media, leading to many hate crimes alongside affecting the mental and emotional well-being of affected individuals. This calls out for methods to detect online hate speech more than ever. While numerous architectures exist for hate speech detection in unimodal setup (i.e., either textual or visual) we have targeted the problem in the context of both text and images inspired by the real-world raw data which involves several modalities. We propose a multimodal hate speech classifier, called as Character Text Image Classifier (CTIC), which builds upon Bidirectional Encoder Representations from Transformers (BERT), Capsule Network, and EfficientNet involving four modalities, namely word embeddings, character embeddings, sentence embeddings, and images. We report the experiments performed upon our proposed model and several other models which have been tested in the process. We train our model with different sampling techniques, and selective training, upon a Twitter dataset, called MMHS150K, consisting of both texts and associated images. Our proposed multimodal approach attains better performance than the previous models constructed upon the dataset.

Depression is a severe psychological disorder that is experienced by a significant number of individuals across the globe. It greatly changes the way one thinks, triggering a constant decline in mood. Studies have shown that gender can act as a good indicator of depression. In this paper, we analyse the effects of gender information in the estimation of depression. We have carried out different experiments on the benchmark data set named Distress Analysis Interview Corpus - a Wizard of Oz (DAIC-WOZ). Concretely, we discovered that a) gender information substantially improves the performance of depression severity estimation, and b) adversarially learning to predict the depression score distributed by gender improves the performance of depression severity estimation.

In recent years, there has been an explosion in the number of research publications from various conferences and journals. This large volume of scholarly articles is an invaluable source of information and knowledge which need to be effectively summarized to be useful. In contrast to the base summary, an extended summary gives low-level and structured information about the scholarly articles. In this paper, we propose to develop an attention-based BiLSTM-CNN framework for the purpose of generating extended summaries of scientific articles. To validate our approach, we conduct experiments on the dataset published by LongSumm shared task 2020. The performances of all our systems were evaluated in terms of a well-known metric ROUGE. We also benchmark our model against baseline techniques and our results reveal that the proposed attention-based deep neural network outperforms other models with a significant margin.

Major Depressive Disorder is one of the most serious mental health concerns haunting human kind. While there have been enough global efforts to reduce clinical depression, it is still not sufficient considering the grim statistics in terms of mental health. With the lopsided distribution of Mental Health Professionals (MHPs) to patients, initiatives should be taken to develop automated systems to assist MHPs in combating this grievous mental illness. In this paper, we propose a Virtual Assistant (VA) acting as a first point of contact for depressed or discouraged users. The VA should be capable of generating motivational and affirmative responses, provide users with a safe environment to share thoughts and seek help and advice anonymously. For addressing these multiple aspects, we curate a large-scale dataset, MotiVAte, consisting of dyadic conversations between the depressed user and the VA (imparting hope and motivation). The proposed dataset is evaluated on state of the art generative models. Empirical results both automatic and human evaluation based are reported for the same. To the best of our knowledge, this is the first application of Natural Language Generation to mental health.

With the ever increasing complexity of the user queries in a multi-domain based task-oriented dialogue system, it is imperative to facilitate robust Spoken Language Understanding (SLU) modules that perform multiple tasks in an unified way. In this paper, we present a novel multi-task approach for the joint modelling of three tasks together, namely, Domain Classification, Intent Detection and Slot-Filling. We hypothesize with the intuition that the cross dependencies of all these three tasks mutually help each other towards their representations and classifications which further simplify the SLU module in a multi-domain scenario. Towards this end, we propose a BERT language model based multi-task framework utilizing capsule networks and conditional random fields for addressing the classification and sequence labeling problems, respectively, for different tasks. Experimental results indicate that the proposed multi-task model outperformed several strong baselines and its single task counterparts on three benchmark datasets of different domains and attained state-of-the-art results on different tasks.

The gender of a user plays a very important role in the development of the personalized online services. However, due to privacy, and anonymity, gender information is usually not available for many users. Since male and female users have differences in their message contents, the messages posted by users can be analyzed for finding their genders. Users with different genders express their emotions differently too. In the current paper, we introduce an emotion-aided gender prediction system. The intuition behind our approach is to predict the gender of a user based on emotional clues. The proposed approach consists of two neural network branches, one for gender and the other for emotion. Both networks share their bottom feature extraction module and are optimized within a multi-task learning framework. Gender and emotion networks are trained over our own annotated dataset having gender and emotion labels. Experimental results show the effectiveness of using emotion for predicting the gender of a user.

In current scenario, transfer learning or domain adaptation has been emerged as fruitful approach to handle problems of distribution mismatch between the training and test data. Standard machine learning techniques utilize labeled data for getting better performance. In practical scenario, due to scarcity of labeled data, the classifier trained on the source domain cannot be utilized efficiently for classifying the target domain data. So, there is a need to use the previously acquired knowledge from a related source domain to classify the information in the target domain. Previous works on single/multiple source domain adaptation have achieved substantial growth in tackling this concern. In this work, we have proposed a novel unsupervised multi-source based approach MSVDA for visual domain adaptation in which data from multiple labeled source domains are utilized to classify the information in the target domain containing unlabeled data. Our proposed approach MSVDA extends the existing Maximum Mean Discrepancy criteria across various source domains and also preserves the discriminative information of various source domains. Further, it learns an optimal classification that diminishes the empirical risk and enhances the consistency rate between the prediction function and the manifold. Various experiments on the two publicly available datasets with multiple-domain scenario settings (double-domain, triple-domain and quadruple-domain scenarios) have demonstrated the efficacy of our proposed method MSVDA over other existing methods.

Developing an adequate and human-like virtual agent has been one of the primary applications of artificial intelligence. In the last few years, task-oriented dialogue systems have gained huge popularity because of their upsurging relevance and positive outcomes. In real-world, users may not always have a predefined and rigid task goal beforehand; they upgrade/downgrade/change their goal component dynamically depending upon their utility value and agent's serving capability. However, existing virtual agents fail to incorporate this dynamic behavior, leading to either unsuccessful task completion or an ungratified user experience. The paper presents an end to end multimodal dialogue system for dynamic and co-operative goal setting, which incorporates i) a multi-modal semantic state representation in policy learning to deal with multi-modal inputs, ii) a goal manager module in a traditional dialogue manager for handling dynamic and goal unavailability scenarios effectively, iii) an accumulative reward (task/persona/sentiment) for task success, personalized persuasion and user-adaptive behavior, respectively. The obtained experimental results and the comparisons with baselines firmly establish the need and efficacy of the proposed system.

Tulika Saha, Apoorva Upadhyaya, Sriparna Saha, Pushpak Bhattacharyya. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Large amounts of multi-modal information online make it difficult for users to obtain proper insights. In this paper, we introduce and formally define the concepts of supplementary and complementary multi-modal summaries in the context of the overlap of information covered by different modalities in the summary output. A new problem statement of combined complementary and supplementary multi-modal summarization (CCS-MMS) is formulated. The problem is then solved in several steps by utilizing the concepts of multi-objective optimization by devising a novel unsupervised framework. An existing multi-modal summarization data set is further extended by adding outputs in different modalities to establish the efficacy of the proposed technique. The results obtained by the proposed approach are compared with several strong baselines; ablation experiments are also conducted to empirically justify the proposed techniques. Furthermore, the proposed model is evaluated separately for different modalities quantitatively and qualitatively, demonstrating the superiority of our approach.

The internet has been one of the fast-growing technologies allowing users to search for information, communicate and collaborate, social networking, promote business, and much more. There are over 1 billion sites on the world-wide web1. It is important to understand the user’s query and return the top results that are majorly related to the query. In Search Result Clustering, results returned by one or more search engines are clustered into meaningful groups.The Search Results Clustering problem is modelled as a multi-view clustering problem in this study. The problem is then solved using an optimization framework. Experiments have been performed considering various combinations of four views including word-mover distance, BERT, TFIDF, and universal sentence encoder. The multi-objective binary differential evolution (MBDE) approach is used to optimise multiple quality measures computed on clusters produced by different views at the same time. We considered a set of possible solutions where each solution represents partitioning information obtained using different views. An agreement index that checks the consistency of partitionings of multiple views is also optimised. To get a single ensembled partitioning after considering various views, consensus partitioning technique and generative modeling are explored. The proposed model can automatically detect the number of clusters for any query. Our approach outperforms by 3.33% F1-score on three widely-used datasets over state-of-the-art techniques.

The pandemic disease COVID-19, originated from the SARS-CoV-2 virus has spread globally. Researchers are working tirelessly on areas including studying the transmission of COVID-19, promoting its identification, designing new vaccines and therapies, and recognizing its socio-economic consequences. This extensive research leads to the exploration of thousands of scientific papers related to biology, chemistry, genetics, health, and economy. Therefore, it is essential to develop an intelligent text mining technique for segregating this rich source of data to perform easy access, information retrieval, and interpretation within minimum time and resources. We propose a multi-objective optimization-based document clustering approach for the CORD-19 (COVID-19 Open Research Dataset) dataset in this paper. Here, a new technique utilizing BioBERT has been proposed, which benefits from the abstract and the document text, rather than only the brief abstract, to perceive a concise understanding of the text to generate clusters with better definitions. The main contributions of the proposed work are two-fold: in the first step, we have used BioBERT to generate the sentence embedding which is further used for the document representation. In the next step, we have developed a multi-objective optimization (MOO) based clustering algorithm for grouping the generated document vector representations. In this MOO-based clustering, we have used Non-dominated Sorting Genetic Algorithm-II and Fuzzy c-means algorithm as the underlying MOO and clustering technique, respectively. This model is evaluated using the Silhouette Score (Silhouette score) and Calinski-Harabasz index (CH index), and the clustering solutions are visualized using word clouds. The clustering results exhibit significant improvements over various other existing clustering models.

Much work has been done on dense image captioning in the English language. In this paper, we propose a Encoder-Decoder architecture for the first time, to the best of our knowledge, to generate Hindi dense image captions. We leverage the proven encoder-decoder architecture and provide a Hindi data-set along with results of dense captioning on the data-set. We also propose a novel post-processing technique to further improve the quality of captions by using multiple discriminators (language discriminator, length penalty and dissimilarity score). Hindi is the fourth most spoken language in the world. It is the official language of India and to the best of our knowledge, this is the first attempt at dense image captioning in the Hindi language. The data-set is manually created by translating Visual Genome captions from English to Hindi under manual supervision. Both qualitative (in terms of adequacy and fluency) and quantitative (in terms of BLEU scores) analysis of the generated captions reveal the effectiveness of our proposed approach compared to several strong baselines.

Image captioning is a multi-modal problem linking computer vision and natural language processing, which combines image analysis and text generation challenges. In the literature, most of the image captioning works have been accomplished in the English language only. This paper proposes a new approach for image captioning in the Hindi language using deep learning-based encoder-decoder architecture. Hindi, widely spoken in India and South Asia, is the fourth most spoken language globally; it is India’s official language. In recent years, significant advancement has been made in image captioning, utilizing encoder-decoder architectures based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Encoder CNN extracts features from input images, whereas decoder RNN performs language modeling. The proposed encoder-decoder architecture utilizes information multiplexing in the encoder CNN to achieve a performance gain in feature extraction. Extensive experimentation is carried out on the benchmark MSCOCO Hindi dataset, and significant improvements in BLEU score are reported compared to the baselines. Manual human evaluation in terms of adequacy and fluency of the generated captions further establishes the proposed method’s efficacy in generating good quality captions.

Software developers frequently use existing bug text reports to help them in understanding the different aspects of the specific defects and changes made to resolve that bug. But, bug reports are usually lengthy in nature and require considerable effort in understanding. In this direction, summarization of bug reports seems to be useful, covering relevant and diversified information. In the current article, we investigate the use of Multi-objective Evolutionary Algorithm (MEA) for BRS and thus, we name our approach as MEABRS. For MEA, we utilize the search capability of multi-objective bi-nary differential evolution (MOBDE) where we simultaneously optimize different aspects of BRS including diversity among sentences, sentence relevance using term weighting scheme, and length of the sentence. A keyword-based objective function is also incorporated in our optimization process to improve the quality of a bug report’s summary and in order to do so, rapid automatic keyword extraction (RAKE) toolkit is utilized. The generated summaries are evaluated with the available gold summaries corresponding to the two benchmark datasets (ADS and SDS) in terms of precision, recall, F-measure, and ROUGE measure. Results obtained demonstrate the efficacy of MEABRS with an average improvement (over both datasets) of 4.5% in terms of F1-Measure. Further, results are also validated using a statistical significance t-test.

Image captioning is the process of generating a textual description of an image that aims to describe the salient parts of the given image. It is an important problem, as it involves computer vision and natural language processing, where computer vision is used for understanding images, and natural language processing is used for language modeling. A lot of works have been done for image captioning for the English language. In this article, we have developed a model for image captioning in the Hindi language. Hindi is the official language of India, and it is the fourth most spoken language in the world, spoken in India and South Asia. To the best of our knowledge, this is the first attempt to generate image captions in the Hindi language. A dataset is manually created by translating well known MSCOCO dataset from English to Hindi. Finally, different types of attention-based architectures are developed for image captioning in the Hindi language. These attention mechanisms are new for the Hindi language, as those have never been used for the Hindi language. The obtained results of the proposed model are compared with several baselines in terms of BLEU scores, and the results show that our model performs better than others. Manual evaluation of the obtained captions in terms of adequacy and fluency also reveals the effectiveness of our proposed approach. Availability of resources : The codes of the article are available at https://github.com/santosh1821cs03/Image_Captioning_Hindi_Language ; The dataset will be made available: http://www.iitp.ac.in/∼ai-nlp-ml/resources.html .

Building Virtual Agents capable of carrying out complex queries of the user involving multiple intents of a domain is quite a challenge, because it demands that the agent manages several subtasks simultaneously. This article presents a universal Deep Reinforcement Learning framework that can synthesize dialogue managers capable of working in a task-oriented dialogue system encompassing various intents pertaining to a domain. The conversation between agent and user is broken down into hierarchies, to segregate subtasks pertinent to different intents. The concept of Hierarchical Reinforcement Learning, particularly options , is used to learn policies in different hierarchies that operates in distinct time steps to fulfill the user query successfully. The dialogue manager comprises top-level intent meta-policy to select among subtasks or options and a low-level controller policy to pick primitive actions to communicate with the user to complete the subtask provided to it by the top-level policy in varying intents of a domain. The proposed dialogue management module has been trained in a way such that it can be reused for any language for which it has been developed with little to no supervision. The developed system has been demonstrated for “Air Travel” and “Restaurant” domain in English and Hindi languages. Empirical results determine the robustness and efficacy of the learned dialogue policy as it outperforms several baselines and a state-of-the-art system.

The rising usage of social media has motivated to invent different methodologies of anonymous writing, which leads to an increase in malicious and suspicious activities. This anonymity has created difficulty in finding the suspect. Author profiling deals with the characterization of an author through some key attributes such as gender, age, language, dialect region variety, personality, and so on. Identifying the gender of the author of a suspect document is a salient task of author-profiling. The linguistic profile of a user can help in determining his/her demographics. Different social media platforms, such as Twitter, Facebook, and Instagram, are used regularly by users for sharing their daily life activities. Moreover, users often post images along with text on different social media platforms; thus, the usage of multimodal information is very common nowadays. In this article, the task of automatic gender prediction from multimodal Twitter data is posed as a classification problem and an efficient multimodal neural framework is proposed for solving this. The popularly used BERT_base is utilized for learning the encoded representation for the text part of the tweet, and recently introduced EfficientNet is used for extracting the features from images. Finally, a direct product-based fusion strategy is applied for fusing the text and image representations, followed by a fully connected layer for predicting the gender of a Twitter user. Plagiarism detection authorship analysis near end duplicate detection (PAN)-2018 author profiling data are used for evaluating the performance of our proposed approach. Our proposed model achieved accuracies of 82.05%, 86.22%, and 89.53% for pure-image, pure-text, and multimodal setting, respectively; outperforming the previous state-of-the-art works in all the cases. Moreover, a deep analysis is carried out to interpret the produced results; different words that serve as clues for gender classification are identified characterizing different gender classes. The supplementary file and the source codes for the proposed approach are available athttps://github.com/chanchalIITP/GenderTCSS.

Protein is an essential macro-nutrient for perceiving a wide range of biochemical activities and biological regulations in living cells. In this work, we have presented a novel multi-modal approach, named MultiPredGO, for predicting protein functions by utilizing two different kinds of information, namely protein sequence and the protein secondary structure. Here, our contributions are threefold; firstly, along with the protein sequence, we learn the feature representation from the protein structure. Secondly, we develop two different deep learning models after considering the characteristics of the underlying data patterns of the protein sequence and protein 3D structures. Finally, along with these two modalities, we have also utilized protein interaction information for expediting the efficiency of the proposed model in predicting the protein functions. For extracting features from different modalities, we have utilized various variations of the convolutional neural network. As the protein function classes are dependent on each other, we have used a neuro-symbolic hierarchical classification model, which resembles the structure of Gene Ontology (GO), for effectively predicting the dependent protein functions. Finally, to validate the goodness of our proposed method (MultiPredGO), we have compared our results with various uni-modal along with two well-known multi-modal protein function prediction approaches, namely, INGA and DeepGO. Results show that the overall performance of the proposed approach in terms of accuracy, F-measure, precision, and recall metrics are better than those by the state-of-the-art methods. MultiPredGO attains an average 13.05% and 30.87% improvements over the best existing comparing approach (DeepGO) for cellular component and molecular functions, respectively.

The feature selection method is the process of selecting only relevant features by removing irrelevant or redundant features amongst the large number of features that are used to represent data. Nowadays, many application domains especially social media networks, generate new features continuously at different time stamps. In such a scenario, when the features are arriving in an online fashion, to cope up with the continuous arrival of features, the selection task must also have to be a continuous process. Therefore, the streaming feature selection based approach has to be incorporated, i.e., every time a new feature or a group of features arrives, the feature selection process has to be invoked. Again, in recent years, there are many application domains that generate data where samples may belong to more than one classes called multi-label dataset. The multiple labels that the instances are being associated with, may have some dependencies amongst themselves. Finding the co-relation amongst the class labels helps to select the discriminative features across multiple labels. In this article, we develop streaming feature selection methods for multi-label data where the multiple labels are reduced to a lower-dimensional space. The similar labels are grouped together before performing the selection method to improve the selection quality and to make the model time efficient. The multi-objective version of the cuckoo search-based approach is used to select the optimal feature set. The proposed method develops two versions of the streaming feature selection method: ) when the features arrive individually and ) when the features arrive in the form of a batch. Various multi-label datasets from various domains such as text, biology, and audio have been used to test the developed streaming feature selection methods. The proposed methods are compared with many previous feature selection methods and from the comparison, the superiority of using multiple objectives and label co-relation in the feature selection process can be established.

An in-depth exploration of protein-protein interactions (PPI) is essential to understand the metabolism in addition to the regulations of biological entities like proteins, carbohydrates, and many more.Most of the recent PPI tasks in BioNLP domain have been carried out solely using textual data.In this paper, we argue that incorporation of multimodal cues can improve the automatic identification of PPI.As a first step towards enabling the development of multimodal approaches for PPI identification, we have developed two multimodal datasets which are extensions and multimodal versions of two popular benchmark PPI corpora (BioInfer and HRPD50).Besides, existing textual modalities, two new modalities, 3D protein structure and underlying genomic sequence, are also added to each instance.Further, a novel deep multi-modal architecture is also implemented to efficiently predict the protein interactions from the developed datasets.A detailed experimental analysis reveals the superiority of the multi-modal approach in comparison to the strong baselines including uni-modal approaches and state-of the-art methods over both the generated multimodal datasets.The developed multi-modal datasets are available for use at https:// github.com/sduttap16/MM_PPI_NLP.

The task of Dialogue Act Classification (DAC) that purports to capture communicative intent has been studied extensively.But these studies limit themselves to text.Non-verbal features (change of tone, facial expressions etc.) can provide cues to identify DAs, thus stressing the benefit of incorporating multi-modal inputs in the task.Also, the emotional state of the speaker has a substantial effect on the choice of the dialogue act, since conversations are often influenced by emotions.Hence, the effect of emotion too on automatic identification of DAs needs to be studied.In this work, we address the role of both multi-modality and emotion recognition (ER) in DAC.DAC and ER help each other by way of multi-task learning.One of the major contributions of this work is a new dataset-multimodal Emotion aware Dialogue Act dataset called EMOTyDA, collected from open-sourced dialogue datasets.To demonstrate the utility of EMOTyDA, we build an attention based (self, inter-modal, inter-task) multi-modal, multi-task Deep Neural Network (DNN) for joint learning of DAs and emotions.We show empirically that multimodality and multi-tasking achieve better performance of DAC compared to uni-modal and single task DAC variants.

Gene is a fundamental, physical and functional unit of hereditary. A proper grouping of genes is needed for a better understanding of the natural patterns amongst them. Most of the existing approaches utilize gene expression values for the clustering of genes. They mostly ignore the semantic similarity between genes obtained from the global database like Gene Ontology(GO). We present a multi-view multi-objective clustering approach where Euclidean distances between the gene expression values and GO-based multi-factored gene-gene semantic similarity are considered as two complementary views and a consensus partitioning is obtained that satisfies both the aspects or views. Two real-life gene expression datasets are used to demonstrate the effectiveness of the proposed multi-view clustering technique. We have compared our approach with various standard single-view, multi-view and multi-objective optimization based clustering approaches. The results show that better co-expressed and biologically significant gene partitions are obtained by our proposed approach. Also, to validate the significance of the obtained clusters statistically and biologically, various biological, statistical and visual significance tests were conducted, and the corresponding results are reported.

Nowadays, micro-blogging sites are getting popular due to the involvement of a large number of users. In the case of natural disasters, a significant amount of relevant information (giving crucial information) are present amongst the tweets. Therefore, there is a need to develop a system that summarizes relevant tweets by extracting informative tweets. In the current paper, we have proposed an unsupervised approach for summarizing the relevant tweets namely, MOOTweetSumm+, which automatically selects the informative tweets. Several tweet-scoring measures: (a) anti-redundancy measuring the dissimilarity between tweets; (b) similarity with outputs provided by LexRank (a graph-based method measuring tweet importance based on the concept of eigen-vector centrality in a graph); (c) BM25 based ranking function; (d) tf-idf based ranking function; (e) length of the tweet; (f) re-tweet count, are simultaneously optimized utilizing a binary differential evolution algorithm. Further, two different versions of the LexRank, utilizing syntactic and semantic similarity, have also been explored. For evaluation, four different disaster-event related datasets are used, and performance is measured in terms of ROUGE scores. An ablation study is also performed to determine which set of measures is best suited for different datasets. From the results obtained, it is clearly evident that our approach improves by 13.2% and 5.8% in terms of ROUGE-2 and ROUGE-L scores, over the existing approaches, respectively.

Cuckoo search (CS) is a highly competitive single objective optimization technique. The algorithm has been widely applied in various diverse application domains and has been found to be efficient in solving various real-life problems. In the present work, we have proposed a new enhanced version of CS algorithm and tested its performance on recently proposed CEC 2017 and CEC 2020 benchmark test problems. The proposed algorithm has been named as CSsin and it employs four major modifications, i) new techniques for global and local search are devised, ii) dual search strategy is followed to enhance exploration and exploitation properties of CS algorithm, iii) a linearly decreasing switch probability has been used to add a balance between local and global search, and iv) linearly decreasing population size is used to reduce the computational burden. Apart from these modifications, the division of iterations has been employed as a further modification. The CSsin algorithm has been tested on IEEE CEC 2017 and CEC 2020 benchmark test problems having various dimension sizes and a comparative study has been performed with respect to state-of-the-art optimization algorithms for single objective bound constraint optimization problems. The results of statistical significance test affirm the competitiveness of the proposed algorithm with respect to state-of-the-art techniques.

In recent years, multi-view multi-task clustering has received much attention. There are several real-life problems that involve both multi-view as well as multi-task clustering, i.e., the tasks are closely related, and each task can be analyzed using multiple views. Traditional multi-task multi-view clustering algorithms utilize single-objective optimization based approaches and cannot apply too-many regularization terms. However, these problems are inherently some multi-objective optimization problems because conflict may be between different views within a given task and also between different tasks, necessitating a trade-off. Based on these observations, in this paper, we have proposed a novel multi-task multi-view multiobjective optimization (MTMV-MO) based clustering algorithm which simultaneously optimizes three objectives, i.e., within-view task relation, within-task view relation and the quality of the clusters obtained. The proposed methodology (MTMV-MO) is evaluated on four different datasets and the results are compared with five state-of-the-art algorithms in terms of Adjusted Rand Index (ARI) and Classification Accuracy (%CoA). MTMV-MO illustrates an improvement of 1.5-2% in terms of ARI and 4-5% in terms of %CoA compared to the state-of-the-art algorithms.

The rate of publishing scientific articles is increasing day by day which has created difficulty for the researchers to learn about the recent advancements in a faster way. Also, relying on the abstract of these published articles is not a good idea as they cover only broad ideas of the article. The summarization of scientific documents (SDS) addresses this challenge. In this paper, we propose a system for SDS having two components: identifying the relevant sentences in the article using citation context; generation of the summary by posing SDS as a binary optimization problem. For the purpose of optimization, a metaheuristic evolutionary algorithm is utilized. In order to improve the quality of summary, various aspects measuring the relevance of sentences are simultaneously optimized using the concept of multi-objective optimization. Inspired by the popularity of graph-based algorithms like LexRank which is popularly used in solving summarization problems of different real-life applications, its impact is studied in fusion with our optimization framework. An ablation study is also performed to identify the most contributing aspects for the summary generation. We investigated the performance of our proposed framework on two datasets related to the computational linguistic domain, CL-SciSumm 2016 and CL-SciSumm 2017, in terms of ROUGE measures. The results obtained illustrate that our framework effectively improves other existing methods. Further, results are validated using the statistical paired t-test.

The unprecedented growth of Internet users has resulted in an abundance of unstructured information on social media including health forums, where patients request health-related information or opinions from other users. Previous studies have shown that online peer support has limited effectiveness without expert intervention. Therefore, a system capable of assessing the severity of health state from the patients' social media posts can help health professionals (HP) in prioritizing the user's post. In this study, we inspect the efficacy of different aspects of Natural Language Understanding (NLU) to identify the severity of the user's health state in relation to two perspectives(tasks) (a) Medical Condition (i.e., Recover, Exist, Deteriorate, Other) and (b) Medication (i.e., Effective, Ineffective, Serious Adverse Effect, Other) in online health communities. We propose a multiview learning framework that models both the textual content as well as contextual-information to assess the severity of the user's health state. Specifically, our model utilizes the NLU views such as sentiment, emotions, personality, and use of figurative language to extract the contextual information. The diverse NLU views demonstrate its effectiveness on both the tasks and as well as on the individual disease to assess a user's health.

The purpose of an image paragraph captioning model is to produce detailed descriptions of the source images. Generally, paragraph captioning models use encoder-decoder based architectures similar to the standard image captioning models. The encoder is a CNN based model, and the decoder is a LSTM or GRU. The standard image captioning models produce unsatisfactory results for the paragraph captioning task due to the lack of diversity in the generated outputs [9]. The paragraphs generated from standard image captioning models lack in language diversity and contain redundant information. In this work, we have proposed an approach with language discriminator for increasing the diversity in language, and dissimilarity score using word mover's distance [4] for reducing redundant information. Using this approach with a state-of-the-art model at testing time, we have improved the METEOR score from 13.63 to 19.01 for the Visual Genome dataset.

Development of Virtual Agents (VAs) for Goal/Task-oriented conversations capable of handling complex tasks pertaining to multiple domains and its various intents is quite an onerous task. Lack of high quality, domain specific conversational data required to train policies is one of the biggest challenges for the success of any dialogue system. In this paper, we present a multi-domain, multi-intent based task-oriented dialogue system by successfully combining Hierarchical Deep Reinforcement Learning and Transfer Learning paradigms. The notion is to exploit or take advantage of the resemblance between domains as various domains share considerable amount of overlapping data or slots. Thus, Options framework along with Transfer Learning is employed to curate VAs with better and faster learning performance. Our proposed approach reduced the data requirement to train multi-domain VAs by atleast 20% for distant domains and almost 38% for close domains. It also significantly curtailed the learning time and aided faster learning for transfer learning based policies.

Speech acts help in uncovering and understanding the communicative intention and behavior of a speaker utterance. This is pertinent for communication on any platform that embodies social media platforms like Twitter. This paper presents a supervised speech act (tweet act in our case) classifier for tweets for assessing the content and intent of tweets, thereby, exploring the valuable communication amongst the tweeters. With the recent success of Bi-directional Encoder Representations from Transformers (BERT), a newly introduced language representation model that provides pre-trained deep bi-directional representations of vast unlabeled data, we introduce BERT-extended that is built on top of BERT. Our model is based on calculating attention weights over the representations of tokens of a sequence to identify tweet acts. The proposed model attained a benchmark accuracy of 75.97% and outperformed several strong baselines and state-of-the-art approaches on an open-source, tweet act annotated Twitter dataset.

Rigorous mathematical investigation of learning rates used in back-propagation in shallow neural networks has become a necessity. This is because experimental evidence needs to be endorsed by a theoretical background. Such theory may be helpful in reducing the volume of experimental effort to accomplish desired results. We leveraged the functional property of Mean Square Error, which is Lipschitz continuous to compute learning rate in shallow neural networks. We claim that our approach reduces tuning efforts, especially when a significant corpus of data has to be handled. We achieve remarkable improvement in saving computational cost while surpassing prediction accuracy reported in literature. The learning rate, proposed here, is the inverse of the Lipschitz constant. The work results in a novel method for carrying out gene expression inference on large microarray data sets with a shallow architecture constrained by limited computing resources. A combination of random sub-sampling of the dataset, an adaptive Lipschitz constant inspired learning rate and a new activation function, A-ReLU helped accomplish the results reported in the paper.

Significant development of communication technology over the past few years has motivated research in multi-modal summarization techniques. A majority of the previous works on multi-modal summarization focus on text and images. In this paper, we propose a novel extractive multi-objective optimization based model to produce a multi-modal summary containing text, images, and videos. Important objectives such as intra-modality salience, cross-modal redundancy and cross-modal similarity are optimized simultaneously in a multi-objective optimization framework to produce effective multi-modal output. The proposed model has been evaluated separately for different modalities, and has been found to perform better than state-of-the-art approaches.

To describe the cellular functions of proteins and genes, a potential dynamic vocabulary is Gene Ontology (GO), which comprises of three sub-ontologies namely, Biological-process, Cellular-component, and Molecular-function. It has several applications in the field of bioinformatics like annotating/measuring gene-gene or protein-protein semantic similarity, identifying genes/proteins by their GO annotations for disease gene and target discovery, etc. To determine semantic similarity between genes, several semantic measures have been proposed in literature, which involve information content of GO-terms, GO tree structure, or the combination of both. But, most of the existing semantic similarity measures do not consider different topological and information theoretic aspects of GO-terms collectively. Inspired by this fact, in this article, we have first proposed three novel semantic similarity/distance measures for genes covering different aspects of GO-tree. These are further implanted in the frameworks of well-known multi-objective and single-objective based clustering algorithms to determine functionally similar genes. For comparative analysis, 10 popular existing GO based semantic similarity/distance measures and tools are also considered. Experimental results on Mouse genome, Yeast, and Human genome datasets evidently demonstrate the supremacy of multi-objective clustering algorithms in association with proposed multi-factored similarity/distance measures. Clustering outcomes are further validated by conducting some biological/statistical significance tests. Supplementary information is available at https://www.iitp.ac.in/sriparna/journals.html.

Cluster ensemble techniques aim to combine the outputs of multiple clustering algorithms to obtain a single consensus partitioning. The current paper reports about the development of a cluster ensemble based technique combining the concepts of multiobjective optimization and deep-learning models for gene clustering where some additional protein-protein interaction information are utilized for generating the consensus partitioning. The proposed ensemble based framework works in four phases: (i) filtering out the irrelevant genes from the microarray dataset: only the statistically significant genes are considered for further data analysis; (ii) generation of diverse base partitionings: a multi-objective optimization-based clustering technique is proposed which simultaneously optimizes three different cluster quality measures and generates a set of partitioning solutions on the Pareto optimal front; (iii) generation of a consensus partitioning: mentha scores, calculated by accessing a highly enriched protein-protein interaction archive named mentha, of different clustering solutions are considered for generating a weighted incidence matrix; (iv) finally, two approaches are used to generate a consensus partitioning from the obtained incidence matrix. The first approach is based on a traditional machine learning method, and another approach exploits the graph partitioning algorithm and two deep neural models to generate the final clustering. To validate the efficacy of the proposed ensemble framework, it is applied on five gene expression datasets. We present a comparative analysis of the proposed technique over different clustering algorithms in terms of biological homogeneity index (BHI) and biological stability index (BSI). The traditional approach attains an average 3 and 2 percent improvements over the best non-dominated solution with respect to BHI and BSI, respectively, whereas deep learning models illustrate an average 6.8 and 1.5 percent improvements over the proposed traditional approach with respect to BHI and BSI, respectively. Subsequently, Welch's t-test is executed to prove that the results obtained by the proposed methods are statistically significant. Availability of data and materials: https://github.com/sduttap16/DeepEnsm.

Identification of speech acts provides essential cues in understanding the pragmatics of a user utterance. It typically helps in comprehending the communicative intention of a speaker. This holds true for conversations or discussions on any fora, including social media platforms, such as Twitter. This article presents a novel tweet act classifier (speech act for Twitter) for assessing the content and intent of tweets, thereby exploring the valuable communication among the tweeters. With the recent success of Bidirectional Encoder Representations from Transformers (BERT), a newly introduced language representation model that provides pretrained deep bidirectional representations of vast unlabeled data, we introduce BERT-Caps that is built on top of BERT. The proposed model tends to learn traits and attributes by leveraging from the joint optimization of features from the BERT and capsule layer to develop a robust classifier for the task. Some Twitter-specific symbols are also included in the model to observe its influence and importance. The proposed model attained a benchmark accuracy of 77.52% and outperformed several strong baselines and state-of-the-art approaches.

Understanding the chemical-disease relations (CDR) is a crucial task in various biomedical domains. Manual mining of these information from biomedical literature is costly and time-consuming. To address these issues, various researches have been carried out to design an efficient automatic tool. In this paper, we propose a multi-view based deep neural network model for CDR task. Typically, multiple representations (or views) of the datasets are not available for this task. So, we train multiple conceptually different deep neural network models on the dataset to generate different abstract features, treated as different views. A novel loss function, "Penalized LF", is defined to address the problem of imbalance dataset. The proposed loss function is generic in nature. The model is designed as a combination of Convolution Neural Network (CNN) and Bidirectional Long Short Term Memory (Bi-LSTM) network along with a Multi-Layer Perceptron (MLP). To show the efficacy of our proposed model, we have compared it with six baseline models and other state-of-the-art techniques, on "chemicals-and-disease-DFE" dataset, a free text dataset created by Li et al. from BioCreative V Chemical Disease Relation dataset. Results show that the proposed model attains highest F1-score for individual classes, proving its efficiency in handling class imbalance problem in the dataset. To further demonstrate the efficacy of the proposed model, we have presented results on BioCreative V dataset and two Protein-Protein Interaction Identification (PPI) datasets, viz., AiMed and BioInfer. All these results are also compared with the state-of-the-art models.

There is a large body of works on multi-view clustering that exploit multiple representations (or views) of the same input data for better convergence. These multiple views can come from multiple modalities (image, audio, text) or different feature subsets. Obtaining one consensus partitioning after considering different views is usually a non-trivial task. Recently, multi-objective based multi-view clustering methods have suppressed the performance of single objective based multi-view clustering techniques. One key problem is that it is difficult to select a single solution from a set of alternative partitionings generated by multi-objective techniques on the final Pareto optimal front. In this article, we propose a novel multi-objective based multi-view clustering framework that overcomes the problem of selecting a single solution in multi-objective based techniques. In particular, our proposed framework has three major components as follows: (i) multi-view based multi-objective algorithm, Multiview-AMOSA, for initial clustering of data points; (ii) a generative model for generating a combined solution having probabilistic labels; and (iii) K -means algorithm for obtaining the final labels. As the first component, we have adopted a recently developed multi-view based multi-objective clustering algorithm to generate different possible consensus partitionings of a given dataset taking into account different views. A generative model is coupled with the first component to generate a single consensus partitioning after considering multiple solutions. It exploits the latent subsets of the non-dominated solutions obtained from the multi-objective clustering algorithm and combines them to produce a single probabilistic labeled solution. Finally, a simple clustering algorithm, namely K -means, is applied on the generated probabilistic labels to obtain the final cluster labels. Experimental validation of our proposed framework is carried out over several benchmark datasets belonging to three different domains; UCI datasets, multi-view datasets, search result clustering datasets, and patient stratification datasets. Experimental results show that our proposed framework achieves an improvement of around 2%--4% over different evaluation metrics in all the four domains in comparison to state-of-the art methods.

This article proposes a novel unsupervised approach (FigSum++) for automatic figure summarization in biomedical scientific articles using a multi-objective evolutionary algorithm. The problem is treated as an optimization problem where relevant sentences in the summary for a given figure are selected based on various sentence scoring features (or objective functions), such as the textual entailment score between sentences in the summary and a figure’s caption, the number of sentences referring to that figure, semantic similarity between sentences and a figure’s caption, and the number of overlapping words between sentences and a figure’s caption. These objective functions are optimized simultaneously using multi-objective binary differential evolution (MBDE). MBDE consists of a set of solutions, and each solution represents a subset of sentences to be selected in the summary. MBDE generally uses a single differential evolution variant, but in the current study, an ensemble of two different differential evolution variants measuring diversity among solutions and convergence toward global optimal solution, respectively, is employed for efficient search. Usually, in any summarization system, diversity among sentences (called anti-redundancy ) in the summary is a very critical feature, and it is calculated in terms of similarity (like cosine similarity) among sentences. In this article, a new way of measuring diversity in terms of textual entailment is proposed. To represent the sentences of the article in the form of numeric vectors, the recently proposed BioBERT pre-trained language model in biomedical text mining is utilized. An ablation study has also been presented to determine the importance of different objective functions. For evaluation of the proposed technique, two benchmark biomedical datasets containing 91 and 84 figures are considered. Our proposed system obtains 5% and 11% improvements in terms of the F -measure metric over two datasets, compared to the state-of-the-art unsupervised methods.

Event extraction is one of the crucial tasks in biomedical text mining that aims to extract specific information concerning incidents embedded in the texts. In this article, we propose a deep learning framework that aims to identify the attributes (severity, course, temporal expression, and document creation time) associated with the medical concepts extracted from electronic medical records. The bi-directional long short-term memory network assisted by the attention mechanism is utilized to uncover the important aspects of the patient’s medical conditions. The attention mechanism specific to the medical disorder mention can focus on various parts of the sentence when different disorders are considered as input. The proposed methodology is evaluated on benchmark ShARe/CLEF eHealth Evaluation Lab 2014 shared task 2 datasets. In addition to the CLEF dataset, we also used the social media text, especially the medical blog posts. Experimental results of the proposed approach illustrate that our proposed approach achieves significant performance improvements over the state-of-the-art techniques and the highly competitive deep learning--based baseline methods.

The mining of adverse drug reaction (ADR) has a crucial role in the pharmacovigilance.The traditional ways of identifying ADR are reliable but time-consuming, non-scalable and offer a very limited amount of ADR relevant information.With the unprecedented growth of information sources in the forms of social media texts (Twitter, Blogs, Reviews etc.), biomedical literature, and Electronic Medical Records (EMR), it has become crucial to extract the most pertinent ADR related information from these free-form texts.In this paper, we propose a neural network inspired multitask learning framework that can simultaneously extract ADRs from various sources.We adopt a novel adversarial learning-based approach to learn features across multiple ADR information sources.Unlike the other existing techniques, our approach is capable to extracting fine-grained information (such as 'Indications', 'Symptoms', 'Finding', 'Disease', 'Drug') which provide important cues in pharmacovigilance.We evaluate our proposed approach on three publicly available realworld benchmark pharmacovigilance datasets, a Twitter dataset from PSB 2016 Social Media Shared Task, CADEC corpus and Medline ADR corpus.Experiments show that our unified framework achieves state-of-the-art performance on individual tasks associated with the different benchmark datasets.This establishes the fact that our proposed approach is generic, which enables it to achieve high performance on the diverse datasets.The source code is available here 1 .

The current paper addresses the online feature selection problem in multi-label classification framework where multi-labelled data with features arriving in an online fashion is considered as input. The proposed approach works in two phases, in the first phase, the best subset of features is selected from the initial available set of features using a multi-objective optimization (MOO) based feature selection technique. In the second phase of the proposed feature selection technique, a newly arrived feature is accepted or rejected based on redundancy with respect to the already selected set of features and relevancy of the arrived feature with respect to the class labels. In order to show the efficacy of the proposed algorithm, it is tested on 7 various types of multi-label datasets of different domains such as text, biology, and audio. The obtained results outperform the results obtained by state-of-the-art approaches in majority of the cases.

In the field of computational bioinformatics, for tasks such as medical diagnosis or disease gene classification, acquiring class labels is very much essential and costly. Now a days, labelling biomedical data became one of the crucial bottlenecks for developing supervised machine learning systems, as well as deep learning systems. On the contrary, weak supervision sources that generate diverse, semi-accurate or programmatically-generated labels are comparatively cheaper and less time consuming. In this paper, we have proposed a framework for integrating weak supervision sources with a generative model to estimate probabilistic class labels of the gene expression data. Here as the weak supervision source, a multi-objective optimization (MOO) based clustering technique is utilized. The non-dominated solutions of Pareto optimal front are viewed as the weak supervised labels. These weak supervised labels are fed to a generative model, which finally assigns probabilistic class labels to different genes. In this work, we have shown that our proposed generative model based clustering technique attains better Silhouette score (some quality measure of clustering) for three real-life NCBI gene expression data sets than other stateof-the-art methods, without using any labeled data. Finally, the superiority of the proposed method is validated by using a statistical and a biological significance test.

In certain multi-objective optimization problems, it may happen that there are two or more different Pareto optimal sets corresponding to the exact same Pareto Front. These are known as multimodal multi-objective optimization problems (MMOPs). Algorithms which are able to generate all the Pareto sets can provide more flexibility in choosing solutions and therefore a possible improvement in the performance. In this work, we build upon the original framework of NAEMO (Neighborhood-sensitive Archived Evolutionary Many-objective Optimization Algorithm) and present Multimodal NAEMO (MM-NAEMO). In MM-NAEMO, the mutation occurs between the points which are local neighbors and each reference line always keeps at least two or more candidate solutions associated with it. Clustering is used to maintain the diversity in the population associated with each reference line, thus providing multiple Pareto sets as output. The algorithm is tested on the test problems of MMO test suite of CEC2019. The experimental results suggest that the proposed algorithm effectively finds majority of the Pareto sets without hampering the Pareto fronts for most of the problems.

Multi-label classification deals with the task of predicting multiple class labels for a given sample. Several performance metrics are designed in the literature to measure the quality of any multi-label classification technique. In general existing multi-label classification approaches focus on optimizing only a single performance measure. The current work builds on the hypothesis that a weighted ensemble of multiple multi-label classifiers will lead to obtain improved results. The appropriate weight combinations for combining the outputs of multiple classifiers can be selected after simultaneously optimizing different multi-label classification metrics like micro F1, hamming loss, 0/1 loss, accuracy, etc. The problem is posed as a multi-objective optimization problem where a set of 13 conflicting objective functions are simultaneously optimized using the search capability of a multi-objective genetic algorithm based technique, namely NSGA-II. The weights of votes for a classifier for different class labels vary over a range quantifying the degrees of confidence of a classifier for different class labels. Several base classifiers are utilized for solving the problem of multi-objective multi-label classification. The effectiveness of the proposed multi-objective based multi-label classifier ensemble approach is shown for 10 data sets of varying complexities, and comparisons are reported for several existing techniques. Obtained experimental results clearly illustrate the efficacy of the proposed technique.

Differential Evolution algorithm is very challenging algorithm and has been found to put forth the basis of evolutionary computation. This algorithm because of its simple structure and linear nature, has been applied to a large number of optimization problems from various diversified fields. In this paper, we propose a new variant of DE by modifying the original LSHADE-cnEpSin algorithm. Two new modifications are proposed, keeping all the modifications of LSHADE-cnEpSin intact. The modifications proposed include the introduction of adaptive parameters by using Weibull distribution based scaling factor and exponentially decreasing crossover rate. Apart from that linearly decreasing population size is also used. The main reason for these adaptations is to make an adaptive algorithm so that no parameter needs to be changed from the end user perspective. The proposed algorithm has been applied to solve CEC2017 and CEC2019 benchmark problems. The numerical results prove that the newly proposed SALSHADE-cnEpSin algorithm performs better than SaDE, JADE, SHADE, LSHADE, CV1.0, CVnew, MVMO and other algorithms.

Data such as videos, genetic information, etc. from real-world applications reside in a high dimensional space. Before performing classification, it is required to project the data from the high dimensional space to a lower dimensional space without losing too much information. Linear discriminant analysis (LDA) is one of the most widely used methods for dimensionality reduction, that maximizes the ratio of the between-class scatter and total data scatter in the projected space using the labeled information. However, in the real world scenario, labeled information is hardly ever available in large quantities, but an abundant amount of unlabeled data is available. In this paper, we propose a Semi-Supervised Discriminant Analysis method called SSDARD, which considers the unlabeled information in the form of a k-NN graph. Different from the existing semi-supervised dimensionality reduction algorithms, our algorithm is more consistent in propagating the label information from labeled data to unlabeled data due to the use of relative distance function instead of normal Euclidean distance function to generate the k-NN graph. To find an appropriate relative distance function, we use pairwise constraints generated from labeled data and satisfy them using Bregman projection. Since the projection is not orthogonal, we require an appropriate subset of constraints. In order to select such subset of constraints, we have further developed a framework called MO-SSDARD, which uses an evolutionary algorithm while optimizing various cluster validity indices simultaneously. The experimental results on various datasets show that our proposed method is superior than various methods concerning various validity indices.

Speech act provides valuable insight for understanding the communicative intention and behaviour of a user utterance. This applies for conversation on any platform which includes social media network such as Twitter. This paper presents a deep learning based tweet act classifier (speech acts for Twitter) for understanding the content and intention of tweets and for discovering the valuable communications amongst the tweeters. A generic taxonomy of seven speech acts has been created for the task. A Convolution Neural Network (CNN) based Deep Learning (DL) architecture is integrated with different classifiers at the final layer such as softmax or linear Support Vector Machine (SVM) to develop a tweet act classifier. To make the classifier even more robust, various hand-crafted features are added to the system to boost its effectiveness towards improving the performance of the system. Experimental results indicate that the proposed model achieved good results with an average F1-score of 0.71 and outperformed state of the art approaches.

Dialogue Act (DA) Classification plays a signifi-cant role in the understanding of an utterance in a dialogue. Components of Spoken Dialogue System (SDS) such as Natural Language Understanding (NLU) and Dialogue Management (DM) modules can significantly exploit the output of the DA classification. In this paper, we propose a task-oriented DA classifier based on both traditional supervised Machine Learning (ML) as well as Deep Learning (DL) techniques. The type and nature of dialogues basically depend on the domain and the DA itself. So, in order to make the model task-oriented, a new tag-set has been designed by studying the properties of the target domain. On the benchmark SwitchBoard (SWBD) and TRAINS corpus, our proposed models have performed exceptionally well with the new tag-set. Experimental results indicate that our proposed models have achieved good accuracy on both the datasets and outperformed several state of the art approaches and the new tag-set is well suited for task-oriented applications.

With the emergence of DNA microarray technology, scientists are continuously studying the expression levels of a large number of genes over different instances of time points. Analyzing the DNA microarray data confirm that the expression levels of two different genes vary simultaneously with the effect of external stimuli exhibiting different patterns. Therefore soft fuzzy clustering plays an important role in detecting patterns belonging to multiple clusters at the same time. Therefore this article proposes a novel fuzzy clustering technique utilizing weighted distance measure instead of the Euclidean distance using Grey Wolf Optimizer (GWO) as the global optimization techniques. Here clustering of microarray data is posed as a single objective optimization problem where the objective is to minimize the variability within the cluster and simultaneously maximizes the variability between the cluster. The newly proposed fuzzy-based weighted GWO clustering technique (Fuzzy-WDGWO) is then compared with some of the existing clustering techniques. Four different artificial datasets and three different real-life gene expression datasets have been considered to verify the efficiency of the proposed Fuzzy-WDGWO clustering technique both numerically as well as pictorially. Experimental analysis and performance evaluation of the proposed Fuzzy-WDGWO clustering technique show the superiority over the other existing clustering techniques such as PSO-FCM, FA-FCM, GWO-FCM, GWO-FCM, DE-FCM.

Entity matching aims at mapping records to different entities where records share the common entity name. The entity matching problem is challenging because several times records do not contain complete information (many of the attributes are missing), unequal distribution of records for different entities, big overlaps between records of different entities. In this paper, we have proposed an unsupervised framework to solve this problem. The aforementioned problem is posed as a partitioning problem. Three unknown artifacts for solving this partitioning problem: optimal partitioning including the optimal number of partitions, suitable distance measure which can be utilized to measure the distance between records and a set of attributes/features which can take part in the distance calculation, are determined automatically using the search capability of a multiobjective optimization technique. Several objective functions which help in measuring the goodness of partitioning are optimized simultaneously by some popular multiobjective optimization techniques, NSGA-II/NSGA-III (Non-dominated Sorting Genetic Algorithm-II/III) to solve the partitioning problem. Total 247 combinations of eight different objective functions are used in the experiments for partitioning fourteen bibliographic datasets. A detailed comparative study of the proposed approach using NSGA-II and NSGA-III.

Intrusion Detection Systems (IDS) has become an indespensive tool to protect the network by detecting the attacks. As each attack is associated with a large number of attributes, it is challenging to select a good set of attributes for achieving better classification accuracy. This work proposes an effective unsupervised feature selection technique using Genetic algorithm (GA) for analyzing IDS data. The search capability of GA has been utilized for optimizing different unsupervised feature quality measures including Pearson correlation, mutual information, and entropy. Different combinations of these features are utilized as fitness functions of the proposed GA based framework. The algorithm is able to find out that subset of features which are uncorrelated and mutually exclusive to each other. Finally, the optimal feature subset obtained is utilized for developing classification systems using some popular machine learning models like decision trees, support vector machines, k-nearest neighbor classifier on the KDD-Cup 99 dataset. The experimental results show that decision tree produces better results than other classifiers. The result confirms 99.62% accuracy, 98.78% detection rate and 0.25% false alarm rate. The most attractive feature of the proposed scheme is that it does not require any labeled information during the feature selection process.

In recent years, social networking sites such as Twitter have become the primary sources for real-time information of ongoing events such as political rallies, natural disasters, and so on. At the time of occurrence of natural disasters, it has been seen that relevant information collected from tweets can help in different ways. Therefore, there is a need to develop an automated microblog/tweet summarization system to automatically select relevant tweets. In this article, we employ the concept of multiobjective optimization in microblog summarization to produce good quality summaries. Different statistical quality measures namely, length, tf-idf score of the tweets, antiredundancy, measuring different aspects of summary, are optimized simultaneously using the search capability of a multiobjective differential evolution technique. Different types of genetic operators including recently developed self-organizing map (a type of neural network) based operator, are explored in the proposed framework. To measure the similarity between tweets, word mover distance is utilized which is capable of capturing the semantic similarity between tweets. For evaluation, four benchmark data sets related to disaster events are used, and the results obtained are compared with various state-of-the-art techniques using ROUGE measures. It has been found that our algorithm improves by 62.37% and 5.65% in terms of ROUGE-2 and ROUGE-L scores, respectively, over the state-of-the-art techniques. Results are also validated using statistical significance t-test. At the end of the article, extension of the proposed approach to solve the multidocument summarization task is also illustrated.

Classification of samples of gene expression profile plays a significant role in prediction and diagnosis of diseases. In the task of sample classification, a robust feature selection algorithm is very much essential to identify the important genes from the high dimensional gene expression data. This paper explores the information of protein-protein interaction with a graph mining technique for finding a proper subset of features (genes), which further takes part in sample classification. Here, our contribution for feature selection is three-fold: first, all the genes are grouped into different clusters based on the integrated information of the gene expression values and their protein interactions using a multi-objective optimization based clustering approach. Second, the confidence scores of the protein interactions are incorporated in a popular graph mining algorithm namely Goldberg algorithm to find out the relevant features. These features are the topologically and functionally significant genes, named as hub genes. Finally, these hub genes are identified varying the degrees of the nodes, and those are utilized for the sample classification task. Different machine learning classifiers are exploited for this purpose, and the classification performance is measured with respect to various performance metrics namely accuracy, sensitivity, specificity, precision, F-measure, and Mathews coefficient correlation. Comparative analysis with respect to two baselines and several existing approaches proves the efficiency of the proposed approach. Furthermore, the robustness of the identified hub-gene modules is endorsed using some strong biological significance analysis.

The current paper aims to improve an efficient algorithm for solving the problem of non-dominated sorting which is one of the dominant steps of any Pareto based multi/many objective optimization algorithms. Recent years witnessed a large number of attempts in developing some solution frameworks for the problem of non-dominated sorting. One such recent approach is `Best Order Sort' which is efficient with respect to the number of dominance comparisons. However, this approach does not perform well in the presence of duplicate solutions. In this paper attempts have been made to modify the `Best Order Sort' and we call this modified version as `Modified Best Order Sort' to remove the above mentioned limitation. The modified best order sort algorithm has been thoroughly analyzed in different scenarios. Current work shows that `Best Order Sort' can be generalized without affecting its best and the worst case time complexities.

Cuckoo Search is a nature inspired evolutionary algorithm to solve real-world optimization problems. It is inspired from the brood parasitism of cuckoos. It is highly competitive and has been used to solve number of problems in the field of science and engineering. A number of modifications have been proposed to enhance its performance in the past. This paper presents an improved version of CS namely CVnew in which three modifications are proposed. The first modification is the introduction of two new search equations to improve the global search while the second one deals with the incorporation of four search equations to improve the local search. As a third modification, a balance between global and local search has been increased by exponentially decreasing the switch probability. The proposed algorithm has been applied to solve single objective real-parameter problems of CEC 2017. The numerical results prove the better performance of CVnew in comparison with SaDE, JADE, SHADE and MVMO.

Current paper explores the use of multi-view learning for search result clustering. A web-snippet can be represented using multiple views. Apart from textual view cued by both the semantic and syntactic information, a complimentary view extracted from images contained in the web-snippets is also utilized in the current framework. A single consensus partitioning is finally obtained after consulting these two individual views by the deployment of a multiobjective based clustering technique. Several objective functions including the values of a cluster quality measure measuring the goodness of partitionings obtained using different views and an agreement-disagreement index, quantifying the amount of oneness among multiple views in generating partitionings are optimized simultaneously using AMOSA. In order to detect the number of clusters automatically, concepts of variable length solutions and a vast range of permutation operators are introduced in the clustering process. Finally, a set of alternative partitioning are obtained on the final Pareto front by the proposed multi-view based multiobjective technique. Experimental results by the proposed approach on several benchmark test datasets of SRC with respect to different performance metrics evidently establish the power of visual and text-based views in achieving better search result clustering.

The self-organizing map (SOM) helps in exploratory phase of data mining by projecting the input data into a lower dimensional map. In recent years SOM has also been applied for classification of data points. The prominent utility of SOM based classification is evident from the use of no labeled data during training. In a multi-class classification problem where classes have high degree of overlap, it would be difficult to design a single-level SOM based classification system which can perform well for all the classes. In order to deal with the multi-class classification efficiently, the current paper proposes to develop a Cascaded SOM based architecture where classes are handled in a hierarchical way. Also, it can be applied for solving any multi-class classification problem where labeled data is limited. As a case study, in the first part of the paper results are shown for single label version of complex email classification problem where classes are highly overlapping to each other and in the second part, results are shown for some multi-labeled data sets. Different representation schemas for emails and a large set of features are also adopted for the purpose of experiment. Proposed Cascaded SOM based classification model performs well in email-classification compared to standard classification approaches and classical SOM based model.

Cluster validity indices are proposed in the literature to measure the goodness of a clustering result. The validity measure provides a value which shows how good or bad the obtained clustering result is, as compared to the actual clustering result. However, the validity measures are not arbitrarily generated. A validity measure should satisfy some of the important properties. However, there are cases when in-spite of satisfying these properties, a validity measure is not able to differentiate the two clustering results correctly. In this regard, sensitivity as a property of validity measure is introduced to capture the differences between the two clustering results. However, sensitivity computation is a computationally expensive task as it requires to explore all the possible combinations of clustering results which are very large in number and these are growing exponentially. So, it is required to compute the sensitivity efficiently. As the possible combinations of clustering results grow exponentially, so it is required to first obtain an upper bound on this possible number of combinations which will be sufficient to compute the value of the sensitivity. In this paper, we obtain an upper bound on the number of possible combinations of clustering results. For this purpose, a generic approach which is suitable for various validity measures and a specific approach which is applicable for two validity measures are proposed. It is also shown that this upper bound is sufficient to compute the sensitivity of various validity measures. This upper bound is very less as compared to the total number of possible combinations of clustering results.

MicroRNAs (miRNAs) are a type of RNAs, which are responsible for monitoring the gene expression values. Recent research asserts that miRNAs form some clustering on chromosomes. The miRNAs belonging to a particular cluster are highly similar in terms of their activity and they are termed as "coregulated" miRNAs. The current paper presents an approach that simultaneously performs two tasks: i) clustering of miRNAs into different categories based on some similarity measures ii) identification of proper weight values for different time points with respect to which expression values are available. In general, a large number of expression values are available for a given miRNA data set. All these values may not be suitable to be used equally to measure the similarity between two miRNAs. In the current study, the problem of proper selection of weight values for different time points and then determining the proper partitioning from the given miRNA data set utilizing the similarity computed using the new set of weight values is formulated as an optimization problem where several cluster validity indices are optimized as the goodness measures. To that end, a multiobjective differential evolution based optimization technique is utilized. The supremacy of the proposed technique is tested on three miRNA data sets in comparison to some recent approaches in terms of some popular performance measures like Silhouette index and DB-index. The observations are further supported by statistical and biological significance tests. Supplementary information is available at https://www.iitp.ac.in/~sriparna/journals.html.

We present a new multiview clustering approach based on multiobjective optimization. In contrast to existing clustering algorithms based on multiobjective optimization, it is generally applicable to data represented by two or more views and does not require specifying the number of clusters a priori . The approach builds upon the search capability of a multiobjective simulated annealing based technique, AMOSA, as the underlying optimization technique. In the first version of the proposed approach, an internal cluster validity index is used to assess the quality of different partitionings obtained using different views. A new way of checking the compatibility of these different partitionings is also proposed and this is used as another objective function. A new encoding strategy and some new mutation operators are introduced. Finally, a new way of computing a consensus partitioning from multiple individual partitions obtained on multiple views is proposed. As a baseline and for comparison, two multiobjective based ensemble clustering techniques are proposed to combine the outputs of different simple clustering approaches. The efficacy of the proposed clustering methods is shown for partitioning several real-world datasets having multiple views. To show the practical usefulness of the method, we present results on web-search result clustering, where the task is to find a suitable partitioning of web snippets.

Shweta Yadav, Asif Ekbal, Sriparna Saha, Pushpak Bhattacharyya. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. 2017.

BACKGROUND: Classification of biological samples of gene expression data is a basic building block in solving several problems in the field of bioinformatics like cancer and other disease diagnosis and making a proper treatment plan. One big challenge in sample classification is handling large dimensional and redundant gene expression data. To reduce the complexity of handling this high dimensional data, gene/feature selection plays a major role. RESULTS: The current paper explores the use of biological knowledge acquired from Gene Ontology database in selecting the proper subset of genes which can further participate in clustering of samples. The proposed feature selection technique is unsupervised in nature as it does not utilize any class label information in the process of gene selection. At the end, a multi-objective clustering approach is deployed to cluster the available set of samples in the reduced gene space. CONCLUSIONS: Reported results show that consideration of biological knowledge in gene selection technique not only reduces the feature space dimensionality in great extent but also improves the accuracy of sample classification. The obtained reduced gene space is validated using strong biological significance tests. In order to prove the supremacy of our proposed gene selection based sample clustering technique, a thorough comparative analysis has also been performed with state-of-the-art techniques.

Microarray technology facilitates to monitor the expression levels of multiple genes simultaneously over a number of time points. For analyzing such microarray gene expression data, fuzzy clustering plays an important role. However, this fact has inspired us to propose a new automatic cluster detection method using modified differential evolution based fuzzy symmetric clustering technique. Here, allocation of gene data points to different clusters has been made based on a newly developed symmetry based cluster validity index. In order to get a best fuzzy partition of a gene dataset, FSym-index needs to be maximized. The search capability of the modified differential evolution based optimization technique in conjunction with the fuzzy symmetry based cluster validity index is applied to ascertain the good fuzzy partition as well as to obtain actual number of cluster present in a data set. The superiority of the proposed fuzzy-VMODEPS is compared with those obtained by some recent automatic clustering techniques, VGAPS, GCUK and HNGA over publicly available three real life gene expression data-sets.

Entity matching is to map the records to the corresponding entity. It is a well known problem studied by many researchers over the last few years. In bibliographic database, the data evolve over time. For example, the email id of an author in DBLP and ArnetMiner which are two bibliographic databases changes with time. Authors also keep on changing their affiliations. The set of authors with whom they work also changes with time. These types of variations make the entity matching task more difficult. In this paper, we have addressed this problem and proposed the nondominated sorting genetic algorithm-II (NSGA-II) based solution framework. The dissimilarities between different records can be measured using various distance measures. One distance measure can be suitable for one data set while some other distance measure can be suitable for some other data sets. So selecting the appropriate distance measure is also difficult. To address this issue, this paper presents an automatic framework which selects the suitable distance measure along with the appropriate partitioning. To encode the partitions, medoid based encoding is used. Several new mutation operations are used to explore the search space efficiently. Silhouette Index along with Xie-Beni Index are optimized simultaneously during the experiments for three bibliographic data sets. The results of our approach are compared with two existing well known techniques, DBLP and ArnetMiner. From the results it is clear that our proposed approach performs well.

Many of the real-life problems involve simultaneous optimization of multiple objectives. In recent years there is an enormous increase in the number of multi-objective optimization problems related to different real-life domains. Evolutionary algorithms are the most popular in solving these types of problems. The non-dominating sorting is one of the steps of any multiobjective evolutionary algorithms. This is used mostly to select the non-dominated set of solutions from a given population. In the past various efficient approaches are proposed in the literature to reduce the complexity of this step. As the evolutionary algorithms inhibit parallelism in it. But not all the existing non-dominating sorting approaches have the parallelism property. So in this paper, we have proposed a new approach named as DCNS (Divide and Conquer based Non-dominating Sorting) which inhibits parallelism in it. It has been shown theoretically and empirically that the proposed approach is computationally efficient than existing state-of-the-art methods.

In steady-state evolutionary algorithms, the parent population is updated each time once a new offspring solution is generated. Due to the updation of the parent population, the non-dominated sorting needs to be applied again and again. The repetition of non-dominated sorting makes steady-state algorithms computationally expensive. But the recent study has identified that the insertion of an offspring solution in the known non-domination level structure of solutions does not change the entire structure. So there is no need to apply the complete non-dominated sorting algorithm again and again. In this regard an efficient non-domination level update approach known as ENLU approach was proposed. In steady-state evolutionary algorithm, the same pair of solutions can be compared multiple times in different generations of the algorithm. In this paper, we have performed the same ENLU approach in a different way so that the same pair of solutions is compared only once if they are in the current population. The worst case time complexity of ENLU approach is O(MN2). So for G generations of the steady-state evolutionary algorithm, the worst case time complexity is GO(MN2). In this paper, we have utilized the same ENLU approach which performs the same number of comparisons all the times. The worst case time complexity of our approach is G(O(MN log N) + O(N2)). However, in terms of space complexity the proposed approach requires O(N2) space as compared to O(N) of the ENLU approach. So we have achieved the speedup at the cost of extra space. At the end, we have explored the possibility of parallelism to make the ENLU approach faster.

Starting from scalability to visualization, several challenges come into play when multi-objective optimization algorithms are applied to many-objective optimization problems. These challenges can be tackled using objective reduction algorithms. This work proposes a correlation based objective reduction algorithm. The set of points representing the Pareto-Front are clustered based on correlation distance. The constituent objectives belonging to the most compact cluster are pruned except the centroid. The reduction and search proceed while alternating between full set and reduced set. Number of clusters is gradually increased in order to zoom in the correlation structure of the objectives and the algorithm stops when the number of clusters equals to the number of objectives. This objective reduction leads to automatic detection of number of objectives, faster convergence as one or more number of objectives can be eliminated in reduction stages and improved performance as both global exploration and local exploitation of the search space are considered. The proposed approach of objective reduction is coupled with a differential evolution based multi-objective optimization technique to develop a solution framework for many-objective optimization problems. The efficacy of the proposed approach is tested on several benchmark functions viz. DTLZ1, DTLZ2, DTLZ3 and DTLZ4 for 10 and 20 objectives. Convergence metric and hypervolume indicator are used for comparing the performance of the proposed solution framework with other popular multi-objective optimization algorithms.

The invention of microarray technology aids in the successful monitoring of the gene expression patterns. Biclustering is a method in which a number of co-regulated genes are identified over subset of conditions. Our aim is to detect all the non trivial biclusters having low mean squared residue(MSR) and high row variance. In this paper, we have proposed a multi-objective simulated annealing based solution framework to solve the biclustering problem from gene expression data sets. Two objective functions MSR and row-variance capturing two important properties of biclusters are optimized in parallel using the search capability of multi-objective simulated annealing based optimization technique, AMOSA. A new encoding strategy and several different search operators are defined for fast convergence of the algorithm. We have done experiment on two real-life data sets and obtained results are quantified by using several cluster validity indices. We have compared our obtained results with some state-of-the-art biclustering techniques.

In recent past there has been phenomenal growth in biomedical literature and health care records. Robust text mining techniques are essential in order to properly organize the documents as well as to extract relevant information. Traditional techniques for document classification focus on machine learning algorithms where learning of classifier is decided on the basis of labeled data and the features that are prominent. In this paper we focus on developing an automated technique for classifying biomedical articles containing protein-protein interaction related information against the others. Our proposed approach is based on deep neural network framework. We investigate the role of convolution neural network (CNN) and propose two model variants. We evaluate the proposed approach on the benchmark datasets of BioCreative-II Interaction Article Subtask (IAS) data sets. Effectiveness of our proposed model is evident with the significant performance gains, 2.8 % in terms of F-measure and 5 % in terms of accuracy over the traditional models.

In the field of pattern recognition, the study of the gene expression profiles of different tissue samples over different experimental conditions has become feasible with the arrival of microarray-based technology. In cancer research, classification of tissue samples is necessary for cancer diagnosis, which can be done with the help of microarray technology. In this paper, we have presented a multiobjective optimization (MOO)-based clustering technique utilizing archived multiobjective simulated annealing(AMOSA) as the underlying optimization strategy for classification of tissue samples from cancer datasets. The presented clustering technique is evaluated for three open source benchmark cancer datasets [Brain tumor dataset, Adult Malignancy, and Small Round Blood Cell Tumors (SRBCT)]. In order to evaluate the quality or goodness of produced clusters, two cluster quality measures viz, adjusted rand index and classification accuracy ( % CoA) are calculated. Comparative results of the presented clustering algorithm with ten state-of-the-art existing clustering techniques are shown for three benchmark datasets. Also, we have conducted a statistical significance test called t-test to prove the superiority of our presented MOO-based clustering technique over other clustering techniques. Moreover, significant gene markers have been identified and demonstrated visually from the clustering solutions obtained. In the field of cancer subtype prediction, this study can have important impact.

Studying the patterns hidden in gene-expression data helps to understand the functionality of genes. In general, clustering techniques are widely used for the identification of natural partitionings from the gene expression data. In order to put constraints on dimensionality, feature selection is the key issue because not all features are important from clustering point of view. Moreover some limited amount of supervised information can help to fine tune the obtained clustering solution. In this paper, the problem of simultaneous feature selection and semisupervised clustering is formulated as a multiobjective optimization (MOO) task. A modern simulated annealing-based MOO technique namely AMOSA is utilized as the background optimization methodology. Here, features and cluster centers are represented in the form of a string and the assignment of genes to different clusters is done using a point symmetry-based distance. Six optimization criteria based on several internal and external cluster validity indices are utilized. In order to generate the supervised information, a popular clustering technique, Fuzzy C-mean, is utilized. Appropriate subset of features, proper number of clusters and the proper partitioning are determined using the search capability of AMOSA. The effectiveness of this proposed semisupervised clustering technique, Semi-FeaClustMOO, is demonstrated on five publicly available benchmark gene-expression datasets. Comparison results with the existing techniques for gene-expression data clustering again reveal the superiority of the proposed technique. Statistical and biological significance tests have also been carried out.

Understanding the temporal orientation of web search queries is an important issue for the success of information access systems. In this paper, we propose a multi-objective ensemble learning solution that (1) allows to accurately classify queries along their temporal intent and (2) identifies a set of performing solutions thus offering a wide range of possible applications. Experiments show that correct representation of the problem can lead to great classification improvements when compared to recent state-of-the-art solutions and baseline ensemble techniques.

In entity name disambiguation, performance evaluation of any approach is difficult. This is due to the fact that correct or actual results are often not known. Generally for evaluation purpose, three measures namely precision, recall and f-measure are used. They all are external validity indices because they need golden standard data. But in Bibliographic databases like DBLP, Arnetminer, Scopus, Web of Science, Google Scholar, etc., gold standard data is not easily available and it is very difficult to obtain this due to the overlapping nature of data. So, there is a need to use some other matrices for evaluation purpose. In this paper, some internal cluster validity index based schemes are proposed for evaluating entity name disambiguation algorithms when applied on bibliographic data without using any gold standard datasets. Two new internal validity indices are also proposed in the current paper for this purpose. Experimental results shown on seven bibliographic datasets reveal that proposed internal cluster validity indices are able to compare the results obtained by different methods without prior/gold standard. Thus the present paper demonstrates a novel way of evaluating any entity matching algorithm for bibliographic datasets without using any prior/gold standard information.

Evaluating a given clustering result is a very difficult problem in real world. Cluster validity indices are developed for this purpose. There are two different types of cluster validity indices available : External and Internal. External cluster validity indices utilize some supervised information and internal cluster validity indices utilize the intrinsic structure of the data. In this paper a new external cluster validity index, MMI has been implemented based on Max-Min distance among data points and prior information based on structure of the data. A new probabilistic approach has been implemented to find the correct correspondence between the true and obtained clustering. Genetic K-means algorithm (GAK-means) and single linkage have been used as the underlying clustering techniques. Results of the proposed index for identifying the appropriate number of clusters is shown for five artificial and two real-life data sets. GAK-means and single linkage clustering techniques are used as the underlying partitioning techniques with the number of clusters varied over a range. The MMI index is then used to determine the appropriate number of clusters. The performance of MMI is compared with existing external cluster validity indices, adjusted rand index (ARI) and rand index (RI). It works well for two class and multi class data sets.

Plagiarism denotes the act of copying someone else's idea (or, works) and claiming it as his/her own. Plagiarism detection is the procedure to detect the texts of a given document which are plagiarized, i.e. copied from from some other documents. Potential challenges are due to the facts that plagiarists often obfuscate the copied texts; might shuffle, remove, insert, or replace words or short phrases; might also restructure the sentences replacing words with synonyms; and changing the order of appearances of words in a sentence. In this paper we propose a technique based on textual similarity for external plagiarism detection. For a given suspicious document we have to identify the set of source documents from which the suspicious document is copied. The method we propose comprises of four phases. In the first phase, we process all the documents to generate tokens, lemmas, finding Part-of-Speech (PoS) classes, character-offsets, sentence numbers and named-entity (NE) classes. In the second phase we select a subset of documents that may possibly be the sources of plagiarism. We use an approach based on the traditional Vector Space Model (VSM) for this candidate selection. In the third phase we use a graph-based approach to find out the similar passages in suspicious document and selected source documents. Finally we filter out the false detections1.

Semi-supervised clustering uses the information of unsupervised and supervised learning to overcome the problems associated with them. Extracted information are given in the form of class labels and data distribution during clustering process. In this paper the problem of semi-supervised clustering is formulated under the framework of multiobjective optimization (MOO). Thereafter, a multiobjective based clustering technique is extended to solve the semi-supervised clustering problem. The newly developed semi-supervised multiobjective clustering algorithm (Semi-GenClustMOO), is used for appropriate partitioning of data into appropriate number of clusters. Four objective functions are optimized, out of which first three use some unsupervised information and the last one uses supervised information. These four objective functions represent, respectively, the, total compactness of the partitioning, total symmetry present in the clusters, cluster connectedness and Adjust Rand Index. These four objective functions are optimized simultaneously using AMOSA, a newly developed simulated annealing based multiobjective optimization method. Results show that it can easily detect the appropriate number of clusters as well as the appropriate partitioning from data sets having either well-separated clusters of any shape or symmetrical clusters with or without overlaps. Seven artificial and four real-life data sets have been used for evaluation to show the effectiveness of the Semi-GenClustMOO technique. In each case class information of 10% randomly chosen data point is known to us1.

In this article, we report the search capability of Genetic Algorithm (GA) to construct a weighted vote-based classifier ensemble for Named Entity Recognition (NER). Our underlying assumption is that the reliability of predictions of each classifier differs among the various named entity (NE) classes. Thus, it is necessary to quantify the amount of voting of a particular classifier for a particular output class. Here, an attempt is made to determine the appropriate weights of voting for each class in each classifier using GA. The proposed technique is evaluated for four leading Indian languages, namely Bengali, Hindi, Telugu, and Oriya, which are all resource-poor in nature. Evaluation results yield the recall, precision and F-measure values of 92.08%, 92.22%, and 92.15%, respectively for Bengali; 96.07%, 88.63%, and 92.20%, respectively for Hindi; 78.82%, 91.26%, and 84.59%, respectively for Telugu; and 88.56%, 89.98%, and 89.26%, respectively for Oriya. Finally, we evaluate our proposed approach with the benchmark dataset of CoNLL-2003 shared task that yields the overall recall, precision, and F -measure values of 88.72%, 88.64%, and 88.68%, respectively. Results also show that the vote based classifier ensemble identified by the GA-based approach outperforms all the individual classifiers, three conventional baseline ensembles, and some other existing ensemble techniques. In a part of the article, we formulate the problem of feature selection in any classifier under the single objective optimization framework and show that our proposed classifier ensemble attains superior performance to it.

In this paper, we report a multiobjective optimization (MOO) based classifier ensemble technique to solve the problem of Named Entity Recognition (NER). Our underlying assumption is that rather than searching for the best feature set for a particular classifier, ensembling of several classifiers which are trained using different feature representations could be a more fruitful approach. But, it is very crucial to select the appropriate classifiers that can participate in final ensembling. Here, we propose a new technique for classifier ensembling based on MOO that can simultaneously optimize several different classification measures. Maximum Entropy (ME) framework is used to generate a number of classifiers by considering the various combinations of the available features. The proposed technique is evaluated for two resource constrained languages, namely Bengali and Hindi. Evaluation results yield the recall, precision and F-measure values of 72.34%, 84.94% and 78.13%, respectively for Bengali, and 64.93%, 83.29% and 72.97%, respectively for Hindi. Experiments also show that the classifier ensemble identified by the proposed multiobjective based approach outperforms all the individual classifiers, two different baseline ensembles and a classifier ensemble identified by the single objective genetic algorithm (GA) based approach.

Appropriate feature selection is a very crucial issue in any machine learning framework, specially in Maximum Entropy (ME). In this paper, the selection of appropriate features for constructing a ME based Named Entity Recognition (NER) system is posed as a multiobjective optimization (MOO) problem. Two classification quality measures, namely recall and precision are simultaneously optimized using the search capability of a popular evolutionary MOO technique, NSGA-II. The proposed technique is evaluated to determine suitable feature combinations for NER in two languages, namely Bengali and English that have significantly different characteristics. Evaluation results yield the recall, precision and F-measure values of 70.76%, 81.88% and 75.91%, respectively for Bengali, and 78.38%, 81.27% and 79.80%, respectively for English. Comparison with an existing ME based NER system shows that our proposed feature selection technique is more efficient than the heuristic based feature selection.

In this paper, we present the problem of appropriate feature selection for constructing a Maximum Entropy (ME) based Named Entity Recognition (NER) system under the multiobjective optimization (MOO) framework. Two conflicting objective functions are simultaneously optimized using the search capability of MOO. These objectives are (i). the dimensionality of features, which is tried to be minimized, and (ii). the corresponding F-measure value of the classifier, trained using the features present, is maximized. The features are encoded in the chromosomes. Thereafter, a multiobjective evolutionary algorithm in the steps of a popular MOO technique, NSGA-II, is developed to determine the appropriate feature subset. The proposed technique is evaluated to determine the suitable feature combinations for NER in a resource-constrained language, namely Bengali. Evaluation results yield the recall, precision and F-measure values of 72.45%, 82.39% and 77.11%, respectively.

In this paper, we report a classifier ensemble technique using the search capability of genetic algorithm (GA) for Named Entity Recognition (NER) in biomedical domain. We use Maximum Entropy (ME) framework to build a number of classifiers depending upon the various representations of a set of features. The proposed technique is evaluated with the JNLPBA 2004 data sets that yield the overall recall, precision and F-measure values of 67.98%, 71.68% and 69.78%, respectively.

The problem of classifying an image into different homogeneous regions is viewed as a task of clustering the pixels in the intensity space. In this letter, a newly developed genetic clustering technique is used for automatically segmenting remote sensing satellite images. Each cluster is divided into several small hyperspherical subclusters, and the centers of all these small subclusters are encoded in a chromosome to represent the whole clustering. For assigning points to different clusters, these local subclusters are considered individually. For the purpose of objective function evaluation, these subclusters are merged appropriately to form a variable number of global clusters. A newly proposed point-symmetry-distance-based cluster validity index,Symindex, is used as a measure of the validity of the corresponding segment. The effectiveness of the proposed technique compared to a fuzzy C-means clustering technique, a recently proposed GAPS clustering withSym-index-based method, and a subtractive clustering technique is demonstrated in identifying different land cover regions from two numeric image data sets and a remote sensing image of a part of the city of Kolkata.

In this paper, a novel point symmetry based pattern classifier (PSC) is proposed. A recently developed point symmetry based distance is utilized to determine the amount of point symmetry of a particular test pattern with respect to a class prototype. Kd-tree based nearest neighbor search is used for reducing the complexity of point symmetry distance computation. The proposed point symmetry based classifier is well-suited for classifying data sets having point symmetric classes, irrespective of any convexity, overlap or size. In order to classify data sets having line symmetry property, a line symmetry based classifier (LSC) along the lines of PSC is thereafter proposed in this paper. To measure the total amount of line symmetry of a particular point in a class, a new definition of line symmetry based distance is also provided. Proposed LSC preserves the advantages of PSC. The performance of PSC and LSC are demonstrated in classifying fourteen artificial and real-life data sets of varying complexities. For the purpose of comparison, k-NN classifier and the well-known support vector machine (SVM) based classifiers are executed on the data sets used here for the experiments. Statistical analysis, ANOVA, is also performed to compare the performance of these classification techniques.

In this paper, an evolutionary technique for the semi-supervised clustering is proposed. The proposed technique uses a point symmetry based distance measure. Semi-supervised classification uses aspects of both unsupervised and supervised learning to improve upon the performance of traditional classification methods. In this paper the existing point symmetry based genetic clustering technique, GAPS-clustering, is extended in two different ways to handle the semi-supervised classi- fication problem. The proposed semi-GAPS clustering algorithmis able to detect any type of clusters irrespective of shape, size and convexity as long as they possess the point symmetry property. Kdtree based nearest neighbor search is used to reduce the complexity of finding the closest symmetric point. Adaptive mutation and crossover probabilities are used. Experimental results demonstrate practical performance benefits of the methodology in detecting classes having symmetrical shapes in case of semi-supervised clustering.

In this paper, the automatic segmentation of multispectral magnetic resonance image of the brain is posed as a clustering problem in the intensity space. Thereafter an automatic clustering technique is proposed to solve this problem. The proposed real-coded variable string length genetic clustering technique (MCVGAPS clustering) is able to evolve the number of clusters present in the data set automatically. Each cluster is divided into several small hyperspherical subclusters and the centers of all these small sub-clusters are encoded in a string to represent the whole clustering. For assigning points to different clusters, these local sub-clusters are considered individually. For the purpose of objective function evaluation, these sub-clusters are merged appropriately to form a variable number of global clusters. A recently developed point symmetry distance based cluster validity index, Sym-index, is optimized to automatically evolve the appropriate number of clusters present in an MR brain image. The proposed method is applied on several simulated T1-weighted, T2- weighted and proton density normal and MS lesion magnetic resonance brain images. Superiority of the proposed method over Fuzzy C-means, Expectation Maximization clustering algorithms are demonstrated quantitatively. The automatic segmentation obtained by multiseed based multiobjective clustering technique (MCVGAPS) is also compared with the available ground truth information.

Identification of the correct number of clusters is an important consideration in clustering where several cluster validity indexes, primarily utilizing the Euclidean distance, have been used in the literature. The property of symmetry is observed in most clustering solutions. In this paper, the symmetry versions of nine cluster validity indexes, namely, Davies-Bouldin index, Dunn index, generalized Dunn index, point symmetry (PS) index,Iindex, Xie-Beni index, FS index,Kindex, and SV index, are proposed. It is empirically established that incorporation of the property of symmetry significantly improves the capabilities of these indexes in identifying the appropriate number of clusters. A recently developed PS-based genetic clustering technique, GAPS clustering, is used as the underlying partitioning algorithm. Results on six artificially generated and five real-life datasets show that symmetry-distance-basedIindex performs the best as compared to all the other eight indexes.

Most clustering algorithms operate by optimizing (either implicitly or explicitly) a single measure of cluster solution quality. Such methods may perform well on some data sets but lack robustness with respect to variations in cluster shape, proximity, evenness and so forth. In this paper, we have proposed a multiobjective clustering technique which optimizes simultaneously two objectives, one reflecting the total symmetry present in the data set and the other reflecting the stability of the obtained partitions over different bootstrap samples of the data set. The proposed algorithm utilizes a recently developed simulated annealing based multiobjective optimization technique, AMOSA, as the underlying optimization method. Here assignment of points to different clusters are done based on the point symmetry based distance rather than the Euclidean distance. Results on several artificial and real-life data sets show that the proposed technique is well-suited to detect the number of clusters from data sets having point symmetric clusters.

An important approach for landcover classification in remote sensing images is by clustering the pixels in the spectral domain into several fuzzy partitions. In this article the problem of fuzzy clustering is posed as one of searching for some suitable number of cluster centers so that some measures of validity of the obtained partitions should be optimized. In this paper a recently developed multiobjective simulated annealing based technique, AMOSA (archived multiobjective simulated annealing technique), is used to perform clustering, taking two validity measures as two objective functions. Here two fuzzy cluster validity functions namely, well-known XB-index and newly developed FSym-index are optimized simultaneously to automatically evolve the appropriate number of clusters present in an image. Thus the proposed algorithm provides a set of final non-dominated solutions, which the user can judge relatively and pick up the most promising one according to the domain requirement. The effectiveness of this proposed clustering technique in comparison with the existing fuzzy C-means clustering is shown for automatically classifying two remote sensing satellite images of the parts of the cities of Kolkata and Mumbai.

In this paper, a new line symmetry based classifier (LSC) is proposed to deal with pattern classification problems. In order to measure total amount of line symmetry of a particular point in a class, a new definition of line symmetry based distance is also proposed in this paper. The proposed line symmetry based classifier (LSC) utilizes this new definition of line symmetry distance for classifying an unknown test sample. LSC assigns an unknown test sample pattern to that class with respect to whose major axis it is most symmetric. The mean of all the training patterns belonging to that particular class is taken as the prototype of that class. Thus training constitutes of computing only the class prototypes and the major axes of those classes. Kd-tree based nearest neighbor search is used for reducing complexity of line symmetry distance computation. The performance of LSC is demonstrated in classifying twelve artificial and real-life data sets of varying complexities. Experimental results show that LSC achieves, in general, higher classification accuracy compared to k-NN classifier. Results indicate that the proposed novel line symmetry based classifier is well-suited for classifying data sets having symmetrical classes, irrespective of any convexity, overlap and size. Statistical analysis, ANOVA is also performed to compare the performance of these classifications techniques.

An important approach for image segmentation is clustering pixels based on their spectral properties. In particular, satellite images contain land cover types, some of which cover significantly large areas, while some (e.g., bridges and roads) occupy relatively much smaller regions. Automatically detecting regions or clusters of such widely varying sizes presents a challenging task. In this letter, a symmetry-based cluster validity index, named Sym-index (Symmetry distance-based index), is proposed. It is able to correctly indicate the presence of clusters of different sizes as long as they are internally symmetrical. A genetic-algorithm-based clustering technique that optimizes the Sym-index is used for image segmentation where the number of clusters is determined automatically. The superiority of the proposed index, as compared to other indices, is established for automatically segmenting the land cover types from SPOT and Indian Remote Sensing satellite images of two different cities in India.

This paper describes a simulated annealing based multiobjective optimization algorithm that incorporates the concept of archive in order to provide a set of tradeoff solutions for the problem under consideration. To determine the acceptance probability of a new solution vis-a-vis the current solution, an elaborate procedure is followed that takes into account the domination status of the new solution with the current solution, as well as those in the archive. A measure of the amount of domination between two solutions is also used for this purpose. A complexity analysis of the proposed algorithm is provided. An extensive comparative study of the proposed algorithm with two other existing and well-known multiobjective evolutionary algorithms (MOEAs) demonstrate the effectiveness of the former with respect to five existing performance measures, and several test problems of varying degrees of difficulty. In particular, the proposed algorithm is found to be significantly superior for many objective test problems (e.g., 4, 5, 10, and 15 objective problems), while recent studies have indicated that the Pareto ranking-based MOEAs perform poorly for such problems. In a part of the investigation, comparison of the real-coded version of the proposed algorithm is conducted with a very recent multiobjective simulated annealing algorithm, where the performance of the former is found to be generally superior to that of the latter.

In this article, a new symmetry based genetic clustering algorithm is proposed which automatically evolves the number of clusters as well as the proper partitioning from a data set. Strings comprise both real numbers and the don't care symbol in order to encode a variable number of clusters. Here, assignment of points to different clusters are done based on a point symmetry based distance rather than the Euclidean distance. A newly proposed point symmetry based cluster validity index, {\em Sym}-index, is used as a measure of the validity of the corresponding partitioning. The algorithm is therefore able to detect both convex and non-convex clusters irrespective of their sizes and shapes as long as they possess the point symmetry property. Kd-tree based nearest neighbor search is used to reduce the complexity of computing point symmetry based distance. A proof on the convergence property of variable string length GA with point symmetry based distance clustering (VGAPS-clustering) technique is also provided. The effectiveness of VGAPS-clustering compared to variable string length Genetic K-means algorithm (GCUK-clustering) and one recently developed weighted sum validity function based hybrid niching genetic algorithm (HNGA-clustering) is demonstrated for nine artificial and five real-life data sets.

Identification of the correct number of clusters and the corresponding partitioning are two important considerations in clustering. In this paper, a newly developed point symmetry based distance is used to propose symmetry based versions of six cluster validity indices namely, DB-index, Dunn-index, generalized Dunn-index, PS-index, I-index and XB-index. These indices provide measures of "symmetricity" of the different partitionings of a data set. A Kd-tree-based data structure is used to reduce the complexity of computing the symmetry distance. A newly developed genetic point symmetry based clustering technique, GAPS-clustering is used as the underlying partitioning algorithm. The number of clusters are varied from 2 to radicn where n is the total number of data points present in the data set and the values of all the validity indices are noted down. The optimum value of a validity index over these radicn-1 partitions corresponds to the appropriate partitioning and the number of partitions as indicated by the validity index. Results on five artificially generated and four real-life data sets show that symmetry distance based I-index performs the best compared to all the other five indices.

In this paper, an automatic segmentation technique of multispectral magnetic resonance image of the brain using a new fuzzy point symmetry based genetic clustering technique is proposed. The proposed real-coded variable string length genetic fuzzy clustering technique (fuzzy-VGAPS) is able to evolve the number of clusters present in the data set automatically. Here, assignment of points to different clusters are made based on the point symmetry based distance rather than the Euclidean distance. The cluster centers are encoded in the chromosomes, whose value may vary. A newly developed fuzzy point symmetry based cluster validity index, FSym-index, is used as a measure of 'goodness' of the corresponding partition. This validity index is able to correctly indicate presence of clusters of different sizes as long as they are internally symmetrical. A Kd-tree based data structure is used to reduce the complexity of computing the symmetry distance. The proposed method is applied on several simulated T1-weighted, T2-weighted and proton density normal and MS lesion magnetic resonance brain images. Superiority of the proposed method over fuzzy C-means, expectation maximization, fuzzy variable string length genetic algorithm (fuzzy-VGA) clustering algorithms are demonstrated quantitatively. The automatic segmentation obtained by fuzzy-VGAPS clustering technique is also compared with the available ground truth information.

An important consideration in clustering is the determination of the correct number of clusters and the appropriate partitioning of a given data set. In this paper, a newly developed point symmetry distance is used to propose a new cluster validity index named Sym -index which provides a measure of "symmetricity" of the different partitionings of a data set. The index is able to address all the above mentioned issues, viz., determining the number of clusters and evolving the proper partitioning as long as the clusters possess the property of symmetry. A Kd-tree-based data structure is used to reduce the complexity of computing the symmetry distance. Results demonstrating the superiority of the Sym-index in appropriately determining the proper partitioning and the number of clusters, as compared to two other recently proposed measures, namely the PS-index and I-index, are provided for three clustering methods viz., two recently developed genetic algorithm based clustering techniques and the average linkage clustering algorithm. Four artificial data sets and two real life data sets are considered for this purpose. The effectiveness of the proposed validity index is then demonstrated for automatically classifying different landcover regions in remote sensing imagery.