The proliferation of multimodal content on social media presents significant challenges in understanding and moderating complex, contextdependent issues such as misinformation, hate speech, and propaganda.While efforts have been made to develop resources and propose new methods for automatic detection, limited attention has been given to jointly modeling label detection and the generation of explanationbased rationales, which often leads to degraded classification performance when trained simultaneously.To address this challenge, we introduce MemeXplain, an explanation-enhanced dataset for propagandistic memes in Arabic and hateful memes in English, making it the first large-scale resource for these tasks.To solve these tasks, we propose a multistage optimization approach and train Vision-Language Models (VLMs).Our results show that this strategy significantly improves both label detection and explanation generation quality over the base model, outperforming the current state-of-the-art with an absolute improvement of ∼ 1.4% (Acc) on ArMeme and ∼ 2.2% (Acc) on Hateful Memes.For reproducibility and future research, we aim to make the MemeXplain dataset and scripts publicly available.1

With the rise of digital communication memes have become a significant medium for cultural and political expression that is often used to mislead audience.Identification of such misleading and persuasive multimodal content become more important among various stakeholders, including social media platforms, policymakers, and the broader society as they often cause harm to the individuals, organizations and/or society.While there has been effort to develop AI based automatic system for resource rich languages (e.g., English), it is relatively little to none for medium to low resource languages.In this study, we focused on developing an Arabic memes dataset with manual annotations of propagandistic content. 1 We annotated ∼ 6K Arabic memes collected from various social media platforms, which is a first resource for Arabic multimodal research.We provide a comprehensive analysis aiming to develop computational tools for their detection.We made the dataset publicly available for the community.

Abstract Recent research in disaster informatics demonstrates a practical and important use case of artificial intelligence to save human lives and suffering during natural disasters based on social media contents (text and images). While notable progress has been made using texts, research on exploiting the images remains relatively under-explored. To advance image-based approaches, we propose MEDIC ( https://crisisnlp.qcri.org/medic/index.html ), which is the largest social media image classification dataset for humanitarian response consisting of 71,198 images to address four different tasks in a multi-task learning setup. This is the first dataset of its kind: social media images, disaster response, and multi-task learning research. An important property of this dataset is its high potential to facilitate research on multi-task learning , which recently receives much interest from the machine learning community and has shown remarkable results in terms of memory, inference speed, performance, and generalization capability. Therefore, the proposed dataset is an important resource for advancing image-based disaster management and multi-task machine learning research. We experiment with different deep learning architectures and report promising results, which are above the majority baselines for all tasks. Along with the dataset, we also release all relevant scripts ( https://github.com/firojalam/medic ).

Laws on privacy preservation challenges supervised learning algorithms in industrial applications and could be an obstacle for the artificial intelligence solutions. In the literature, this issue is never discussed for the algorithm’s design. Indeed, algorithms do not behave the same when the input is modified to protect privacy. Particularly, the unmodified data samples predicts with low confidences show high vulnerability to decision changes. To overcome this challenge, we propose a novel solution that enhances classifier’s robustness by particularly addressing the vulnerable samples. It consists of a novel formulation of the learning objective by hybridizing similarity learning, decision margin and intra-class distance. Experimental results and evaluation on a challenging vehicle image dataset exhibit the high effectiveness and potentials of our method for the privacy preserving classification problems.

In this paper, a real-world problem of the vehicle-type classification for automatic toll collection (ATC) is considered. This problem is very challenging because any loss of accuracy even of the order of 1% quickly turns into a significant economic loss. To deal with such a problem, many companies currently use optical sensors (OSs) and human observers to correct the classification errors. Herein, a novel vehicle classification method is proposed. It consists in regularizing the problem using one camera to obtain vehicle class probabilities using a set of convolutional neural networks (CNNs) and, then, uses the Gradient boosting-based classifier to fuse the continuous class probabilities with the discrete class labels obtained from the OS. The method is evaluated on a real-world dataset collected from the toll collection points of the VINCI Autoroutes French network. The results show that it performs significantly better than the existing ATC system and, hence, will vastly reduce the workload of human operators.

Image Quality (IQ) plays significant role for both human vision and artificial vision applications. This last decade, the number of camera increases exponentially, but the exploitation of the information depends on the quality of these acquired images, and sequences. A large number of objective blind IQ assessment (OBIQA) methods were proposed which find a global IQ estimation regardless the application. In this work, we assume that is impossible that such a method can be applied to all applications, however, for a given problem an objective model of the quality can be provided. Consequently, this paper addresses this issue for an automatic vehicle type classification application and proposes a novel OBIQA based classification approach. The proposed method first extracts a set of selective image features, then learns to classify images accordingly. In other terms, it aims to prevent misclassification and localize the source of poor images. Experiments show that it performs better than the state-of-the-art methods and can be used for similar applications.

Toll Vehicle Classification is an important task. Indeed, it has many uses in traffic management and toll collection systems. In this paper, Vinci Autoroutes group Networks (the biggest French Highways concession) are considered, where every year, millions of vehicles are classified in realtime. Then, a small decrease in classification performance can have serious economic losses. Therefore, the accuracy and the time complexity become critical for the toll collection system. The current classification algorithm uses the scene features' to detect vehicles classes. However, it requires a large labeled datasets, and has a limitations when multiple vehicles are in the scene. Herein, we propose a novel context-aware vehicle classification method that takes profit from the semantic spatial relationship of the objects. The experiments show that our method is performing as accurately as the existing model with significantly lower labeled datasets (74 times smaller). Moreover, the obtained accuracy of the proposed method is 99.97% compared to 99.79% achieved by the current method when using the same training set.

In this work, a challenging vehicle type classification problem for automatic toll collection task is considered, which is currently accomplished with an Optical Sensors (OS) and corrected manually. Indeed, the human operators are engaged to manually correct the OS misclassified vehicles by observing the images obtained from the camera. In this paper, we propose a novel vehicle classification algorithm, which first uses the camera images to obtain the vehicle class probabilities using several Convolutional Neural Networks (CNNs) models and then uses the Gradient Boosting based classifier to fuse the continuous class probabilities with the discrete class labels obtained from two optical sensors. We train and evaluate our method using a challenging dataset collected from the cameras of the toll collection points. Results show that our method performs significantly (98.22% compared to 75.11%) better than the existing automatic toll collection system and, hence will vastly reduce the workload of the human operators. Moreover, we provide an in-depth analysis w.r.t. the learning strategies:e.g., choice of the optimization algorithm of the CNN model. Our results and analysis highlights interesting perspectives and challenges for the future work.

International audience

Recent advances in depth imaging sensors provide easy access to the synchronized depth with color, called RGB-D image. In this paper, we propose an unsupervised method for indoor RGB-D image segmentation and analysis. We consider a statistical image generation model based on the color and geometry of the scene. Our method consists of a joint color-spatial-directional clustering method followed by a statistical planar region merging method. We evaluate our method on the NYU depth database and compare it with existing unsupervised RGB-D segmentation methods. Results show that, it is comparable with the state of the art methods and it needs less computation time. Moreover, it opens interesting perspectives to fuse color and geometry in an unsupervised manner.

Model Based Clustering (MBC) is a method that estimates a model for the data and produces probabilistic clustering. In this paper, we propose a novel MBC method to cluster three dimensional directional features. We assume that the features are generated from a finite statistical mixture model based on the von Mises-Fisher (vMF) distribution. The core elements of our proposed method are: (a) generate a set of vMF Mixture Models (vMFMM) and (b) select the optimal model using a parsimony based approach with information criteria. We empirically validate our proposed method by applying it on simulated data. Next, we apply it to cluster image normals in order to perform depth image analysis.

In this paper, we propose an unsupervised clustering method for axially symmetric directional unit vectors. Our method exploits the Watson distribution and Bregman Divergence within a Model Based Clustering framework. The main objectives of our method are: (a) provide efficient solution to estimate the parameters of a Watson Mixture Model (WMM), (b) generate a set of WMMs and (b) select the optimal model. To this aim, we develop: (a) an efficient soft clustering method, (b) a hierarchical clustering approach in parameter space and (c) a model selection strategy by exploiting information criteria and an evaluation graph. We empirically validate the proposed method using synthetic data. Next, we apply the method for clustering image normals and demonstrate that the proposed method is a potential tool for analyzing the depth image.

We present an adaptive weight based superpixel segmentation method for the goal of creating mesh representation that respects the 3D scene structure. We propose a new fusion framework which employs both dense optical flow and color images to compute the probability of boundaries. The main contribution of this work is that we introduce a new color and optical flow pixel-wise weighting model that takes into account the non-linear error distribution of the depth estimation from optical flow. Experiments show that our method is better than the other state-of-art methods in terms of smaller error in the final produced mesh.

BanglaOCR is currently the only open source optical character recognition (OCR) software for the Bangla (Bengali) script developed by the Center for Research on Bangla Language Processing (CRBLP). Tesseract, maintained by Google, is considered to be one of the most accurate free open source OCR engines currently available. In this paper, we present a new OCR for the Bangla/Bengali script that combines the recognition power of Tesseract and the Bangla script processing power of BanglaOCR by integrating the Tesseract recognition engine into BanglaOCR. We first present the complete methodology to build the combined OCR, followed by the implementation strategy. In this paper, we focus on the training data preparation process, Tesseract integration procedure and the post-processing techniques. The techniques described in this paper can be readily applied to build OCRs for other scripts as well.