Social media platforms are being increasingly used by malicious actors to share unsafe content, such as images depicting sexual activity, cyberbullying, and self-harm. Consequently, major platforms use artificial intelligence (AI) and human moderation to obfuscate such images to make them safer. Two critical needs for obfuscating unsafe images is that an accurate rationale for obfuscating image regions must be provided, and the sensitive regions should be obfuscated (e.g. blurring) for users' safety. This process involves addressing two key problems: (1) the reason for obfuscating unsafe images demands the platform to provide an accurate rationale that must be grounded in unsafe image-specific attributes, and (2) the unsafe regions in the image must be minimally obfuscated while still depicting the safe regions. In this work, we address these key issues by first performing visual reasoning by designing a visual reasoning model (VLM) conditioned on pre-trained unsafe image classifiers to provide an accurate rationale grounded in unsafe image attributes, and then proposing a counterfactual explanation algorithm that minimally identifies and obfuscates unsafe regions for safe viewing, by first utilizing an unsafe image classifier attribution matrix to guide segmentation for a more optimal subregion segmentation followed by an informed greedy search to determine the minimum number of subregions required to modify the classifier's output based on attribution score. Extensive experiments on uncurated data from social networks emphasize the efficacy of our proposed method. We make our code available at: https://github.com/SecureAIAutonomyLab/ConditionalVLM

Cyberbullying has rapidly evolved with the evolution of online platforms, transcending traditional text-based forms to include images and other multimedia content. Two major challenges are identified in detecting cyberbullying images: recognizing cyberbullying-related visual factors and addressing the context-dependent nature of such images. In this paper, we conduct a comprehensive investigation of the ability of Large Vision-Language Models (LVLMs) to evaluate visual factors related to cyberbullying, and to interpret the context-dependent nature of such images. Furthermore, by proposing a diverse set of prompting strategies, we optimize LVLMs for cyberbullying image detection. In particular, through our carefully crafted Chain-of-Thought (CoT) methodology, we guide the model through structured reasoning pathways to interpret complex visual factors and account for their context. Our results show that the structured reasoning pathways significantly enhance model performance, achieving state-of-the-art accuracy and precision while remaining efficient by eliminating the need for any extensive training process.

Online hate is an escalating problem that negatively impacts the lives of Internet users, and is also subject to rapid changes due to evolving events, resulting in new waves of online hate that pose a critical threat. Detecting and mitigating these new waves present two key challenges: it demands reasoning-based complex decision-making to determine the presence of hateful content, and the limited availability of training samples hinders updating the detection model. To address this critical issue, we present a novel framework called HateGuard for effectively moderating new waves of online hate. HateGuard employs a reasoning-based approach that leverages the recently introduced chain-of-thought (CoT) prompting technique, harnessing the capabilities of large language models (LLMs). HateGuard further achieves prompt-based zero-shot detection by automatically generating and updating detection prompts with new derogatory terms and targets in new wave samples to effectively address new waves of online hate. To demonstrate the effectiveness of our approach, we compile a new dataset consisting of tweets related to three recently witnessed new waves: the 2022 Russian invasion of Ukraine, the 2021 insurrection of the US Capitol, and the COVID-19 pandemic. Our studies reveal crucial longitudinal patterns in these new waves concerning the evolution of events and the pressing need for techniques to rapidly update existing moderation tools to counteract them. Comparative evaluations against state-of-the-art approaches illustrate the superiority of our framework, showcasing a substantial 10.59% to 88% improvement in detecting the three new waves of online hate. Our work highlights the severe threat posed by the emergence of new waves of online hate and represents a paradigm shift in addressing this threat practically.

The (ab)use of encryption and compression in hiding illegal digital content complicates efforts by law enforcement agencies (LEAs) to procure evidence to support the elements of proof required in criminal prosecution. This reinforces the importance of designing solutions to determine the file type of an encrypted file (e.g., videos and still images in the context of illegal picture investigations), which can be used to build probable cause in a court order application to have the file decrypted. While machine learning (ML) has shown immense capabilities in several detection tasks, the suitability of ML for detecting file types in encrypted or compressed files has not been explored. Furthermore, since detecting file types in real-world LEA applications is a high-stake decision-making problem, existing ML techniques that do not provide prediction uncertainty are not as useful. In this work, we take the first step toward detecting file types in encrypted or compressed files using ML for LEA applications based on their Byte Frequency Distributions (BFD). We then compose a dataset1of BFDs of 300,000 encrypted and compressed data from 12,000 diverse files for five different file types. We conduct an in-depth analysis of our dataset and demonstrate the utility of ML techniques in detecting file types of content in encrypted and compressed files based on BFDs. Informed by these findings, we present our proposed framework, eDefender, designed to facilitate the detection of file types in encrypted and compressed files for LEA applications, by employing uncertainty quantification of detection scores based on ensembling. eDefender successfully flags directories with encrypted or compressed image/video-type files with an F1-score of 90.7%.

Recent studies regarding the COVID-19 pandemic have revealed the widespread propagation of hateful content during this period. While significant research has focused on COVID-19-related online hate in text (e.g., text-based tweets), the role of memes in propagating online hate during the pandemic has been largely overlooked. Memes are a popular mechanism used by Internet users to convey their thoughts and opinions on a variety of topics. However, memes have emerged as an important mechanism through which ideologically potent and hateful content spreads on social media platforms. In this work, we focus on investigating the role of memes in the propagation of online hate during the COVID-19 pandemic. We first collect a novel dataset of 4,001 COVID-19-related hateful memes and their replies over a 3-year period from Twitter. Then, we carry out the first large-scale investigation into the impact of these memes on Twitter users, by studying the psychological reactions of Twitter users to these memes using various text analysis methods. We find that COVID-19-related hateful memes have a significantly greater negative impact on Twitter users in comparison to text-based hateful tweets, and increasing negativity towards such memes over the 3-year period. Our new dataset of COVID-19-related hateful memes and findings from our work pave the way for studying the dissemination and moderation of COVID-19-related online hate through the medium of memes.

Hate speech has emerged as a major problem plaguing our social spaces today. While there have been significant efforts to address this problem, existing methods are still significantly limited in effectively detecting hate speech online. A major limitation of existing methods is that hate speech detection is a highly contextual problem, and these methods cannot fully capture the context of hate speech to make accurate predictions. Recently, large language models (LLMs) have demonstrated state-of-the-art performance in several natural language tasks. LLMs have undergone extensive training using vast amounts of natural language data, enabling them to grasp intricate contextual details. Hence, they could be used as knowledge bases for context-aware hate speech detection. However, a fundamental problem with using LLMs to detect hate speech is that there are no studies on effectively prompting LLMs for context-aware hate speech detection. In this study, we conduct a large-scale study of hate speech detection, employing five established hate speech datasets. We discover that LLMs not only match but often surpass the performance of current benchmark machine learning models in identifying hate speech. By proposing four diverse prompting strategies that optimize the use of LLMs in detecting hate speech. Our study reveals that a meticulously crafted reasoning prompt can effectively capture the context of hate speech by fully utilizing the knowledge base in LLMs, significantly outperforming existing techniques. Furthermore, although LLMs can provide a rich knowledge base for the contextual detection of hate speech, suitable prompting strategies play a crucial role in effectively leveraging this knowledge base for efficient detection.

During the global COVID-19 pandemic, people utilized social media platforms, especially Twitter, to spread and express opinions about the pandemic. Such discussions also drove the rise in COVID-related offensive speech. In this work, focusing on Twitter, we present a comprehensive analysis of COVID-related offensive tweets and their targets. We collected a COVID-19 dataset with over 747 million tweets for 30 months and fine-tuned a BERT classifier to detect offensive tweets. Our offensive tweets analysis shows that the ebb and flow of COVID-related offensive tweets potentially reflect events in the physical world. We then studied the targets of these offensive tweets. There was a large number of offensive tweets with abusive words, which could negatively affect the targeted groups or individuals. We also conducted a user network analysis, and found that offensive users interact more with other offensive users and that the pandemic had a lasting impact on some offensive users. Our study offers novel insights into the persistence and evolution of COVID-related offensive tweets during the pandemic

A large number of photos shared online often contain private user information, which can cause serious privacy breaches when viewed by unauthorized users. Thus, there is a need for more efficient privacy control that requires automatic detection of users' private photos. However, the automatic detection of users' private photos is a challenging task, since different users may have different privacy concerns and a generalized one-size-fits-all approach for private photo detection would not be suitable for most users. User-specific detection of private photos should, therefore, be investigated. Furthermore, for effective privacy control, the exact sensitive regions in private photos need to be pinpointed, so that sensitive content can be protected via different privacy control methods. In this paper, we propose a novel system, AutoPri, to enable automatic and user-specific content-based photo privacy control in online social networks. We collect a large dataset of 31, 566 private and public photos from real-world users and present important observations on photo privacy concerns. Our system can automatically detect private photos in a user-specific manner using a detection model based on a multimodal variational autoencoder and pinpoint sensitive regions in private photos with an explainable deep learning-based approach. Our evaluations show that AutoPri can effectively determine user-specific private photos with high accuracy (94.32%) and pinpoint exact sensitive regions in them to enable effective privacy control in user-centered online social networks.

Determining which photos are sensitive is difficult. Although emerging computer vision systems can label content items, previous attempts to distinguish private or sensitive content fall short. There is no human-centered taxonomy that describes what content is sensitive or how sharing preferences for content differs across recipients. To fill this gap, we introduce a new sensitive content elicitation method which surmounts limitations of previous approaches, and, using this new method, collected sensitive content from 116 participants. We also recorded participants' sharing preferences with 20 recipient groups. Next, we conducted a card sort to surface user-defined categories of sensitive content. Using data from these studies, we generated a taxonomy that identifies 28 categories of sensitive content. We also establish how sharing preferences for content differs across groups of recipients. This taxonomy can serve as a framework for understanding photo privacy, which can, in turn, inform new photo privacy protection mechanisms.

Online hate speech and offensive language have been widely recognized as critical social problems. To defend against this problem, several recent works have emerged that focus on the detection and explanation of hate speech and offensive language using machine learning approaches. Although these approaches are quite effective in the detection and explanation of hate speech and offensive language samples, they do not explore the impact of the representation of such samples. In this work, we introduce a novel, pronunciation-based representation of hate speech and offensive language samples to enable its detection with high accuracy. To demonstrate the effectiveness of our pronunciation-based representation, we extend an existing hate-speech and offensive language defense model based on deep Long Short-term Memory (LSTM) neural networks by using our pronunciation-based representation of hate speech and offensive language samples to train this model. Our work finds that the pronunciation-based presentation significantly reduces noise in the datasets and enhances the overall performance of the existing model.

The emergence of COVID-19 has engendered a new wave of online hate speech in social media platforms such as Twitter. Its widespread effects range from acts of cyber-harassment towards certain ethnic communities (e.g., the Asian community), to targeting older people belonging to age groups correlated with higher mortality rates (termed infamously as "Boomer Remover"). Thus, an urgent need arises for a timely mitigation of this new wave of online hate speech. In this work, we aim to discover the hate-related keywords linked to COVID-19 in hateful tweets posted on Twitter so that users posting such keywords can be asked to reconsider posting them. We first collect a new dataset of tweets targeting older people supplementing with a dataset targeting the Asian community. Then, we develop an approach to analyze the datasets with BERT (a transformer-based model) attention mechanism and discover 186 novel keywords targeting the Asian community and 100 keywords targeting older people. Based on our study, we then propose a control mechanism wherein a user can be asked to reconsider using certain sensitive words identified by our approach. We further perform an exploratory analysis of BERT attention mechanism and find that the most high-impact, long distance attentions are learned in the earlier or later layers of the model depending on the underlying data distribution. Our study indicates that the BERT model in some cases uses a hate keyword and an associated group or individual to make predictions, a finding that is inline with existing hate-speech research, which suggests that hate-speech is often aimed at certain groups or individuals.

We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN. Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters. CrescendoNet provides a new way to construct high performance deep convolutional neural networks with simple network architecture. Moreover, by investigating a various combination of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior, which gives CrescendoNet an anytime classification property. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training.

The privacy control models of current Online Social Networks (OSNs) are biased towards the content owners' policy settings. Additionally, those privacy policy settings are too coarse-grained to allow users to control access to individual portions of information that is related to them. Especially, in a shared photo in OSNs, there can exist multiple Personally Identifiable Information (PII) items belonging to a user appearing in the photo, which can compromise the privacy of the user if viewed by others. However, current OSNs do not provide users any means to control access to their individual PII items. As a result, there exists a gap between the level of control that current OSNs can provide to their users and the privacy expectations of the users. In this paper, we propose an approach to facilitate collaborative control of individual PII items for photo sharing over OSNs, where we shift our focus from entire photo level control to the control of individual PII items within shared photos. We formulate a PII-based multiparty access control model to fulfill the need for collaborative access control of PII items, along with a policy specification scheme and a policy enforcement mechanism. We also discuss a proof-of-concept prototype of our approach as part of an application in Facebook and provide system evaluation and usability study of our methodology.

Current collaborative photo privacy protection solutions can be categorized into two approaches: controlling the recipient, which restricts certain viewers' access to the photo, and controlling the content, which protects all or part of the photo from being viewed. Focusing on the latter approach, we introduce privacy-enhancing obfuscations for photos and conduct an online experiment with 271 participants to evaluate their effectiveness against human recognition and how they affect the viewing experience. Results indicate the two most common obfuscations, blurring and pixelating, are ineffective. On the other hand, inpainting, which removes an object or person entirely, and avatar, which replaces content with a graphical representation are effective. From a viewer experience perspective, blurring, pixelating, inpainting, and avatar are preferable. Based on these results, we suggest inpainting and avatar may be useful as privacy-enhancing technologies for photos, because they are both effective at increasing privacy for elements of a photo and provide a good viewer experience.

Cyberbullying can have a deep and long lasting impact on its victims, who are often adolescents. Accurately detecting cyberbullying helps prevent it. However, the noise and errors in social media posts and messages make detecting cyberbullying very challenging. In this paper, we propose a novel pronunciation based convolutional neural network (PCNN) to address this challenge. Upon observing that the pronunciation of misspelled words in informal online conversations is often unchanged, we used the phoneme codes of the text as the features for a convolutional neural network. This procedure corrects spelling errors that did not alter the pronunciation, thereby alleviating the problem of noise and bullying data sparsity. To overcome class imbalance, a common problem in cyberbullying datasets, we implement three techniques that include threshold-moving, cost function adjusting, and a hybrid solution in our model. We evaluate the performance of our models using two cyberbullying datasets collected from Twitter and Formspring.me. The results of our experiment show that PCNN can achieve improved recall and precision compared to baseline convolutional neural networks.