The incel subculture has gained increasing attention due to its toxic nature and its association with real-world violence. This paper investigates the prevalence and characteristics of violent threatful communication within incel forums, focusing on a platform known as Blackpill. We have trained a machine learning model to detect violent threatful language and analyzed the posts. The analysis concentrated on three key aspects: the identity of perpetrators (categorized into first-person, third-person, or generalized), the targets (individuals, groups, or general targets), and the types of violence described (general violence, sexual violence, self-harm, and military violence). The analysis showed that the most common type violent threatful communication involved generalized perpetrators targeting groups. Additionally, 13.5% of the violent threatful communication contained coded language, including references to video games to obscure violent intentions. A smaller proportion of the posts (4.1%) glorified past mass shooters and violent criminals.This research highlights the complexities of identifying violent rhetoric in online forums and the use of coded language to evade detection, emphasizing the need for refined models in threat detection.

Human language has many functions. Our communication on social media carries information about how we relate to ourselves and others, that is our identity, and we adjust our language to become more similar to our community - in the same way as we dress and style and act to show our commitment to the groups we belong to. Within a community, members adopt the community's language, and the common language becomes a unifying factor.

Harmful communication, such as toxic language and threats directed toward individuals or groups, is a common problem on most social media platforms and online spaces. While several approaches exist for detecting toxic language and threats in English, few attempts have detected such communication in Swedish. Thus, we used transfer learning and BERT to train two machine learning models: one that detects toxic language and one that detects threats in Swedish. We also examined the intersection between toxicity and threat. The models are trained on data from several different sources, with authentic social media posts and data translated from English. Our models perform well on test data with an F1-score above 0.94 for detecting toxic language and 0.86 for detecting threats. However, the models' performance decreases significantly when they are applied to new unseen social media data. Examining the intersection between toxic language and threats, we found that 20% of the threats on social media are not toxic, which means that they would not be detected using only methods for detecting toxic language. Our finding highlights the difficulties with harmful language and the need to use different methods to detect different kinds of harmful language.

One of the most challenging threats to the security of society is attacks from violent lone offenders. Identifying potential offenders is difficult since they act alone and do not necessarily communicate with others. However, several targeted violent attacks have been preceded by communication published on social media and the internet. Such communication is a valuable component when conducting risk and threat assessments.In this paper, we introduce a diagnostic measure of the risk of violent behavior based on text analysis. Using automated text analysis, we extract psychological variables and warning indicators from a given text and summarize these in an index that we denote as the general risk index. When developing the general risk index, we analyzed data (text) from 208 288 users on 32 online environments with diverse ideologies/orientations, including 76 previous violent lone offenders. A receiver operating characteristics (ROC) analysis showed that, when using the general risk index, it was possible to correctly classify between 90% and 96% of the cases depending on the comparison sample. These results support the predictive validity of the general risk index, suggesting that the risk index can be used to identify individuals with an increased risk of committing violent attacks that need further investigation.

For decades, threat assessment professionals have used structured professional judgment instruments to make decisions about, for example, the likelihood of violent behavior of an individual. However, with the increased use of social media, most people use online digital platforms to communicate, which is also the case for potential violent offenders. For example, many mass shootings in recent years have been preceded by communication in online forums. In this paper, we introduce methods to identify markers of the warning behaviors Leakage, Fixation, Identification, and Affiliation and examine their discriminant validity. Our results show that violent offenders score higher on these markers and that these markers were present among a significantly higher proportion of violent offenders as compared to the normal population. We argue that our method can be used to predict potential planned, purposeful, or instrumental targeted violence in written communication. Automated methods for detecting warning behavior from written communication can serve as a complement to traditional threat assessment and provides unique opportunities for threat assessment beyond traditional methods.

In this study, we trained three machine learning models to detect toxic language on social media. These models were trained using data from diverse sources to ensure that the models have a broad understanding of toxic language. Next, we evaluate the performance of our models on a dataset with samples of data from a large number of diverse online forums. The test dataset was annotated by three independent annotators. We also compared the performance of our models with Perspective API - a toxic language detection model created by Jigsaw and Google's Counter Abuse Technology team. The results showed that our classification models performed well on data from the domains they were trained on (Fl = 0.91, 0.91, & 0.84, for the RoBERTa, BERT, & SVM respectively), but the performance decreased when they were tested on annotated data from new domains (Fl = 0.80, 0.61, 0.49, & 0.77, for the RoBERTa, BERT, SVM, & Google perspective, respectively). Finally, we used the best-performing model on the test data (RoBERTa, ROC = 0.86) to examine the frequency (/proportion) of toxic language in 21 diverse forums. The results of these analyses showed that forums for general discussions with moderation (e.g., Alternate history) had much lower proportions of toxic language compared to those with minimal moderation (e.g., 8Kun). Although highlighting the complexity of detecting toxic language, our results show that model performance can be improved by using a diverse dataset when building new models. We conclude by discussing the implication of our findings and some directions for future research.

Suicide is a global health problem with more than 700,000 individuals dying by self-destruction each year, yet it is classified as a low base rate behavior that is difficult to prognosticate. Aiming to advance suicide prediction and prevention, we examined the potential use of machine learning and text analyses models to predict suicide risk based on written communications. Specifically, we used a dataset consisting of more than 27,000 general writings unrelated to suicide, 193 genuine suicide notes from individuals who committed suicide, and an additional 89 suicide posts shared on sub-Reddits for an in-the-wild test to examine the prediction accuracy of two machine learning models (SVM & RoBERTa) and a linguistic marker model. Our tests showed that the machine learning models performed better than the linguistic marker model when examined on the test data. However, the linguistic marker model achieved higher results in the wild, correctly classifying 88% of written communications as a "high risk of suicide" versus 56% and 70% of the machine learning models. The best in-the-wild performing model was adopted in an online suicide risk assessment tool called Edwin to honor Edwin Shneidman for his numerous contributions to the field of suicidology. Finally, discrepancies between training and real-world data, vocabulary variation across domains, and the limited number of benchmarks constitute limitations that need to be addressed in future research.

One of the most challenging threats towards the security of the society is attacks from violent lone offenders, individuals that act alone or with minimal help from others without any economic gains or direct orders from organizations. Over the past few years, several terror attacks have been accompanied by manifestos published on social media platforms that outline ideology, motivation, and in some cases tactical choices. The trend in publishing manifestos and other communication on social media sites before committing an attack has increased the need for threat assessment in digital environments. Most existing methods for threat assessment are developed to be used in offline settings where information about an individual is accessible and cases where the individual is present and can answer questions. In this paper, we present seven indicators that can be used to assess the potential threat of violence based on digital communication only. The seven indicators are designed to be used when analyzing texts and can be seen as a complement to other risk assessment protocols.

In this paper, we present a tool for assessing the risk of targeted violence in written communication: the profile risk assessment tool (PRAT). The tool reads a text and extracts a profile based on a set of assessment factors including, for example, personality, emotionality, identification, fixation, and leakage. These factors have all been shown to have relevance in risk assessment by previous research. In PRAT, each assessment factor is broken down into a set of indicators to enable creating a profile for subsequent threat. To our knowledge, PRAT is the first threat assessment tool for digital communication.

Analysis and mining of social media has become an important research area. A challenging problem in this area consists in the identification of a group of users with similar patterns. In this paper, we propose the classification of users based on their activity profiles (e.g., periods of the day when the user is most and least active in online communications). Activity profiles can be useful for many purposes, such as marketing and user behavior analysis. They can also serve as a basis for other techniques such as stylometric and time analysis in order to increase the precision and scalability of multiple aliases identification techniques. We have implemented a prototype tool and applied it on a dataset from the ICWSM data set Boards.ie, showing the usefulness of our classification.

Monitoring and analysis of web forums is becoming important for intelligence analysts around the globe since terrorists and extremists are using forums for spreading propaganda and communicating with each other. Various tools for analyzing the content of forum postings and identifying aliases that need further inspection by analysts have been proposed throughout literature, but a problem related to this is that individuals can make use of several aliases. In this paper we propose a number of matching techniques for detecting forum users who make use of multiple aliases. By combining different techniques such as time profiling and stylometric analysis of messages the accuracy of recognizing users with multiple aliases increases, as shown in experiments conducted on the ICWSM dataset boards.ie.