Online Library

Do social media and search companies exacerbate antisemitism and hate through their own design and system functions? In this joint study by the ADL Center for Technology and Society (CTS) and Tech Transparency Project (TTP), we investigated search functions on both social media platforms and Google. Our results show how these companies’ own tools–such as autocomplete and auto-generation of content–made finding and engaging with antisemitism easier and faster.1 In some cases, the companies even helped create the content themselves.

Our researchers compiled a list of 130 hate groups and movements from ADL’s Glossary of Extremism, picking terms that were tagged in the glossary with all three of the following categories: “groups/movements,” “white supremacist,” and “antisemitism.”2 The researchers then typed each term into the respective search bars of Facebook, Instagram and YouTube, and recorded the results.

Social media platforms play a key role in fostering the outreach of extremism by influencing the views, opinions, and perceptions of people. These platforms are increasingly exploited by extremist elements for spreading propaganda, radicalizing, and recruiting youth. Hence, research on extremism detection on social media platforms is essential to curb its influence and ill effects. A study of existing literature on extremism detection reveals that it is restricted to a specific ideology, binary classification with limited insights on extremism text, and manual data validation methods to check data quality. In existing research studies, researchers have used datasets limited to a single ideology. As a result, they face serious issues such as class imbalance, limited insights with class labels, and a lack of automated data validation methods. A major contribution of this work is a balanced extremism text dataset, versatile with multiple ideologies verified by robust data validation methods for classifying extremism text into popular extremism types such as propaganda, radicalization, and recruitment. The presented extremism text dataset is a generalization of multiple ideologies such as the standard ISIS dataset, GAB White Supremacist dataset, and recent Twitter tweets on ISIS and white supremacist ideology. The dataset is analyzed to extract features for the three focused classes in extremism with TF‐IDF unigram, bigrams, and trigrams features. Additionally, pretrained word2vec features are used for semantic analysis. The extracted features in the proposed dataset are evaluated using machine learning classification algorithms such as multinomial Naïve Bayes, support vector machine, random forest, and XGBoost algorithms. The best results were achieved by support vector machine using the TF‐IDF unigram model confirming 0.67 F1 score. The proposed multi‐ideology and multiclass dataset shows comparable performance to the existing datasets limited to single ideology and binary labels.

The whitepaper has three parts. The first part provides some contextual background, describing the diverse range of online services utilised by terrorists and extremists and the process by which propaganda is disseminated online. The second part details industry responses. As well as referrals from users and law enforcement, it describes the use of AI for proactive detection and collaborative, cross-platform initiatives. The third part describes four issues for discussion: transparency; definitional clarity; the impact on those targeted; and, the use of online data for predictive purposes.

Over the past decade, a large number of extremist and hate groups have turned to internet platforms to inspire mass violence. Currently, there is little reliable evidence on how such campaigns radicalize targeted audiences. We provide systematic, large-scale, microevidence on the effect of Islamic State propaganda on social media. We use several machine learning algorithms to detect recruitment messages in online propaganda, identify their dissemination on Twitter, and quantify the reactions of exposed users. Analyzing content produced by the Islamic State between 2015 and 2016 shows that propaganda conveying the material, spiritual, and social benefits of joining ISIS increased online support for the group, while content displaying brutal violence decreased endorsement of ISIS across a wide range of videos. Only the group’s most extreme supporters reacted positively to violent propaganda.

Mobilization is central to the emergence, survival and success of armed groups challenging the state, and has lately expanded to new arenas with the rise of social media. Using a new dataset of rebel group Twitter use, we examined the topics contained in rebel group social media communications to understand how different messaging strategies impact civilian engagement with rebel messages. Rather than benefiting solely from direct calls to action, we found that rebel groups also increased civilian engagement through indirect messages of self-promotion. While direct appeals received more engagement than indirect appeals, their effects were tempered by audience fatigue when relied on too heavily. We additionally found that including images further enhanced the impact of a mobilizing message. These findings expanded our understanding of rebel communications and mobilization, with important implications for combating the use of social media as a recruitment tool for violent extremism.