Online Library

Deplatforming websites—removing infrastructure services they need to operate, such as website hosting—can reduce the spread and reach of extremism and hate online, but when does deplatforming succeed? This report shows that deplatforming can decrease the popularity of extremist websites, especially when done without warning. We present four case studies of English-language, U.S.-based extremist websites that were deplatformed: the Daily Stormer, 8chan/8kun, TheDonald.win/Patriots.win, and Nicholas Fuentes/America First. In all of these cases, the infrastructure service providers considered deplatforming only after highly publicized or violent events, indicating that at the infrastructure level, the bar to deplatforming is high. All of the site administrators in these four cases also elected to take measures to remain online after they were deplatformed. To understand how deplatforming affected these sites, we collected and analyzed publicly available data that measures website-popularity rankings over time.

We learned four important lessons about how deplatforming affects extremist websites:

• It can cause popularity rankings to decrease immediately.
• It may take users a long time to return to the website. Sometimes, the website never regains its previous popularity.
• Unexpected deplatforming makes it take longer for the website to regain its previous popularity levels.
• Replicating deplatformed services such as discussion forums or live-streaming video products on a stand-alone website presents significant challenges, including higher costs and smaller audiences.

Our findings show that fighting extremism online requires not only better content moderation and more transparency from social media companies but also cooperation from infrastructure providers like Cloudflare, GoDaddy, and Google, which have avoided attention and critique.

Increasingly, global society is focused on the harm caused by material accessed on the internet. Individuals of all ages and backgrounds–policy makers, academics, politicians, mental health professionals, medical professionals and the general public–are seeking to understand the processes that govern the relationship between exposure to harmful content online and later harmful cognitions and behaviors. Several commentators have placed into stark focus several questions surrounding the use (and misuse) of the internet, and prevention of harm from it. Three issues come immediately to mind: First, what can be done to prevent the manifestation of harmful activities stemming from being online? Second, what is the underlying relationship between harmful material accessed online, and harmful behavioral outcomes? And third, what is the best approach (and how to we know what the “best” is) to prevent the manifestation of harmful behavior stemming from harmful content online? Rather than looking at each of these issues as a stovepipe, and seeking to answer each in isolation, in this chapter we adopt an integrationist perspective on the issue of preventing online harm online. Specifically, we will look at the theories of how and why exposure to extreme and harmful material online can cause negative real-world outcomes. We focus on the innovative Google-led program Jigsaw, 1 to explore the many and varied approaches that can be adopted to challenge and counter the issues of extreme and harmful material online. Finally, leveraging new approaches from public health (Su et al., 2021; Shortland et al., 2021), we outline the major ethical, practical, and theoretical issues that pervade technology-based efforts to tackle the issue of extreme and harmful material online.

Since the fall of the Caliphate, the activities and overall threat posed by Western jihadists has undoubtedly diminished. 1 A recent study released by the Program on Extremism, for example, demonstrated a steady decline in jihadist activity in the United States (US) since 2020. In this three-year period, only twenty-nine Americans have been charged, compared to eighty-two in the previous three years. 2 Similarly, Europe has experienced a steady annual reduction in jihadist arrests from 718 in 2016 to 260 in 2021.3 This is the second period since 9/11 where there has been a marked decline in the ability to radicalize and mobilize people to support the jihadi movement or conduct attacks in the West. 4 Whether or not the cycle of (relative) success and decline will repeat itself depends on a number of factors, some of which are harder to predict than others. What is clear for now is that there remains a pool of committed Western jihadists who are working to keep the movement alive in the US and Europe in hope that future opportunities for mass mobilization may arise. This report investigates how Western jihadists and efforts to radicalize Westerners have adapted to the post-Caliphate reality and the current downward trajectory and complements the Program’s recent quantitative analysis of the Islamic State (IS) threat in the US Some Western jihadi strategies have remained the same, such as the calls to conduct lone actor attacks in the West. However, there have been some marked shifts in strategic communications. For instance, in some cases the discourse about Muslim grievances resembles those that were prevalent during the previous lull in Western jihadist activity, such as a refocusing on Muslim prisoners in the West. The report opens with an overview of how Western jihadists have responded to the online post-Caliphate world in which many of the online platforms popular among jihadists have become increasingly proactive in removing jihadist accounts and content. Here, there is evidence of a reversion to an earlier, pre-Caliphate, era of online jihadism in which much activity and file sharing took place on forums rather than mainstream social media platforms.

Online platforms have banned (“deplatformed”) influencers, communities, and even entire websites to reduce content deemed harmful. Deplatformed users often migrate to alternative platforms, which raises concerns about the effectiveness of deplatforming. Here, we study the deplatforming of Parler, a fringe social media platform, between 2021 January 11 and 2021 February 25, in the aftermath of the US Capitol riot. Using two large panels that capture longitudinal user-level activity across mainstream and fringe social media content (N = 112, 705, adjusted to be representative of US desktop and mobile users), we find that other fringe social media, such as Gab and Rumble, prospered after Parler’s deplatforming. Further, the overall activity on fringe social media increased while Parler was offline. Using a difference-in-differences analysis (N = 996), we then identify the causal effect of deplatforming on active Parler users, finding that deplatforming increased the probability of daily activity across other fringe social media in early 2021 by 10.9 percentage points (pp) (95% CI [5.9 pp, 15.9 pp]) on desktop devices, and by 15.9 pp (95% CI [10.2 pp, 21.7 pp]) on mobile devices, without decreasing activity on fringe social media in general (including Parler). Our results indicate that the isolated deplatforming of a major fringe platform was ineffective at reducing overall user activity on fringe social media.

Radical/extremist Islamist actors use social media to disseminate uncompromising stories of monist religious political orders and identities. As a reaction, counter-movements to online Islamist radicalism/extremism emerged in Western societies (and beyond), while uncertainty about effective outcomes remains widespread. In a bid to understand how inclusionary and exclusionary discursive spaces are created, we ask: How do some Muslim actors create discursive spaces open to self-reflection, pluralism and liberal-democratic principles, while others construct illiberal, particularistic and non/anti-democratic spaces? To respond to this question, we compare two contrasting storytellers, one who agitates for exclusionary Islamist radicalism/extremism (Generation Islam) and one who offers inclusionary prevention and deradicalization work against that (Jamal al-Khatib). We draw on novel narrative approaches to the Discourse Historical Approach (DHA) in Critical Discourse Studies (CDS), via which we compare text-level and context-level narratives disseminated about three Muslim-related crises: the racist terrorist attacks/genocide to represent the national, European and global level. Our two-layered, DHA-inspired narrative analysis illustrates that, at the level of text, narrative persuasion varies between both contrasting actors. While Jamal al-Khatib disseminates persuasive stories, Generation Islam is much less invested in narrative persuasion; it seems to address an already convinced audience. These two text-level strategies reveal their meaning in two antagonistic narrative genres: Jamal al-Khatib’s “self-reflexive savior” creates an inclusionary discursive space represented in a self-ironic narrative genre, while Generation Islam’s ”crusading savior” manufactures an exclusionary discursive space represented in a romance featuring a nostalgic return to the particularistic Islamic umma.

Cooperation between law enforcement and tech companies is widely regarded as necessary to tackle online terrorist content. Both sectors have publicly stated their commitment to working together and there are examples of mutual cooperation. Yet there are also impediments to such collaboration, including different cultures and operating practices, and there have been high-profile instances of non-cooperation. The informality of existing collaborations has also led to concerns about censorship, mission creep and a lack of accountability and oversight.

The focus of this report is on how to resolve the impediments to closer cooperation between law enforcement and the tech sector in order to realise the benefits of mutual collaboration, while simultaneously addressing concerns about due process and accountability. The report utilises an interview-based methodology to examine the experiences and opinions of personnel from both sectors who have first-hand experience of mutual cooperation. It provides empirically grounded insights into this under-researched topic.