Online Library

Increasingly, global society is focused on the harm caused by material accessed on the internet. Individuals of all ages and backgrounds–policy makers, academics, politicians, mental health professionals, medical professionals and the general public–are seeking to understand the processes that govern the relationship between exposure to harmful content online and later harmful cognitions and behaviors. Several commentators have placed into stark focus several questions surrounding the use (and misuse) of the internet, and prevention of harm from it. Three issues come immediately to mind: First, what can be done to prevent the manifestation of harmful activities stemming from being online? Second, what is the underlying relationship between harmful material accessed online, and harmful behavioral outcomes? And third, what is the best approach (and how to we know what the “best” is) to prevent the manifestation of harmful behavior stemming from harmful content online? Rather than looking at each of these issues as a stovepipe, and seeking to answer each in isolation, in this chapter we adopt an integrationist perspective on the issue of preventing online harm online. Specifically, we will look at the theories of how and why exposure to extreme and harmful material online can cause negative real-world outcomes. We focus on the innovative Google-led program Jigsaw, 1 to explore the many and varied approaches that can be adopted to challenge and counter the issues of extreme and harmful material online. Finally, leveraging new approaches from public health (Su et al., 2021; Shortland et al., 2021), we outline the major ethical, practical, and theoretical issues that pervade technology-based efforts to tackle the issue of extreme and harmful material online.

Since the fall of the Caliphate, the activities and overall threat posed by Western jihadists has undoubtedly diminished. 1 A recent study released by the Program on Extremism, for example, demonstrated a steady decline in jihadist activity in the United States (US) since 2020. In this three-year period, only twenty-nine Americans have been charged, compared to eighty-two in the previous three years. 2 Similarly, Europe has experienced a steady annual reduction in jihadist arrests from 718 in 2016 to 260 in 2021.3 This is the second period since 9/11 where there has been a marked decline in the ability to radicalize and mobilize people to support the jihadi movement or conduct attacks in the West. 4 Whether or not the cycle of (relative) success and decline will repeat itself depends on a number of factors, some of which are harder to predict than others. What is clear for now is that there remains a pool of committed Western jihadists who are working to keep the movement alive in the US and Europe in hope that future opportunities for mass mobilization may arise. This report investigates how Western jihadists and efforts to radicalize Westerners have adapted to the post-Caliphate reality and the current downward trajectory and complements the Program’s recent quantitative analysis of the Islamic State (IS) threat in the US Some Western jihadi strategies have remained the same, such as the calls to conduct lone actor attacks in the West. However, there have been some marked shifts in strategic communications. For instance, in some cases the discourse about Muslim grievances resembles those that were prevalent during the previous lull in Western jihadist activity, such as a refocusing on Muslim prisoners in the West. The report opens with an overview of how Western jihadists have responded to the online post-Caliphate world in which many of the online platforms popular among jihadists have become increasingly proactive in removing jihadist accounts and content. Here, there is evidence of a reversion to an earlier, pre-Caliphate, era of online jihadism in which much activity and file sharing took place on forums rather than mainstream social media platforms.

Online platforms have banned (“deplatformed”) influencers, communities, and even entire websites to reduce content deemed harmful. Deplatformed users often migrate to alternative platforms, which raises concerns about the effectiveness of deplatforming. Here, we study the deplatforming of Parler, a fringe social media platform, between 2021 January 11 and 2021 February 25, in the aftermath of the US Capitol riot. Using two large panels that capture longitudinal user-level activity across mainstream and fringe social media content (N = 112, 705, adjusted to be representative of US desktop and mobile users), we find that other fringe social media, such as Gab and Rumble, prospered after Parler’s deplatforming. Further, the overall activity on fringe social media increased while Parler was offline. Using a difference-in-differences analysis (N = 996), we then identify the causal effect of deplatforming on active Parler users, finding that deplatforming increased the probability of daily activity across other fringe social media in early 2021 by 10.9 percentage points (pp) (95% CI [5.9 pp, 15.9 pp]) on desktop devices, and by 15.9 pp (95% CI [10.2 pp, 21.7 pp]) on mobile devices, without decreasing activity on fringe social media in general (including Parler). Our results indicate that the isolated deplatforming of a major fringe platform was ineffective at reducing overall user activity on fringe social media.

Radical/extremist Islamist actors use social media to disseminate uncompromising stories of monist religious political orders and identities. As a reaction, counter-movements to online Islamist radicalism/extremism emerged in Western societies (and beyond), while uncertainty about effective outcomes remains widespread. In a bid to understand how inclusionary and exclusionary discursive spaces are created, we ask: How do some Muslim actors create discursive spaces open to self-reflection, pluralism and liberal-democratic principles, while others construct illiberal, particularistic and non/anti-democratic spaces? To respond to this question, we compare two contrasting storytellers, one who agitates for exclusionary Islamist radicalism/extremism (Generation Islam) and one who offers inclusionary prevention and deradicalization work against that (Jamal al-Khatib). We draw on novel narrative approaches to the Discourse Historical Approach (DHA) in Critical Discourse Studies (CDS), via which we compare text-level and context-level narratives disseminated about three Muslim-related crises: the racist terrorist attacks/genocide to represent the national, European and global level. Our two-layered, DHA-inspired narrative analysis illustrates that, at the level of text, narrative persuasion varies between both contrasting actors. While Jamal al-Khatib disseminates persuasive stories, Generation Islam is much less invested in narrative persuasion; it seems to address an already convinced audience. These two text-level strategies reveal their meaning in two antagonistic narrative genres: Jamal al-Khatib’s “self-reflexive savior” creates an inclusionary discursive space represented in a self-ironic narrative genre, while Generation Islam’s ”crusading savior” manufactures an exclusionary discursive space represented in a romance featuring a nostalgic return to the particularistic Islamic umma.

Cooperation between law enforcement and tech companies is widely regarded as necessary to tackle online terrorist content. Both sectors have publicly stated their commitment to working together and there are examples of mutual cooperation. Yet there are also impediments to such collaboration, including different cultures and operating practices, and there have been high-profile instances of non-cooperation. The informality of existing collaborations has also led to concerns about censorship, mission creep and a lack of accountability and oversight.

The focus of this report is on how to resolve the impediments to closer cooperation between law enforcement and the tech sector in order to realise the benefits of mutual collaboration, while simultaneously addressing concerns about due process and accountability. The report utilises an interview-based methodology to examine the experiences and opinions of personnel from both sectors who have first-hand experience of mutual cooperation. It provides empirically grounded insights into this under-researched topic.

Unlike one-to-one threats, terrorist threat texts constitute a form of violence and a language crime that is committed in a complex context of public intimidation, and are communicated publicly and designed strategically to force desired sociopolitical changes . Contributing to law enforcement and threat assessors’ fuller understanding of the discursive nature of threat texts in terrorism context, this paper examines how language is used dialogically to communicate threats and to construct both the purpose of threatened actions and the victims. The paper uses a critical discourse analytic approach and takes a set of eleven digital threat texts made by two jihadists as a case study. It draws on van Dijk’s concept of ideology , the law enforcement-based taxonomy of threat types as reported by Napier and Mardigian , van Leeuwen’s model of social actor representation and discursive construction of purpose of social actions , and Martin and White’s Engagement system . The analysis reveals victims specified and genericised, excluded and adversary. This linguistic construction is underpinned by a dichotomous conceptualisation of the social actors’ affiliations, positions, values, cultural activities, goals, and material and symbolic resources. The threats are delivered to the victims, agents acting on their behalf (e.g. security forces) or property associated with them (e.g. oil refinery), and are of two primary types—direct, and veiled. The former are predominant and serve inter alia to augment the public-intimidation impact of terrorist discourse. Threatened violence is of goal-, means- and/or effect-oriented social purposes, which suggest a categorisation of threats based on these purposes. The analysis reveals a dialectic, refutative nature of argumentation, and a discourse pregnant with heteroglossic utterances that contract (i) to close off and disalign with state officials’ contradictory voices, and (ii) to produce tension, providing clues to terrorists’ motivations and what constitutes the heart of political violence.