Part One – The Value of a Comparative Approach
By Stuart Macdonald and Allysa Czerwinsky
There is no doubt that terrorists and violent extremists use AI. As well as its documented use in the attacks in Las Vegas, Florida State University, Palm Springs and Tumbler Ridge, researchers have evidenced its use through both open-source and interview-based studies.
Alongside this scholarship, another strand of research has taken as its starting point the features and affordances of different AI tools. It assesses the various potential ways in which these might be utilised. Then it extrapolates from these, to evaluate their attractiveness to malevolent actors.
The aim of our new report Artificial Intelligence, Violent Extremism and Nihilistic Violence is to combine the best of these two approaches: to produce rigorous empirically-grounded research on how AI is actually being used by extremists, the findings of which are capable of wider generalisability and application. Specifically, the report adopts a comparative approach, examining the variety of ways in which a range of AI tools are exploited by four categories of threat actor: Salafi-jihadism, the extreme right-wing, incels and violent misogyny, and networks associated with nihilistic extreme violence and other hybridised threats.
This blog is the first of a two-part series. Here we outline one subset of our findings, on the use of AI to create and edit images and videos. Our aim is to show that the same AI capability can have markedly different meanings, uses, and risks depending on the threat actor. The second blog will build on this by exploring the important implications of this finding for the AI safety field.
Salafi-jihadism
Across all four categories of threat actor, AI tools were used to generate and edit image and video content. But the purpose and effects of this content diverged significantly. In the Salafi-jihadist context, AI-generated content largely avoided features such as gore or violence, which could trigger security mechanisms, while also avoiding the depiction of human faces, which is not permissible in Salafi Islam. Instead, this content tended to favour symbolic and theologically relevant imagery. Frequently, it evoked the aesthetics associated with classic catchphrases of official IS propaganda, such as AI-generated images of lions, lion cubs and horses. These were typically combined with nasheeds or other elements to ensure proper interpretation among the group’s followers.
Other images aimed to highlight the mujahideen’s military prowess – using AI-generated pictures of armed horsemen, pickup trucks or anonymous, usually masked, mujahideen – or to create a reflective, spiritually resonant atmosphere by, for example, portraying prostrating individuals or prayer sites. Other techniques for circumventing content moderation included the use of AI-powered interior design generators to create images of luxurious rooms, typically featuring a television screen displaying propaganda videos. And the use of optical illusions, in which images of popular Salafi-Jihadist leaders are transformed into landscapes or objects, requiring the user to determine the pattern and recognise the person concealed therein.
Extreme Right-Wing
Unsurprisingly, the substance of the extreme right AI-generated images and videos varied according to the particular ideological form: while some imagery blended Christian iconography with Nazi symbolism, in others the focus was on visualising coded, memeified, or otherwise blatant Holocaust denialism and antisemitic conspiracy theories. As well as producing life-like imagery of recognisable ideologues, there were innumerable other visuals that drew on extreme right lore or iconography. In this sense, AI provided a means to produce eye-catching material, grounded in popular culture and meme culture alike, made for the attention economy.
Also common were memeified depictions of terrorists from the extreme right. Some of these positioned them in ‘cute’ scenes (e.g., the Christchurch terrorist riding a child’s pink bike) or deliberately edgy ones (e.g., the Christchurch terrorist ordering from a kebab shop). Others presented them dancing, or as DJs, or as spectators at a sporting event. Such visuals were typically overlaid with a figure indicating the number of victims murdered. The banality of this content not only serves to downplay and desensitise users to acts of extreme violence; it also makes it more difficult for tech companies to detect and potentially action the content from a moderation point of view. Indeed, some of the visuals we observed had received many thousands of engagements on platforms like TikTok.
Nihilistic extreme violence and other hybridised threats
Crude AI-generated memes were also widespread within violent nihilistic communities. In this context, however, the perpetrators of a broad variety of violent actions were glorified, including school and other mass shootings, right-wing and jihadist terrorist attacks, non-ideological knife attacks, child sexual abuse-related offences, war criminals, and other individuals associated with criminality, homicide, or other violent crimes. We also observed a number of the same trends that were found within the extreme right. AI tools were used to create dancing videos featuring perpetrators of mass casualty attacks, as well as ones showing them DJing with their victims dancing in the audience. In one video, for example, the perpetrator of the Southport attack was DJing in front of an audience of young girls.
AI tools were also used to edit authentic content, such as a Christmas-themed version of the Christchurch attack video in which the perpetrator appears as Santa and the victims as elves. Other examples included the ‘Minions Gore’ trend, in which terrorist or gore-related content is made to look like a scene from a Minions movie, and edited attack videos in which the victims appear as cats or other cute animals. As well as potentially desensitising users to violence, the risk here is that such content acts as a gateway to gore sites and other harmful online spaces as users seek out the original versions of the AI-edited videos they are viewing.
Incels and violent misogyny
Celebratory content centred on mass attackers was also prevalent within incel communities. Interestingly, these sometimes featured attackers unconnected to the incel community itself, such as the perpetrators of the 1999 Columbine Massacre and the 2022 Buffalo Shooting. There were also two uses of AI tools that were distinctive to incels and violent misogynists. The first was the use of nudify apps to create sexually explicit images and videos. Users also generated images of themselves or community ‘saints’ with attractive girlfriends. Described as aspirational ‘lifefuel’, such content reinforced community messaging that access to desirable romantic partners satiates a desire for violence.
The second was the use of AI image generators, photo-editing applications and AI filters to enhance photographs, identify perceived areas for improvement, generate idealised self-representations, and create face morphs, face swaps, or projections of a more ‘attractive’ self. Such ‘looksmaxxing’ extended beyond users’ own images to prominent incel figures, whose appearances were digitally altered to appear more conventionally attractive or hypermasculine, thus reinforcing appearance-based status ideals within incel culture.
Important distinctions
As this brief overview illustrates, there are important differences both between different categories of threat actor and within them – a finding that ran throughout the variety of AI tools examined in our report, from AI music and voice cloning to ideological roleplay and the translation and summation of extremist materials. An appreciation of these distinctions – in terms of intent, audience, legitimacy, symbolism, and mobilisation – is vital to understand the idiosyncrasies of different actors and nuances in how they perceive and use different AI tools.
In part two of this series, we explain that generic safeguards which lack this contextual validity are vulnerable to evasion and circumvention, and explore the implications of this insight for AI safety evaluations.
Stuart Macdonald is a Professor of Law at Swansea University, UK, and Coordinator of the VOX-Pol Network.
Dr. Allysa Czerwinsky is a Research Fellow in AI Trust and Security at the University of Manchester, UK
Image credit: Photo by Steve A Johnson on Unsplash