Recommendations for Policy and Practice
By Allysa Czerwinsky (University of Manchester) & Stuart Macdonald (Swansea University)
As AI use is increasingly embedded within contemporary terrorism and violent extremism, there remains a pressing need to strengthen AI safety initiatives and regulatory responses to better align with how models and tools are used in practice across threat actor categories. Our new report, Artificial Intelligence, Violent Extremism and Nihilistic Violence, illustrates that the same capabilities of AI models and tools have different meanings, uses, and risks across Salafi-jihadist, extreme right wing, incel and violent misogynist, and nihilistic extremist and hybridised threat actors. Crucially, these nuances are often absent in existing approaches to AI safety evaluations, particularly those that adopt generic guardrails or generalised assessments of capability.
Our findings here align with existing research demonstrating that generic safety guardrails are often vulnerable to evasion, circumvention, and deliberate misuse. Moreover, AI-generated extremist content operates below existing thresholds for content moderation and regulatory response, many of which centre obvious names, slogans, and prompts or focus on a narrow set of harms, including weapons, attack planning, direct and clear incitement, or explicit propaganda. A focus on generic guardrails against explicit harms, or on capability-centric approaches to moderation and regulation, misses important nuances in how extremists use AI tools in qualitatively distinct ways.
This blog is the second of a two-part series. Here, we offer concrete recommendations for preventing and responding to the use of AI tools by violent extremist and nihilistic threat actors. These recommendations are aimed at specific stakeholder groups across sectors, including convenors within civil society and industry; AI model and tool developers; content-hosting services; and governments.
For Convenors
As extremist use of AI remains an ecosystem-wide problem, collaboration within and across sectors is essential for facilitating a robust response. Establishing cross-industry signals-sharing channels that bring together AI labs and social media platforms ensures routine collaboration and information-sharing, allowing for more robust understandings and responses to AI use across the threat landscape.
Several existing channels show promise for facilitating information-sharing on AI-driven and AI-simulated threats. GIFCT established its Signals Community of Practice in early 2026, bringing together member companies and platforms within its Membership Advisory Program to develop a shared understanding of threats while protecting legal and ethical safeguards. Moreover, the Frontier Model Forum convenes developers of leading AI models to identify best practices for AI safety and facilitate information sharing. Though not specifically attuned to terrorism or extremism, a collaborative channel between the world’s leading AI labs could meaningfully contribute to strengthening AI safety amidst emerging terrorist and violent extremist threats.
Additionally, building mechanisms that support multi-stakeholder contributions to signals sharing – particularly by bridging AI developers, content hosting services, external researchers, and violence prevention practitioners – ensures that novel insights from researchers and practitioners inform the policies and practices of developers and platforms. Creating opportunities for structured pipelines to share validated insights on how threat actors use services and where harm is concentrated, especially within niche platforms and websites unlikely to participate in formal signals-sharing channels, allow for robust evidence to shape improvements to safety, alongside facilitating responsiveness through rapid information sharing. Such evidence pipelines already exist in adjacent work on online harms and disinformation, including the CETaS AI Disinformation Incident Repository at the Alan Turing Institute, offering models for signals-sharing that can be adapted to the terrorism and violent extremism field.
For AI Model and Tool Developers
Our findings illustrate the importance of accounting for nuance and context within specific threat actor types when developing evaluations of AI safety. Generic categories of ‘violent extremist content’ that lack context-specific considerations of how language, aesthetics, symbols, and use differ between threat actor types flatten distinctions within and across the threat landscape, impeding effective response efforts from developers to address and prevent the exploitation of models and tools. As such, developers should implement evidence-based model testing and safety evaluations that consider nuanced terrorist and violent extremist risks, as well as share information about testing through transparency reports and model system cards for specific safety evaluations.
Relatedly, participating in cross-industry signals-sharing channels can ensure developers better understand the harms created by AI-generated content and improve safety guardrails over time. Incorporating evidence-based insights from researchers widens AI safety evaluations beyond the knowledge of internal teams, contributing to the development of stronger evaluations and improved safety floors across AI services. For instance, our work demonstrates that effective guardrails against misuse – including limiting model capabilities and impeding users’ abilities to bypass or evade safeguards – had a discernible effect on users’ experiences, insights that can help shape prevention efforts.
For Content-Hosting Services
All of the content analysed within our work was posted to or shared within public social media platforms and websites, necessitating more robust solutions to identifying and responding to novel AI-generated material. Monitoring for new patterns in policy-violating content produced using generative AI models and tools, and sharing these insights via signals-sharing channels, can help ensure that effective content moderation policies and practices better address the intent and impact of terrorist and extremist content across platforms. Moreover, effective responses must account for how threat actors’ use of AI evades model guardrails, whilst synthetic outputs operate below existing safety thresholds. Without sharing signals of emerging extremist content and misuse, developers are left to create safeguards on user inputs alone, with limited visibility into how AI-generated content is interpreted and circulated across platforms.
Algorithmic recommendation systems play a vital role in amplifying the reach and visibility of violent extremist content, particularly for users who have searched or viewed similar content in the past. Moreover, algorithmic amplification of AI-generated extremist content can deepen polarisation in moments of crisis, making it a useful tool for widening reach. This was particularly evident in the following the 2024 Southport stabbing and the 2026 Bondi Beach attack, where AI disinformation circulating in the immediate aftermath strengthened anti-immigrant sentiment and fuelled offline mobilisation.
As such, preventing the algorithmic amplification of terrorist and violent extremist content – as well as related harmful content falling short of response thresholds – using proportionate, rights-respecting measures is critical. Implementing alternative enforcement measures for content operating below terrorist and violent extremist response thresholds (such as down-ranking) may help prevent harmful content spreading through recommendation systems. Additionally, participating in efforts like the Christchurch Call Initiative on Algorithmic Outcomes can help platforms assess the degree to which recommender systems are currently promoting harmful content and develop strategies to mitigate amplification risks.
For Governments
Our research demonstrates a pressing need for robust AI safety regulations related to terrorist and violent extremist use of AI, and governments are uniquely positioned to support and promote cross-sector collaborations to better address the challenges of AI-driven and AI-simulated extremist content illustrated in our report. Funding independent, expert-designed AI safety evaluations on terrorism and violent extremism, alongside creating incentives for developers to adopt them and report performance outcomes consistently and transparently, can help strengthen existing responses to terrorist and violent extremist use of AI.
Methodological contributions to AI safety
In using a systematic, comparative, actor-specific approach to track AI-generated content in extremist ecosystems, our research highlights pressing issues with current approaches to AI safety evaluations focusing on broad capabilities and threats. Safeguards and risk assessments need actor-specific evidence to better understand how AI is being used in terrorist and violent extremist networks, often in ways not captured by generic misuse categories and generalised assumptions about capability. Without this context, safeguards remain vulnerable to evasion and circumvention. Moreover, this work is a collaborative effort bringing together researchers with extensive expertise across each of the four threat actor categories, an approach equally beneficial to future research on AI, extremism, and safety.
Dr. Allysa Czerwinsky is a Research Fellow in AI Trust and Security at the University of Manchester, UK.
Stuart Macdonald is a Professor of Law at Swansea University, UK, and Coordinator of the VOX-Pol Network.