Online Library

From politicians to podcast hosts, online platforms have systematically banned (“deplatformed”) influential users for breaking platform guidelines. Previous inquiries on the effectiveness of this intervention are inconclusive because 1) they consider only few deplatforming events; 2) they consider only overt engagement traces (e.g., likes and posts) but not passive engagement (e.g., views); 3) they do not consider all the potential places users impacted by the deplatforming event might migrate to. We address these limitations in a longitudinal, quasi-experimental study of 165 deplatforming events targeted at 101 influencers. We collect deplatforming events from Reddit posts and then manually curate the data, ensuring the correctness of a large dataset of deplatforming events. Then, we link these events to Google Trends and Wikipedia page views, platform-agnostic measures of online attention that capture the general public’s interest in specific influencers. Through a difference-in-differences approach, we find that deplatforming reduces online attention toward influencers. After 12 months, we estimate that online attention toward deplatformed influencers is reduced by -63% (95% CI [-75%,-46%]) on Google and by -43% (95% CI [-57%,-24%]) on Wikipedia. Further, as we study over a hundred deplatforming events, we can analyze in which cases deplatforming is more or less impactful, revealing nuances about the intervention. Notably, we find that both permanent and temporary deplatforming reduce online attention toward influencers; Overall, this work contributes to the ongoing effort to map the effectiveness of content moderation interventions, driving platform governance away from speculation.

The whitepaper has three parts. The first part provides some contextual background, describing the diverse range of online services utilised by terrorists and extremists and the process by which propaganda is disseminated online. The second part details industry responses. As well as referrals from users and law enforcement, it describes the use of AI for proactive detection and collaborative, cross-platform initiatives. The third part describes four issues for discussion: transparency; definitional clarity; the impact on those targeted; and, the use of online data for predictive purposes.

Over the past decade, a large number of extremist and hate groups have turned to internet platforms to inspire mass violence. Currently, there is little reliable evidence on how such campaigns radicalize targeted audiences. We provide systematic, large-scale, microevidence on the effect of Islamic State propaganda on social media. We use several machine learning algorithms to detect recruitment messages in online propaganda, identify their dissemination on Twitter, and quantify the reactions of exposed users. Analyzing content produced by the Islamic State between 2015 and 2016 shows that propaganda conveying the material, spiritual, and social benefits of joining ISIS increased online support for the group, while content displaying brutal violence decreased endorsement of ISIS across a wide range of videos. Only the group’s most extreme supporters reacted positively to violent propaganda.

Little is known about online behaviours of violent extremists generally or differences compared to non-violent extremists who share ideological beliefs. Even less is known about desistance from posting behaviour. A sample of 99 violent and non-violent right-wing extremists to compare their online patterns of desistance within a sub-forum of the largest white supremacy web-forum was analysed. A probabilistic model of desistance was tested to determine the validity of criteria set for users reaching posting desistance. Findings indicated that the criteria predicted “true” desistance, with 5% misidentification. Each consecutive month without posting in the sub-forum resulted in a 7.6% increase in odds of posting desistance. There were no significant differences in effects for violent versus non-violent users, though statistical power was low.

Mobilization is central to the emergence, survival and success of armed groups challenging the state, and has lately expanded to new arenas with the rise of social media. Using a new dataset of rebel group Twitter use, we examined the topics contained in rebel group social media communications to understand how different messaging strategies impact civilian engagement with rebel messages. Rather than benefiting solely from direct calls to action, we found that rebel groups also increased civilian engagement through indirect messages of self-promotion. While direct appeals received more engagement than indirect appeals, their effects were tempered by audience fatigue when relied on too heavily. We additionally found that including images further enhanced the impact of a mobilizing message. These findings expanded our understanding of rebel communications and mobilization, with important implications for combating the use of social media as a recruitment tool for violent extremism.