Online Library

The ever-increasing amount of user-generated content online has led, in recent years, to an expansion in research and investment in automated content analysis tools. Scrutiny of automated content analysis has accelerated during the COVID-19 pandemic, as social networking services have placed a greater reliance on these tools due to concerns about health risks to their moderation staff from in-person work. At the same time, there are important policy debates around the world about how to improve content moderation while protecting free expression and privacy. In order to advance these debates, we need to understand the potential role of automated content analysis tools.

This paper explains the capabilities and limitations of tools for analyzing online multimedia content and highlights the potential risks of using these tools at scale without accounting for their limitations. It focuses on two main categories of tools: matching models and computer prediction models. Matching models include cryptographic and perceptual hashing, which compare user-generated content with existing and known content. Predictive models (including computer vision and computer audition) are machine learning techniques that aim to identify characteristics of new or previously unknown content.

The substantive focus of this article is a small collection of image-based case studies of significant criminal acts of interpersonal cruelty that are now in the public domain. In all instances those engaged in criminal acts of violence record and document aspects of their own behavior. The case studies range from military personnel and terrorists to examples from popular culture. Self-created images have the potential to serve as evidence of criminal behavior. For the viewing public recorded images are attributed with a “commonsense” evidential documentary potential. The article looks at the documentary method and the rationale behind recorded images serving evidential purposes.

The menace presented by Islamic State’s (IS) self-proclaimed ‘caliphate’ is uniquely challenging on a
number of levels. Tactically, its military operations demand lateral thinking, since the group exists as a
nebulous, swarming network constantly seeking to expand its reach. Strategically, too, it is pioneering
new insurgent methods, establishing numerous, complex administrative and institutional foundations
in Iraq and Syria that are already deeply rooted and sure to prolong the war for years to come, as the
middle ground between civilian and soldier is systematically destroyed. It is in psychological terms,
though, that IS has truly transformed the state of play. Its vast propaganda operation is unrivalled,
involving devoted media teams from West Africa to Afghanistan who work relentlessly, day and night,
in the production and dissemination of the ‘caliphate’ brand. So far, most of our attempts to
meaningfully mitigate IS’s ability to globally engage have been left floundering.
Numerically speaking, it is an uphill struggle. Though there are some commendable efforts being
undertaken by counter violent extremism practitioners and civil society organisations, they are dwarfed
in size by IS’s media behemoth, which produces on average 38 individual batches of propaganda each
day – videos, photo essays, articles and audio programmes. Apart from practicalities, the counter effort
is, from the offset, structurally impaired from success. Indeed, the cult of the counter-narrative has left
coalition partners working from within a reactive paradigm, something that means it is perpetually on
the back foot when it comes to presenting an alternative to what IS offers.
The difficulties we face in the information war on IS are not something of which we can opt out. Hence,
we need to recognise our weaknesses and circumnavigate the obstacles we face. Arguably the most
damaging of those weaknesses has been a persistent tendency to misunderstand just what it is that IS
is doing – myriad questions have been asked, and most left unanswered.
In the Quilliam Foundation’s latest research into IS propaganda, Senior Researcher Charlie Winter
presents us with a truly ground-breaking window into the mind of the propagandist, demystifying the
media war more than ever before. Between 17 July and 15 August 2015, the Islamic month of Shawwal,
Charlie compiled an exhaustive archive of IS propaganda, creating not just a snapshot of its output, but
a comprehensive, 30-day view of it.
Over the course of the data collection period, he recorded 1146 separate propaganda “events”,
discrete batches of data that were disseminated with a view to bolstering the IS world view, be that
through graphic violence or millenarian scenes of vividly lit fairgrounds. Each event was recorded
according to 7 variables and then grouped by narrative and subcategory, enabling detailed analysis. By
postponing any assessment until the data had been collected in full, Charlie was able to circumvent IS’s
tactical saturation of the Internet and consider its messaging in an aggregated, considered manner. In
so doing, as important trends, iniquities and anomalies that are otherwise impossible to discern become
strikingly apparent, he has presented us with an important tactical and strategic insight into the virtual
‘caliphate’.
When it comes to IS propaganda, it is imperative that we understand it in as granular and nuanced a
manner as possible. Using data to test the hypothesis of the July 2015 report ‘The Virtual ‘Caliphate’:
Understanding Islamic State’s Propaganda Strategy’, Charlie has illuminated the bare bones of the IS
brand. It is high time we recognised that there is no elixir that can deliver us from IS’ information
supremacy, no catch-all counter-narrative to undercut its carefully cultivated and choreographed
image. In this absence, we must instead seek to enrich our understanding. The IS ‘caliphate’ is marketing
itself on an industrial scale. If we are to destroy its brand, we must first be able to fathom its depths.

When toxic online communities on mainstream platforms face moderation measures, such as bans, they may migrate to other platforms with laxer policies or set up their own dedicated website. Previous work suggests that, within mainstream platforms, community-level moderation is effective in mitigating the harm caused by the moderated communities. It is, however, unclear whether these results also hold when considering the broader Web ecosystem. Do toxic communities continue to grow in terms of user base and activity on their new platforms? Do their members become more toxic and ideologically radicalized? In this paper, we report the results of a large-scale observational study of how problematic online communities progress following community-level moderation measures. We analyze data from r/The_Donald} and r/Incels, two communities that were banned from Reddit and subsequently migrated to their own standalone websites. Our results suggest that, in both cases, moderation measures significantly decreased posting activity on the new platform, reducing the number of posts, active users, and newcomers. In spite of that, users in one of the studied communities (r/The_Donald) showed increases in signals associated with toxicity and radicalization, which justifies concerns that the reduction in activity may come at the expense of a more toxic and radical community. Overall, our results paint a nuanced portrait of the consequences of community-level moderation and can inform their design and deployment.

VOX-Pol Blog post.

VOX-Pol Blog post.