The Radicalisation Awareness Network’s (RAN) Communications & Narratives’ Working Group meeting on 13 and 14 February, 2017 in Brussels focused on how to measure the impact of an online counter or alternative narrative message; the following post is adapted from the ex-ante paper circulated prior to the meeting, which drew from the Institute for Strategic Dialogue’s (ISD) The Counter-narrative Monitoring and Evaluation Handbook (2016).
By Henry Tuck
Goals and Objectives of Counter-narrative Campaigns
The very first thing to consider in the planning phase of a counter-narrative campaign is what its overall goal is, and within that what objectives might be set for the campaign. Setting a clear goal will keep your campaign true to its original purpose. Setting a series of objectives, related to the overall goal of the campaign, will provide a series of quantifiable milestones against which to measure the success of a campaign.
Evaluation and impact measurement are important when campaigning, everybody agrees on that. To properly evaluate and measure impact, clear objectives, tangible (sub) goals, key performance indicators, clearly described output and outcome should be determined before starting the campaign. And while the campaign is running, there should be mid-term or interim reviews. Very often, however, the evaluation is not done properly or not at all. Instead, campaigners do the “famous” after-the-fact evaluation, trying to make sense of the data in hindsight.
This is a missed opportunity since a proper evaluation enables you to adapt your ongoing campaign if needed and to include lessons learned to improve your next campaign. How can we make it easier to evaluate and include impact measurement from the start?
- Objectives should be specific. They should relate not to abstract ambitions such as ‘engage young people’, but quantifiable measures of a desired effect, such as ‘have 1000 online conversations with young people through comments or direct messages.’
- They should also be measurable, with campaigners confident before the inception of the campaign that they will be able to discern, from available metrics and evaluation activities, whether or not they were achieved.
- Finally, they should be realistic. Considering a campaign’s time-span, budget, intended audience size and available resources, as well as the performance of previous campaigns, can help campaigners decide what objectives are realistic.
Metrics
The monitoring and evaluation (M&E) of online counter-narrative campaigns centres on the metrics drawn from the websites and social media platforms used during your campaign. There are hundreds of different metrics that can be observed and analysed in many different ways as part of the M&E process. Deciding which metrics matter, and how you want to analyse them, is an important part of M&E planning and should be established before a project starts. Ultimately, the key to understanding whether your campaign met its goals or not is to have a clear understanding of what success and failure would look like, and to decide which metrics are best placed to help you judge the eventual outcomes of the campaign.
Social media metrics fall into two broad categories; awareness and engagement.
- Awareness is the total number of people who view your campaign content (whether videos, ads, posts, websites or other digital content), and who those people are. It is important to consider awareness metrics which help you understand the characteristics of your audience: how, when and where they were exposed to your campaign, and which parts of the campaign reached them.
- Engagement is defined as the volume and types of interaction between audience members, campaigners or campaign materials. Engagements can include everything from likes and shares to email responses, and can be positive or negative. The number and nature of engagements can help campaigners understand their audience’s reactions to the campaign or its content. These metrics can be combined and analysed to build a comprehensive picture of a counter-narrative campaign’s performance, and help campaigners understand the impact they are having.
Impact is a measurable change in the audience’s knowledge, attitudes or behaviour that can be attributed to exposure to or engagement with counter-narrative content. Awareness and engagement metrics, when properly analysed, can be brought together to help evaluators understand the impact of their campaign. Additional indicators, such as evidence of offline action, or the qualitative evaluation of online comments, can contribute to the overall impact picture.
The measurement of impact – and ultimately the answer to the question of whether a campaign succeeded or failed – will be defined by the goals and objectives set at the very start of the project. In all counter-narrative campaign M&E, awareness and engagement are critical to understanding impact. However, the relative importance of each metric, and the particular metrics evaluators focus on, is dependent on the type of counter-narrative campaign. For example, a downstream counter-narrative campaign – one designed for those individuals who are at higher risk of joining extremist groups – might place a greater emphasis on metrics that measure sustained engagement. An upstream counter narrative campaign – focused on a broader audience – might by contrast concentrate on achieving greater awareness. If a campaign is intended for a particular audience, it would be appropriate to consider not only awareness in general, but the awareness of the specific audience groups which are the focus of your campaign.
This Blog post was authored by the ISD’s Research and Policy Manager, Henry Tuck. Follow him on Twitter: @HenryTuck.