By Linda Raftree, Nicola Harford and Anastasia Mirzoyants
As COVID-19 arrived to disrupt the globe, Social and Behaviour Change Communication (SBCC) practitioners and their audiences, especially youth, increasingly embraced digital spaces and tools. Over the past three years, iMedia has been exploring digital SBCC and the ways that organisations have adopted and adapted their M&E approaches.
Platforms such as Instagram, YouTube, Facebook, TikTok, WhatsApp and other messaging tools (often in combination with traditional media) have expanded the SBCC ecosystem and enabled transmedia approaches that offer deeper, more holistic, and effective ways of engaging audiences.
While the move to digital SBCC began pre-pandemic, COVID-19 motivated many organisations to adapt rapidly. However, multiple challenges arose: for example, existing behaviour change theories can be difficult to apply to new digital settings; often the most vulnerable populations have at best intermittent access to digital devices and platforms; and there is work to be done on designing research and evaluation that captures online-offline behaviour change.
Sharing learning at the European Evaluation Society Conference
In June 2022 iMedia attended the European Evaluation Society (EES) Conference in Copenhagen to share findings from our ongoing research into Monitoring, Evaluation, Research and Learning (MERL) of Digital SBCC approaches. We were joined by Shujaaz Inc, a network of social ventures working with East African youth aged 15-24. This session and the insights shared therein built on what we learned during our 2021 EES session.
Shujaaz Inc’s goal is to break down barriers that hold young people back and inspire them to transform and improve their own futures as well as those of their communities. Shujaaz Inc uses a holistic approach to SBCC, combining analogue and digital media, to address a range of issues in young people lives in order to achieve wholesome, holistic change. A core element of its success is a trusted brand and relevant, authentic, relatable stories and role models delivered to 9.5 million East Africans through analogue (comic book and events) and digital channels (social media, WhatsApp, and its own platform, MESH).
Shujaaz in-house research tem has developed an innovative research strategy, which routinely delivers rich, nuanced, actionable insights to inform the content and delivery of SBCC campaigns. A formative mixed method research methodology called GroundTruth uses human centred design and participatory research approaches and includes gamified experiences that allow young people to reveal rather than report the truth about their knowledge, attitudes, norms and behaviours related to a particular issue or topic. One example of a GroundTruth activity, which works very well in digital research (such as WhatsApp focus group discussions (FGDs)) requires participants to share emojis describing their most recent relationship. The emojis serve as an entry point to probing topics and trigger points related to relationships.
By keeping close touch with its audience through multiple Player-First (i.e., user-centred) engagement strategies, Shujaaz can incorporate SBCC elements into all its interactions with the audience and thus is able to rapidly respond to shifts in how young East Africans are thinking, feeling and acting, caused by changes in their context – such as political and economic events. This agility and connectedness contribute to Shujaaz ability to adjust and alter the topics, storylines and messages that its characters are addressing.
This is just one example of many possible approaches to digitally enabled research and M&E. As we shared in our session, in 2021 iMedia conducted research on how digital SBC organisations are approaching M&E, with an in-depth focus on four organisations (Shujaaz Inc, Honey&Banana in Nigeria, C’est la Vie in West Africa, and Girl Effect India). We found that the organisations were using at least 25 different methods for achieving eight general learning goals. Interestingly, in the digital SBCC sphere, most organisations tend to blend their monitoring and evaluation with research and learning (which we refer to as “MERL”).
Eight purposes of digital MERL from our 2021 research
- Monitoring reach – how many people are visiting a website or social media page? How many are starting a conversation with a chatbot?
- Learning about digital discovery pathways – understanding where the journey starts and how effective we are at driving users to key engagement points, such as calling a helpline, making an appointment, purchasing contraceptives etc.
- Monitoring engagement – here we look at quality of the engagement such as time spent on the site, pages and videos viewed, likes or shares, number and type of comments. How well are we maintaining the person’s attention? Combined with other methods, this involves learning about the ideal quantity, frequency and combination of interventions for generating impact.
- Learning about the audience – using rich quantitative and qualitative methods throughout the whole engagement pathway, such as FGDs, to better understand the audience’s lives, attitudes, behaviours and needs. Live sessions can generate understanding and help youth share their views on issues like family planning (FP) and sexual reproductive health (SRH). These sessions can be done remotely via WhatsApp, Zoom, Google meet, Facebook Live, Instagram Live, etc. They can be supplemented with systematic tracking of demographic data collected at different engagement points. Text analytics and sentiment analysis are increasingly being used as well (e.g., sentiments expressed in comments or automated sentiment indicators offered by various social media platforms).
- Learning about barriers and motivators to behaviour change – before or during interventions using FGDs, calls, surveys, or follow-ups with individuals are used to get feedback on their experiences with accessing contraception. Advanced tools such as AI-powered data analysis are also applied to find patterns in call centre recordings or SMS exchanges.
- Monitoring behaviour change intentions and follow-through – noting behaviour change intentions during call-centre conversations, following up to check actions taken (e.g. visit a clinic), monitoring the online sales record and purchases of FP products or redemption of e-vouchers. Shujaaz Inc, for example, recently started using a new platform KatiKati, which allows them to (1) engage people in [e.g. SRH] crises, (2) refer them to a last-mile partner, such as a counselling centre, (3) monitor if they engaged with that centre, and (4) get insights on their experience.
- Monitoring audience satisfaction – instant messaging or AI/manual analysis of audio or text records from calls or social media posts enable human or automated customer satisfaction check-ins.
- Evaluating intervention impact – measuring knowledge and self-reported behaviours before / after interventions, one-off or longitudinal; this can also use experimental methods (RCT or A/B testing). A variety of methods may be applied, combining online and offline data collection and analysis.
Challenges and risks with digital MERL approaches
While remote and online M&E will be core to future efforts, particularly for digital SBCC, in our session we also noted some of the key challenges that need to be overcome with digital MERL.
- Trust and credibility. If your audience doesn’t know you very well, a digital-heavy or digital-only approach might be less successful, especially in times of crisis like during COVID. Shujaaz Inc has developed trust and credibility among its audiences through more than a decade of consistent engagement with young people, grounded in the Player-First commitment and focused on youth-defined priorities. This helps the Shujaaz Inc teams to engage young people in conversations and activities even around sensitive and taboo subjects, which they would not discuss otherwise. For a digital-only newcomer, collecting in-depth data and insights might be a challenge.
- Ground truthing or corroborating digital insights with face-to-face research to check whether online findings reflect the true sentiments of the audience. It’s also important to check whether the digital data tell the whole story and whether they are being interpreted accurately.
As highlighted above, it is critical to continue asking questions about the same issue in a variety of different ways, directly and indirectly. According to Shujaaz Inc research, “Young people in a stage of transition like to experiment. They do not have an established identity; they like to play with different identities. Digital spaces help them experiment in this way. Based on our studies, young people may have up to 10 identities they are actively using online, because they can. We need to be aware of that when we are analysing the data.” Triangulation is critical, especially cross-verification using traditional methods.
- Danger of ‘extractive’ data collection. This is always something to watch, which can be even more of an issue with digital data capture. It’s important to ensure that the audience/respondents get something useful out of the data collection process and that they can see how their participation translates into better services or programmes or some kind of direct impact. Shujaaz Inc always warns its partners about the danger of ‘pushing the issue,’ e.g., emphasizing something that is irrelevant to youth because of external pressure to address that issue in a particular ‘approved’ or ‘established’ manner. Young people are smart and will see through this. Moreover, young people’s rapid transition from childhood to adulthood leads them to change their perspectives quickly and unexpectedly. “It is really important to be creative and ever changing and to be transparent. Who is asking, why are they asking, what is going to be done with this information?” – these are some of the questions young people ask of all adults interacting with them, including researchers.
- Inclusion and bias. Issues of ‘elite capture,’ and ‘convenience sampling’ can skew data. Elite capture is typically an issue with digital-only data collection as more elite sections of society tend to have greater access and use of digital devices and platforms. Convenience sampling in the case of digital happens when samples are geared towards populations that can access the Internet or digital devices and others are left out. This situation was exacerbated during the COVID pandemic because of travel restrictions. According to Shujaaz Inc research, 60% of people are now online in Kenya and digital channels are a good way to reach youth, even some of the most vulnerable. However in Tanzania, where Shujaaz has been active since 2014, only 1/3 of the population is online, which means Shujaaz has to adapt its methods to the levels of connectivity.
The big questions going forward
Digital SBCC is playing a promising role in reinforcing accurate information and supporting norm shifts through social communication and influence. Digital platforms and engagements can link people to offline sources of information and services, as well as to supplementary online support and communication channels. Some donors are already making significant investments in digital SBC with the hope that it can solve some of the challenges faced in the FP world: for example, tackling provider bias toward adolescents/youth, and bias against certain FP methods; myths and misconceptions around FP; and negative social and gender norms that inhibit individuals from seeking information and taking up services.
Many of the MERL approaches and methods outlined above could be considered by other sectors as they move their programmes and services online. Understanding how to learn from their digital strategies will be an essential part of the transition, and iMedia is distilling lessons to contribute to this wider purpose.
Big questions remain, however. Will this promise pan out? Will digital SBC be able to transform conversations and give greater access to information? Will it become a positive influence on individuals, especially young people, who currently do not have such access? Will it prove adaptable and scalable, whilst protecting privacy and maintaining trust? This is the hope, and going forward, research and learning will need to keep pace with the growth of digital approaches.
This post is based on research funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.