As COVID-19 arrived to disrupt the globe, social and behaviour change communication (SBCC) practitioners and their audiences, especially youth, increasingly embraced digital spaces and tools. Over the past three years, iMedia has been exploring digital SBCC and the ways that organisations have adopted and adapted their M&E approaches.
Platforms such as Instagram, YouTube, Facebook, TikTok, WhatsApp and other messaging tools (often in combination with traditional media) have expanded the SBCC ecosystem and enabled transmedia approaches that offer deeper, more holistic, and effective ways of engaging audiences.
While the move to digital SBCC began pre- pandemic, COVID-19 forced organisations to adapt rapidly. Old behaviour change theories can be difficult to apply to new digital settings. Additionally, there are gaps in understanding how to reach vulnerable populations virtually and how to design research and evaluation that capture online-offline behaviour change, especially when populations have intermittent access to digital devices and platforms.
Sharing learning at the European Evaluation Society Conference
In June 2022 iMedia attended the European Evaluation Society (EES) Conference in Copenhagen to share findings from our ongoing research into Monitoring and Evaluation (M&E) of Digital SBCC approaches. We were joined by Shujaaz, a social venture working with East African youth aged 15-24. This session and the insights shared therein built on what we learned during our 2021 EES session.
Shujaaz’ goal is to break down barriers that hold young people back and inspire them to transform and improve their own futures as well as those of their communities. Shujaaz uses a holistic approach to SBCC, combining analogue and digital media, to address various aspects of young people lives in order to achieve wholesome, holistic change. A core element of its success is a comic reaching 7.7 million East Africans and a collection of social media channels that reach an additional 3 million young people.
According to Anastasia Mirzoyants, Head of Knowledge and Learning at Shujaaz, the organisation’s research strategy routinely delivers rich, nuanced, actionable insights that inform the design and modification of its programmes. A formative methodology called GroundTruth uses a human centred development approach that includes “gamified experiences that allow young people to reveal rather than report the truth.” For example, Shujaaz conducts focus group discussions on WhatsApp and asks youth to share emojis describing their most recent relationship. The emojis serve as an entry point to probing topics and trigger points related to relationships.
By keeping close touch with its audience through multiple engagement strategies, Shujaaz is able to incorporate SBCC elements into its work that rapidly respond to current events and to shifts in how young people are feeling or thinking, and to adjust and alter topics, storylines, and messages that its characters are addressing.
This is just one example of many possible methods. As we shared in our session, in 2021, iMedia conducted research on how digital SBC organisations are approaching M&E, with an in-depth focus on four organisations (Shujaaz, Honey&Banana – Nigeria, C’est la Vie – West Africa, and Girl Effect India). We found that the organisations were using at least 25 different methods for achieving eight general M&E goals. Interestingly, in the digital SBCC sphere, most organisations tend to blend their monitoring and evaluation with research and learning (which we refer to as “MERL”).
Eight purposes of digital MERL from our 2021 research
- Monitoring reach – how many people are visiting a website or social media page? How many are starting a conversation with a chatbot?
- Learning about digital discovery pathways – understanding where the journey starts and how effective we are at driving users to key engagement points, such as calling a helpline, making an appointment, purchasing contraceptives etc.
- Monitoring engagement – here we look at quality of the engagement such as time spent on the site, pages viewed, likes or shares, number and type of comments. How well are we maintaining the person’s attention? Combined with other methods, learning what is the ideal quantity, frequency and combination of interventions for generating impact?
- Learning about the audience – using rich quantitative and qualitative methods throughout the whole engagement pathway, often face-to-face and focus group discussions (FGDs) to better understand the audience’s lives, attitudes, behaviours and needs. In live sessions, understanding and addressing youth’s views on issues like family planning and sexual reproductive health – these can be done remotely via WhatsApp, Zoom, Google meet, Facebook Live, Instagram Live, etc. Supplemented with systematic tracking of demographic data collected at different engagement points. Text analytics are increasingly being used as well (e.g., comments or SMS sentiment analysis).
- Learning about barriers and motivators to behaviour change – before or during interventions using FGDs, calls, surveys, or follow-ups with individuals to get feedback on their experiences with accessing contraception. Applying advanced tools such as AI-powered data analysis to find patterns in call centre recordings or SMS exchanges.
- Monitoring behaviour change intentions and follow-through – noting behaviour change intentions during call-centre conversations, follow up to check actions taken (e.g. visit a clinic), monitoring online sales record and purchases of FP products or redemption of e-vouchers. Shujaaz, for example, recently started using a new platform KatiKati, which allows them to (1) engage people in crises, (2) refer them to a last-mile partner, such as a sexual reproductive health counselling centre, (3) monitor if they engaged with that centre, and (4) get insights on their experience.
- Monitoring audience satisfaction – human or automated customer satisfaction check-ins through instant message or AI/manual analysis of audio or text records from automated calls or social media posts.
- Evaluating intervention impact – measuring knowledge and self-reported behaviours before / after interventions, one-off or longitudinal; can also use experimental methods (RCT or A/B testing). A variety of methods may be applied, combining online and offline data collection and analysis.
Challenges and risks with digital MERL approaches
While remote and online M&E will be core to future efforts, particularly for digital SBCC, in our session we also noted some of the key challenges that need to be overcome with digital MERL.
- Trust and credibility. If your audience doesn’t know you very well and a digital heavy or digital-only approach is used, it might be less successful, especially in times of crisis like during COVID. Shujaaz has extremely high levels of credibility due to how it engages with young people, and this makes it easier for them to connect with young people for data collection.
- Ground truthing or corroborating digital insights with face-to-face research to check whether online findings reflect the true sentiments of the audience. It’s also important to check whether the digital data tells the whole story and whether it’s being misinterpreted.
As Anastasia highlighted, it is critical to continue asking questions about the same issue in a variety of different ways, directly and indirectly. “Young people in a stage of transition like to experiment. They do not have an established identity; they like to play with different identities. Digital spaces help them experiment in this way. Based on our studies, young people can have up to 7-10 identities they are actively using online, because they can. We need to be aware of that when we are analysing the data.” Triangulation is critical, especially cross-verification using traditional methods.
- Danger of ‘extractive’ data collection. This is always something to watch, and can be even more of an issue with digital data capture. It’s important to ensure that the audience/respondents get something useful out of the data collection process and that they can see how their participation translates into better services or programmes or some kind of direct impact. Anastasia emphasized the danger of ‘pushing the issue,’ e.g., emphasizing something that is irrelevant to youth because of external pressure. She commented that young people are smart and will see through this. “It is really important to be creative and ever changing and to be transparent. Who is asking, why are they asking, what is going to be done with this information?”
- Inclusion and bias. Issues of ‘elite capture,’ and ‘convenience sampling’ can skew data. Elite capture is typically an issue with digital-only data collection as more elite sections of society tend to have greater access and use of digital devices and platforms. Convenience sampling in the case of digital happens when samples are geared towards populations that can access the Internet or digital devices and others are left out. This situation was exacerbated during the COVID pandemic because of travel restrictions. Anastasia noted that 60% of people are now online in Kenya where Shujaaz primarily works; however as they expand their work into Tanzania, where only 1/3 of the population is online, they will need to adapt their methods to the levels of connectivity.
The big questions going forward
Digital SBCC is playing a promising role in reinforcing accurate information and supporting norm shifts through social communication and influence. Digital platforms and engagements can link people to offline sources of information and services, as well as to supplementary online support an d communication channels. Some donors are already making significant investments in digital SBC with the hope that it can solve some of challenges faced in the FP world: for example, provider bias toward adolescents/youth, and bias against certain FP methods; myths and misconceptions around FP; and negative social and gender norms that inhibit individuals from seeking information and taking up services.
Many of the MERL approaches and methods outlined above could be considered by other sectors as they move their programmes and services online. Understanding how to learn from their digital strategies will be an essential part of the transition, and iMedia is distilling lessons to contribute to this wider purpose.
Big questions remain, however. Will this promise pan out? Will digital SBC be able to transform conversations and give greater access to information and positive influences to individuals, especially young people, who currently do not have this access? Will it prove adaptable and scalable, whilst protecting privacy and maintaining trust? This is the hope, and going forward, research and learning will need to keep pace with the growth of digital approaches.
This post is based on research funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.