By Jeremy Holland

I once edited a book on combining methods and data to get the best of all worlds in applied research. In the book, David Booth wrote a chapter entitled ‘strong fences make good neighbours’ which advocated for a clear demarcation between ‘acontextual’ survey-based data collection and ‘contextual’ in-depth interpretive research. Weakening that fence was asking for trouble, he argued. I saw his point then and still see it now. It centres on sacrificing explanatory depth in a quest for external validity. But I have also been frustrated by mixed-method studies in which the fence is actually more like a gulf; the two research methodologies don’t talk to each other and to an audience in a way which does justice to both. It can also weaken the trustworthiness of findings, enabling data mining in both camps.

This frustration was most likely what spurred a methodological innovation of sorts, outlined in the most recent CDI Practice Paper. A group of fellow researchers came up with a design for a qualitative research module that could generate integrated robust quantitative and qualitative data as part of a cluster-sampled impact assessment. The context was the From Protection to Production (PtoP) research project, coordinated by FAO, that looked at the impacts of cash transfer programmes across a range of programme and country contexts in Sub-Saharan Africa. Did cash transfers have economic and social impacts on beneficiary households and their communities that went beyond their primary safety net function?

Our brief was to design a qualitative research module to explore the dynamics of social and economic change triggered by the introduction of a small-but-predictable source of additional income in different programme contexts. We decided to push the envelope a little on the design of this module in terms of generating a mix of quantitative and qualitative data. We designed a ‘within-module’ integration of the generation and analysis of both quantitative and qualitative data. We did so through the standardised use of participatory and qualitative research methods across 36 carefully sampled sites in six countries. Participatory group analysis additionally enabled local people to analyse the influence of the cash transfer on their lives. Local people in Ghanaian villages, for instance, analysed economic and social changes amongst cash transfers beneficiaries whom they confirmed as the very poorest, locally described as ‘bottles’ (scratch them and you get nothing).

Of course, this combined methods angle is by no means new. Plenty of research uses sampling protocols to provide in-depth explanatory analysis of survey-based data. What was different about this research was to generate household income and expenditure data in a group setting and analyse this data in one sitting.

The research confirmed that cash transfers provided a valuable source of additional income for extremely vulnerable beneficiary households, whether elderly women looking after orphaned grandchildren or beneficiary households coping with health shocks. Through group-based quant-qual analysis, randomly sampled beneficiaries were able to reflect on just how significant this extra income stream was for their households. In some instances, beneficiaries were able to reduce their reliance on working as casual labourers or increase their investment in livelihood productivity, diversification or trading.

Critically, the methodology was designed to prevent our research teams ‘exaggerating the average’ experience of beneficiaries. One such case in a district of Kenya was more dramatic and headline-grabbing. It concerned a female motorcycle taxi business that emerged through the combined efforts of a small group of cash transfer beneficiaries. Each woman bought a ‘moto’ and hired a young driver - who gave a daily profit minimum to the owner, keeping the rest as earnings. The moto business was reportedly thriving. A senior researcher on our team promoted this as an illustrative case of the transformative impact of a cash transfer on livelihoods and household income. Dubbing this beneficiary group the “motorcycle mamas”, the researcher pushed to highlight this case as representative of transformational impact. The participatory household and expenditure data, mapped against beneficiary profiles, confirmed, however, that this beneficiary impact was highly atypical. Indeed, it even pointed to a probable ‘inclusion error’ in their cash transfer eligibility. In other words, while the case raised interesting questions about how a small injection of cash could prompt capital investment and livelihood diversification, it was not a likely scenario for the majority of typical beneficiaries, who were far more vulnerable and asset-poor.

Reflecting on this within-module approach we recognise that there are trade-offs involved. Cluster sampling can introduce bias. Spending a week in a community for qualitative insight is not as good as spending a month or a year. But these are trade-offs we can live with, particularly when you throw into the mix debates about the high costs and over-long feedback cycles in large-scale surveys or in-depth ethnographic research. More significantly, we saw the value of this integration in helping to separate the marginal gains from the motorcycle mamas.

Images: © Clare O’Brien; reproduced with permission

Partner(s): Itad, Institute of Development Studies