«Edited by Helene Bie Lilleør and Ulrik Lund-Sørensen Practical Action Publishing Ltd The Schumacher Centre, Bourton on Dunsmore, Rugby, ...»
The deliberate choice to undertake a broad evaluation of the RIPAT intervention has required the use of a number of methodological approaches. The evaluation has involved a large-scale survey to determine the impact of the project activities, a conventional evaluation study to assess the implementation process, and an observational study to investigate the context and the extent to which the project technologies have been adopted in the local communities. This chapter explains how the impact study made use of quantitative methods to measure the scale and extent of the impact of RIPAT, whereas both the implementation and context and adoption studies made use of qualitative methods to assess changes in agricultural practices among the target group and in their communities.
3.2 The combination of methodological approaches In social science research there is an ongoing debate about the pros and cons of various methodological approaches for assessing the results and value of social interventions.
Rather than adopting a single methodological approach, this study has analysed the diversity and complexity of the RIPAT intervention using both quantitative and qualitative methods.
Quantitative and qualitative methodologies imply different understandings of what it is important to know, as well as how it may be known. The combined efforts of quantitative and qualitative researchers can provide a more complete picture than would be obtained by any single methodological approach alone – similarly, the results from a range of methods may inspire new analytical questions across the different approaches.
The production of this book is an illustration of this, in that all the chapters have been subjected to several rounds of critique by all the researchers involved, each one commenting on the others’ preliminary interpretations and conclusions in the light of their own insights. The aim has been to reach an agreement on the presentation of results without settling for bland compromises or reducing the strength of each method.
In a quantitative study there is strict separation of the phases of the process:
preparation (including the specification of hypotheses), data collection, and analysis.
Once the research questions and survey questionnaires have been defined through careful consideration of the kinds of data needed, the collection of that data proceeds in a rigorous manner, so that the data are homogeneous throughout the process. Analysis of the data is a separate phase, in which different research questions are tested against the data in order to identify correlations and causal relationships. Some results may be anticipated, but others may be entirely unexpected.
A characteristic of qualitative methods, particularly those used in the anthropological, explorative tradition, is the integrated process of data collection and analysis.
Critical discussions of how to interpret data begin even before data collection itself, and continue throughout the study process, as they help to shape the form and type of data generated. The character of the data themselves therefore changes and expands in the course of the project. Thus, there is no strict separation of data generation and analysis, as these are dependent upon each other. Analytical perspectives and conclusions based on empirical material gradually grow out of this process, as an increasingly accurate and detailed account of the observations made and insights obtained.
The RIPAT study uses a mixture of methods in order to validate the findings by means of a methodological triangulation – that is, assessing a point from two or more perspectives in order to permit a comparison of the results, and therefore to ensure that those results are accurate (Denzin, 2006). In addition to applying different methods, the evaluation study has also used another type of triangulation in order to enhance its analysis and understanding of the complex set of factors and dynamics in play in the RIPAT project context, namely that of involving a number of different evaluators.
These evaluators have had different entry points to the study process and have come to it with different research profiles, in terms of both the nature of their knowledge of the subject and their expertise in applying specific approaches. In general terms, the quantitative approach was used to measure the scale and extent of the impact of RIPAT; the qualitative approach was used to assess the meaning and character of this impact from the farmers’ point of view, as well as to document and assess the way in which the project that brought about this impact was implemented.
26 Farmers’ ChoiCe
3.3 Data collection tools A number of tools were used to gather both quantitative and qualitative data for the evaluation study.
The quantitative data were based on surveys using structured questionnaires. In this type of research, trained interviewers ensure that each question is answered in quantitative terms or with a response from a predefined range of options. The respondents in the RIPAT evaluation were asked questions about themselves (age, gender, education, occupation, marital status, number of children), about their household (members, accommodation, ownership of assets, past crises such as illnesses or harvest failures), and about their agricultural practices (which crops are grown, their yields, types of technologies used, ownership of livestock). Respondents living in RIPAT villages were also asked a range of RIPAT-related questions.
For such a highly structured questionnaire to work well, intensive piloting of all questions is necessary to ensure that an exhaustive list of possible responses is prepared prior to the actual data collection. The aim is that all respondents should be asked exactly the same questions in the same way, and that their answers are selected from exactly the same list of options, thereby enabling consistent analysis of the responses.
The qualitative data were collected through various types of interviews (some with a limited set of questions, others of a more in-depth and open-ended nature); direct observations (using narrative descriptions); participant observations; analysis of written project documents; and focus group discussions. The anthropological researchers involved in evaluating the project emphasized the use of traditional ethnographic fieldwork methods, including participant observations of the daily lives of farming households, in order to gain a basic understanding of the living conditions, opinions, and priorities of local families. This participant observation approach was supplemented by informal and formal interviews and conversations. The ethnographic perspective enabled researchers to place RIPAT within the context of people’s lives, rather than treating people as ‘recipients’ of RIPAT. An important aspect of the explorative approach is that it not only gives answers to predefined questions and subjects of interest, but, more importantly, it also inspires the researchers to develop and ask new questions during the data collection process; this occurred during the data collection for this study.
In addition to open-ended interviews and direct observations, focus groups were used in the qualitative studies to a considerable extent, particularly by the researchers from the Danish Institute for International Studies (DIIS). In this qualitative evaluation method, small groups of people are brought together to discuss specific topics under the guidance of a moderator. The evaluators lead the focus group discussions and take the initiative in probing and asking additional questions as warranted by the situation.
The group process tends to elicit more information than individual interviews, because people express different views and engage in dialogue with one another. The moderator may facilitate the dialogue and explore the members’ motivations and feelings, allowing participants to bring up information or different opinions at any time.
3.4 The impact study The main aim of the quantitative impact assessment of RIPAT was to estimate the extent to which the RIPAT intervention has affected the level of food security and poverty 28 Farmers’ ChoiCe An interview was conducted in each household with the person responsible for most agricultural decisions, typically the head of the household. However, in RIPAT households, the person interviewed was always the RIPAT group member, irrespective of whether or not this person was the head of the household. Since households with female heads are overrepresented among RIPAT farmers, the same kind of overrepresentation among the comparison households was sought. In addition to the heads of households, representatives of all local village governments and representatives from all RIPAT 1 and RIPAT 3 farmers’ groups were also interviewed.
The problem of non-random selection The ideal methodology for assessing the impact of an intervention such as RIPAT would be to design the intervention from the outset as a randomized controlled trial, i.e. to imitate the randomized treatment models used in medical science when, for instance, the effect of a new drug is tested. This overcomes the problem of the ‘missing counterfactual’: what would have been the situation for the participating farmers had they not participated in the RIPAT project? Obviously, we will never know the answer to this. The challenge is therefore to find a good approximation for the missing counterfactual. A randomized controlled trial does this by collecting data both prior to and after project implementation from both treatment and control groups. The members of these two groups are randomly selected at the outset, which makes the groups directly comparable. Tracking the control group households over time then becomes the best approximation for the missing counterfactual for the treatment group. Using this methodology of comparing matched groups, the trial will be able to detect any impact of the intervention.
When the RIPAT 1 project originally started in 2006, it was thought of as a pilot project, where the implementation strategy was to be continuously adapted to local circumstances and demands from the participating farmers. There are no baseline data in existence from before the project activities started, and the project implementation was not randomized.
This is problematic from an impact assessment viewpoint, because the careful process of selection of project participants in terms of both villages and households explicitly breached the principle of randomization. The ‘skewed’ selection process (in evaluation terms) could in and of itself bias assessment results dramatically, if it were not taken into consideration when comparisons are made. As explained in Chapter 2, Section 2.3, the selection of participants in the RIPAT projects involved matching them against a long list of criteria. Thus RIPAT farmers are not a random sample of farmers from a given village, but rather a group of farmers with certain characteristics, ranging from the size of their farms to their willingness to learn and engage in new activities. Comparing RIPAT farmers with a random sample of farmers from non-RIPAT villages would therefore be problematic, because it is impossible to determine whether these comparison farmers would have been RIPAT farmers if they had had the option.
Likewise, RIPAT villages themselves were not chosen at random, but on the basis of their degree of poverty, suitability in terms of agricultural production, and the village government’s willingness to cooperate with RECODA. Even though Larsen and Lilleør made an attempt to select similar villages as comparison villages, they are not certain that this attempt was successful.
evaluation methods Addressing the problem of non-random selection Unless fully controlled and randomized, this problem of ‘skewed’ selection for the purpose of comparison is often a serious difficulty in any impact assessment. However, luckily it is possible to imitate to some extent a randomized controlled trial in the impact assessment of RIPAT 1. Working on the assumption that the selection of both villages and households was carried out in roughly the same way and with the same results in Arumeru and Karatu districts, Larsen and Lilleør can use the time lag between the start of RIPAT 1 and the start of RIPAT 3 to estimate the impact of RIPAT on outcome variables.
The assumption that the selection procedures were the same in the implementation of both RIPAT 1 and RIPAT 3 implies that households participating in the two projects differed from their respective sets of comparison households to the same degree at the outset. Because the two projects were implemented two years apart, it is possible to measure an impact on, for example, food security from the two-year lead that RIPAT 1 households have over RIPAT 3 households. This method was used to obtain the results presented in Chapter 5 by measuring the differences between RIPAT 1 households and RIPAT 3 households, using the differences between their comparison households to take any district-level variations into account, and thus effectively cancelling out both the village-level and the household-level selection bias mentioned above in the results. Box
5.3 in Chapter 5 presents the estimation method in more detail.
As Larsen and Lilleør note, the only major flaw in this method is that the evaluation results will be underestimated to the extent that any effect of RIPAT on the outcome variables had already taken place in RIPAT 3 by January 2011, two-and-a-half years after the project start. This will have been the case for more immediate outcomes that would have had time to appear within this two-and-a-half-year time span.
3.5 The implementation study The core aim of the implementation study was an analysis of the relevance, effectiveness, efficiency, and sustainability of the RIPAT intervention, focusing specifically on RIPAT 1 but also drawing on lessons learned and improvements implemented in RIPAT 2, 3 and 4.
The study team also looked at the institutional sustainability of the RIPAT intervention, assessing how successful RECODA has been in transferring the RIPAT concept to local government institutions, and at the prospects for a further strengthening of the institutional component of the RIPAT intervention. The study team was composed of researchers from the DIIS and the Food and Agriculture Organization of the United Nations (FAO).