The use of face-to-face interviews in mixed mode design: The problem of the achieved sample
The use of face-to-face interviews in mixed mode design: The problem of the achieved sample

Summary/Abstract: Although the face-to-face mode has been losing its appeal due to declining response rates and rising costs, it nevertheless remains the most universal and effective survey mode. Therefore, it could be expected that its use in a mixed mode survey would increase the effectiveness of this design by boosting the response rate and improving the sample composition, not only in comparison with the most commonly used mix of self-administered modes, but also in comparison with unimode face-to-face surveys. The inclusion of the face-to-face mode, which is not only effective but also completely different from self-administered modes, should encourage additional categories of respondents to participate in surveys and, in this way, facilitate the achievement of objectives set for the mixed mode design. Two experiments carried out in connection with the European Social Survey (ESS) were aimed at verifying whether the face-to-face interview used in the mixed mode design did indeed produce the aforementioned benefits, compared to the use of face-to-face interviewing in unimode surveys. The idea was to boost the response rate and improve the sample composition. Both experiments were based on the sequential design with elements of the concurrent design. In the former, the face-to-face mode was used in the initial phase, whereas in the latter it was used in the follow-up phase. This mode was combined with self-administered modes, including the most effective of them, i.e. the mail mode. In the experiment aimed at verifying the effectiveness of mixed mode design with face-to-face used first, the ESS main study rounds 6 and 7 were treated as an initial phase. In the follow-up, the interviewers suggested that refusers and non-contacted persons/their proxies take part in the survey, offering a choice between the mail and the EMS (e-mail) mode. The face-to-face phase achieved a high response rate of 74.9% in ESS 6 and 65.8% in ESS 7. The use of self-administered modes in the follow-up increased the response rate by only a fraction of a percentage point in both experiments. Relatively few nonrespondents/proxies agreed to participate in the follow-up, and the completed questionnaires were finally sent back by relatively few of them. In our second experiment, face-to-face interviewing was used in the follow-up phase, with self-administered modes used first. The self-administered modes used included regular mail, web and EMS modes. Despite using a rigorous research design in both phases, the obtained response rate was lower by nearly 10 percentage points in comparison with that achieved in the ESS 7 main study carried out in parallel. Although face-to-face proved to be the most effective mode in the experiment, the response rate achieved when using it was rather low, and exceeded that achieved in the initial self-administered phase only by a few percentage points. Analysis of the sample composition obtained in the mixed mode design using face-to-face in the initial phase and self-administered modes in the follow-up (Experiment I) seemed pointless. A negligible increase in response rate (a fraction of a percentage point) compared with the initial phase cannot significantly affect the total mixed mode sample distributions. The analysis of sample composition in the second experiment, based on the reversed sequence (Experiment II, i.e. self-administered modes first), covered six socio-demographics: respondents’ sex, age, size of domicile (no. of inhabitants), level of education, number of people in the household and the main activity. Their distributions were compared with those achieved in the ESS 7 main study and with the statistical data for the total population. The mixed mode sample did not differ from the ESS 7 main study sample in terms of five out of six characteristics included in the analysis. A difference was only found in the case of respondents’ main activity and is related to the much higher participation of permanently sick or disabled individuals in the mixed mode sample. This means that using a mix of these modes (self-administered and face-to-face, in this sequence) does not improve the sample composition in terms of socio-demographics. In the case of sex and size of domicile (no. of inhabitants), it is not reasonable to expect an improvement, as the distributions of these characteristics obtained in the ESS 7 are very close to those for the total population. However, the distributions of age and level of education differ more from the population, while the distribution of the number of people in the household differs very significantly. Therefore, both samples differ from the total population in a similar way. On the other hand, the use of such a design, despite a lower response rate, did not lead to deterioration of the sample composition. The similarity of the total mixed mode sample and the sample achieved in the ESS 7 main study results from using the face-to-face mode in the follow-up. For all six analysed characteristics, the sample composition obtained in the initial self-administered phase differs more from the sample achieved in the ESS 7 main study than does the total mixed mode sample from the ESS 7 sample. The greatest, statistically significant differences, were found in the size of domicile (no. of inhabitants) and level of education, whereas smaller differences were found in the sample composition in terms of age. This suggests that if self-administered modes are used in the initial phase, then the use of the face-to-face mode in the follow-up is conducive to selection effects and improvements in the composition of the total mixed mode sample. The findings from both experiments may undermine the sense of using a mixed mode design with the face-to-face mode used both in the initial phase, and in the follow-up. Application of the face-to-face first design only generates additional costs related to the implementation of the self-administered phase, with no clear outcomes such as an increase in the response rate. In turn, the application of self-administered modes first and face-to-face in the follow-up does not bring the effects expected from the mixed mode design, not only in terms of response rate, but also in terms of sample composition. Moreover, it fails to achieve another postulate behind the mixed mode design, i.e. reduction of survey costs. The cost of a single case in this experiment was nearly identical to that in the ESS 7, completed entirely face-to-face. It was equivalent to approx. EUR 40. However, the results of both experiments could have been affected by the use of a rigorous research design in the face-to-face phase. Both in the ESS main study (initial phase in Experiment I), and in the face-to-face follow-up in Experiment II, the interviews were carried out by experienced, highly motivated interviewers, each sampled person received an unconditional gift, and the fieldwork lasted several months. This design also drove up the costs of both surveys. However, a rigorous research design is applied rarely, even in academic surveys. For this reason, the final section of the article discusses potential consequences for the results of a mixed mode survey if a standard research design were to be used in the face-to-face phase instead of a rigorous design. Other potential reasons behind the results obtained in both experiments are also discussed.

• Details
• Contents