Do repeated contact attempts with hard-to-get respondents improve the achieved sample? The example of the European Social Survey 7 in Poland
Do repeated contact attempts with hard-to-get respondents improve the achieved sample? The example of the European Social Survey 7 in Poland

Summary/Abstract: The declining response rates in surveys observed in the last twenty five years or so, undermine public trust in survey results, reduce the precision of estimates and, above all, increase the risk of non-response bias in the results. One way to reduce the non-response rate is to make repeated contact attempts with hard-to-get respondents, i.e. hard-to-reach and reluctant individuals. While the costs involved in repeated contact attempts are relatively high, the effectiveness of these efforts, especially in face-to-face surveys, is relatively underresearched and the existing findings are inconsistent. Meanwhile, the significance of the problem is growing since the share of hard-to-get respondents has been increasing. This phenomenon is apparent in the results of the European Social Survey (ESS) in Poland. While in 2002 (ESS round 1) the average number of contact attempts needed to conduct one interview was 1.71, this figure rose to 2.28 in 2014/15 (ESS 7). Over that time, the share of the hardest-to-get respondents (i.e. successfully interviewed after 5 or more contact attempts) more than quadrupled: from 2.1% to 9.6%. This paper addresses two questions: 1) Do repeated contact attempts with hard-to-get respondents in face-to-face surveys improve the response rate significantly? 2) Do they improve the sample composition? The data used to answer these questions is taken from the European Social Survey round 7 (2014/15) in Poland. That survey, conducted in accordance with a rigorous research design (highly motivated interviewers, two advance letters, incentives for respondents, repeated contact attempts with hard-to-get respondents etc.) achieved a response rate of 65.8%. Our analysis compares a sample comprising only easy-to-get respondents (who were successfully interviewed during the first or second contact attempt) against the total ESS 7 sample (also covering the respondents who needed three or more contacts to be interviewed). The composition of those samples is then compared against the general population. Regarding respondents who were successfully interviewed during the first or second contact attempt as easy-to-get is in line with the practice of research institutes, at least in Poland. The response rates obtained by those institutes in academic surveys indicate that interviewers do not make subsequent attempts to contact hard-to-get respondents. The data about accessibility and refusal conversions, measured by the number of contact attempts needed to conduct a survey interview, is taken from Contact Forms (CF), routinely used in the ESS since its first round. In the CF, the interviewer must enter detailed information about each attempt to contact a specific sampled person. The findings from our analysis indicate that repeated contact attempts do improve the response rate by more than 20 percentage points. The initial response rate, calculated for easy-to-get respondents, was merely 45.3%. The final response rate was mostly increased thanks to contact attempts with hard-to-reach respondents whereas refusal conversions played a much smaller role. This finding is in line with the results obtained in other studies. When comparing easy-to-get respondents against the total ESS 7 sample, we took seven socio-demographic variables into account: sex, age, size of domicile (number of inhabitants), level of education, number of people in the household, main activity and monthly per capita income in the household (deciles). In the case of the first five demographics, general population data is available as well. The composition of both analysed samples was compared against the general population for those demographics. The analysis included a dissimilarity index, which shows the percentage of people in the contingency table that should be classified into another cell to achieve an identical distribution in groups under comparison. When comparing the easy-to-get sample against the total ESS 7 sample, we also used the Z test, showing the significance of differences between categories for each variable. Our analysis has shown that the composition of the entire ESS sample, which also includes interviews conducted as a result of the third and subsequent contact attempt, is considerably different from the sample comprising only easy-to-get respondents, i.e. those who were successfully interviewed during the first or the second contact attempt. Out of the seven key socio-demographics included in the analysis, five showed significant differences in distributions. The only variables without such differences were respondent’s sex and the number of people in the household. The sample achieved as a result of up to two contact attempts contains a larger number of elderly people, those living in the countryside and those having either primary education or the first stage of basic education. In comparison with the total ESS sample, the easy-to-get sample has fewer people living in major cities, people with tertiary education, those in paid work and those achieving the highest income. However, in order to assess the composition of those samples, it is important to compare them with the general population. The results of such a comparison for five demographic variables indicate that the share of nearly all of the indicated categories is closer to the general population in the total ESS sample than in the easy-to-get sample. An exception is the case of respondents with either primary or the first stage of basic education where the reverse is true. It is worth noting that the number of people in the household, i.e. the variable which revealed the greatest differences between the two samples and the general population, turned out to be completely resistant to repeated contact attempts and refusal conversion attempts. Generally speaking, the results of the analysis presented in this paper suggest that the standard face-to-face survey research design, which assumes no more than two contact attempts with the sampled persons, poses a risk for the representativeness of the achieved sample; certain categories of respondents will be overrepresented while others will be underrepresented. This risk can be considerably reduced thanks to multiple contact attempts. Our analysis demonstrates that repeated contact attempts with hard-to-get respondents bring the benefit not only of an increase in the response rate but also of an improvement in the sample composition. Unfortunately, the approach involving multiple contact attempts entails much higher survey costs.

• Details
• Contents