Mixed-Mode Surveys: A Strategy to Reduce Costs and Enhance Response Rates

Background

Current budget constraints and expectations to rigorously evaluate programs combine to create difficult conditions for Extension. In order to continue to determine program outcomes, Extension needs to limit costs and preserve the integrity of data collection.

Mixed-mode surveys present one opportunity to conduct quality survey research at lower costs (Dillman, Smyth, & Christian, 2009; Millar & Dillman, 2011). A mixed-mode survey is implemented when different media (e.g., paper and Web-based) are utilized for data collection. If implemented carefully, mixed-mode surveys can also reduce coverage error and improve response rate (Dillman et al., 2009; Israel, 2010, 2011).

For a follow-up evaluation on a series of on-farm food safety workshops, we applied a mixed-mode survey. The workshops were conducted in eight locations across Pennsylvania in winter 2011 and aimed to train produce growers on the implementation and documentation of Good Agricultural Practices (GAPs). We found that a mixed-mode survey design nearly doubled the response rate between early and late respondents.

Methods

Contact information for participants was initially gathered during the workshops. All participants were asked to provide mailing addresses and email addresses only if they used their email regularly. When the time came to conduct the follow-up evaluation 6 months later in fall 2011, initial communication was based on the contact information provided. Those who provided only a mailing address were first sent a pre-notice and subsequently a paper survey via postal mail. Those who provided their email address were emailed a pre-notice followed by a survey invitation with a link to the Web-based survey hosted by SurveyMonkey. The mail and online surveys had unified formats (identical question order, structure, and wording), helping to reduce measurement error (Dillman et al., 2009). In total, 219 surveys were either emailed (132) or mailed (87). After sending the survey, we found five cases missing at random, reducing the total to 214.

Three weeks after sending the first survey, paper replacement surveys were mailed to all 139 non-respondents, regardless of whether they were first contacted by email or postal mail. With no assurance that emails were successfully delivered to inboxes or opened by the workshop participants, we changed the communication method (Dillman et al., 2009). For those who initially received a paper survey via postal mail, the replacement survey color was changed in order to help track responses of early and late mail respondents. The same color replacement survey was also mailed to those non-respondents who were initially emailed. We used serial numbers to differentiate between those late respondents who were initially contacted via postal mail and those who were initially contacted via email.

Ten days after mailing the replacement surveys, we began telephone calls to non-respondents in order to encourage participants to complete and return the survey. We switched the contact method to telephone calls in order to contrast our previous online and postal mailing communication attempts (Dillman et al., 2009). Telephone calls were made over 5 days. A maximum of three calls were made to directly communicate with the workshop participants. Repeated phone calls were made on different days and at different points in time. We left messages on either an answering machine or with the person who answered. If we spoke directly with the respondent, we offered to resend the survey through email, if appropriate, or mail another copy of the paper survey if it had not yet arrived.

Response Rates

The response rates indicate that email respondents comprised the majority of early respondents (51 versus 23). Overall, 74 out of 214 (34.5%) surveys were returned after the first mailing. In the 10 days after we mailed replacement surveys but before we began the telephone calls, we received an additional 29 (13.5%) paper surveys. After the phone calls were completed, we received another 41 surveys (19.1%): 35 via postal mail and six via the Web. In all, follow-up efforts with non-respondents resulted in a 32.6% (70 out of 214) increase for a total response rate (67.1%). The mixed-mode effort nearly doubled the response rate between early and late respondents and increased the overall response rate by nearly one-third. Table 1 presents a breakdown of response rates.

Table 1.
Response Rates Based on Mode and Respondent
Category (N=214)

	Email	Postal Mail	Total
Early respondent	51 (23.8%)	23 (10.7%)	74 (34.5%)
Late respondent before telephone calls	0 (0.0%)	29 (13.5%)	29 (13.5%)
Late respondent after telephone calls	6 (2.8%)	35 (16.3%)	41 (19.1%)
Total	57 (26.6%)	87 (40.6%)	144 (67.1%)

Conclusions and Recommendations

A mixed-mode survey saved an estimated $250 by reducing the number of postal mailings and yet maintained responses (67.1%) at an acceptable rate (60%) for mailed surveys (Singleton & Straits, 2005). Although Web-only surveys further reduce costs, they typically have lower response rates and risk increased coverage error (Dillman et al., 2009; Manfreda, Bosnjak, Berzelak, Haas, & Vasja, 2008; Shinn, Baker, & Briers, 2007).

When implementing a mixed-mode survey, careful attention to details is important. The following suggestions should help guide the use of mixed-mode surveys.

• The types of mode most accessible to the target population must be considered. Planning an appropriate sequence of communication can help increase response rate (Millar & Dillman, 2011).
• Survey format for the different modes must be taken into account. While unified formats reduce measurement error, changing the format based on mode type can provide more ease for respondents in some cases (Dillman et al., 2009).
• Keeping an updated respondent list that tracks survey mode and return date reduces the cost of mailing replacement surveys and time dedicated to telephone calls.

Despite the potential benefits that mixed-mode surveys offer, caution is advisable. Testing for differences between modes is important because responses on questions can differ according to the type of mode (Israel, 2010). By creating a variable for mode type in the SPSS data file, differences in responses to questions can be tested in a similar manner to early and late respondent comparisons (Radhakrishna & Doamekpor, 2008).

With careful implementation, mixed-mode surveys provide an opportunity for Extension to adapt to budget limitations and still conduct rigorous survey research. Future studies should incorporate mixed-mode surveys and experiment with which modes and mode sequences are most appropriate for different target populations. Doing so will help Extension reduce costs and increase response rates.

Acknowledgments

Partial funding for the work reported here was provided by the USDA Sustainable Agriculture Research and Education Program.

References

Dillman, D.A., Smyth, J.D., & Christian, L.M. (2009). Internet, mail, and mixed-mode surveys: The tailored design method (3rd ed.). Hoboken: John Wiley & Sons.

Israel, G. (2010). Using Web-hosted surveys to obtain responses from Extension clients: A cautionary tale. Journal of Extension [On-line], 48(4) Article 4FEA8. Available at: http://www.joe.org/joe/2010august/a8.php

Israel, G. (2011). Strategies for obtaining survey responses from Extension clients: Exploring the role of e-mail requests. Journal of Extension [On-line], 49(3) Article 3FEA7. Available at: http://www.joe.org/joe/2011june/a7.php

Manfreda, K. L., Bosnjak, M., Berzelak, J., Haas, I., & Vasja, V. (2008). Web surveys versus other survey modes. International Journal of Market Research, 50(1), 79-104.

Millar, M. M., & Dillman, D. A. (2011). Improving response to Web and mixed-mode surveys. Public Opinion Quarterly, 75(2), 249-269.

Radhakrishna, R., & Doamekpor, P. (2008). Strategies for generalizing findings in survey research. Journal of Extension [On-line], 46(2) Article 2TOT1. Available at: http://www.joe.org/joe/2008april/tt1.php 

Shinn, G., Baker, M., & Briers, G. (2007). Response patterns: Effect of day of receipt of an e-mailed survey instrument on response rate, response time, and response quality. Journal of Extension [On-line], 45(2) Article 2RIB4. Available at: http://www.joe.org/joe/2007april/rb4.php

Singleton, R.A., Jr., & Straits, B.C. (2005). Approaches to social research (4thh ed.). New York: Oxford University Press.