Rural Coverage Bias in Online Surveys?: Evidence from Oklahoma Water Managers

Background

Oklahoma is updating its comprehensive state water plan, which created demand for Extension efforts on several water issues. To guide our efforts, Oklahoma Cooperative Extension Service personnel are collecting information from the state's water system managers. Given time and cost constraints, we are using an online survey format.

Previously, limited adoption of Internet/email in rural communities has resulted in rural coverage bias (low proportion of rural respondents) in online surveys. This is a major concern for the validity and reliability of online surveys of rural populations. Archer (2003) states that due in part to this problem, online surveys are not yet, but soon will be, a valid method for collecting data from rural communities. We pre-tested an online survey to evaluate whether it would achieve comparable results to a mail-based survey.

Purpose of the Online Survey

Our sampling frame was water supply managers in rural and urban communities across Oklahoma. During pre-test, managers were asked to take either an online survey that was developed using SurveyMonkey.com or a traditional paper survey by mail. Online surveys are preferred for several reasons, such as low cost, ease of data cleaning and organizing, and features that reduce survey completion time (Dillman, Smith, & Christian, 2008). For example, SurveyMonkey.com provides templates for various types of questions, which makes question structuring straightforward. It also offers features that are expected to increase response rates: questions are interactive and responses are easily recorded by mouse click, skip logic can reduce the time required to take the survey, and respondent progress can be shown. Figure 1 provides an example survey question.

Figure 1.
Example Question from the Water Managers Survey

However, there are additional risks from using an online survey. One major risk is rural coverage bias, or a disproportionately low response rate from rural populations. Whitacre (2008) estimated that 53% of rural communities and 87% of urban communities in Oklahoma have access to broadband Internet, which suggests rural coverage bias could be a problem. Before implementing our online survey, we pre-tested it to determine whether an online survey would achieve acceptable response rates (RR) in general, but particularly from rural managers.

Implementing the Survey

We collected water managers' contact information from two sources: (1) the Oklahoma Rural Water Association (ORWA) directory and (2) the Oklahoma Municipal League (OML) directory. These included address and telephone information, but no email addresses. In May and June 2009, we called the telephone numbers and asked managers to participate in an online survey. Managers who agreed were asked for an email address and were sent a secure link to the survey via email. We implemented the survey using the Tailored Design Method (Dillman, Smith, & Christian, 2008) and previous methodologies found in the Extension literature (e.g., O'Neill, 2004). The steps were as follows.

• The population was determined from available directories.


• Calls were made to everyone in the directories.


• A summary of the research was presented and respondents were asked to choose an online or mail survey.


• Email addresses were collected and secure links to the survey were sent to managers.


• Follow up calls were made 2 weeks after the initial email to non-respondents

Results

We estimated a population of 627 operating water systems in Oklahoma, with 81% of them rural (ORWA) and 19% of them urban (OML). We successfully contacted 536, with 436 (81%) of them representing rural and 100 (19%) from urban systems. We found 419 water managers (78%) agreeing to take the online survey, 84 (16%) choosing a paper survey, and only 33 (6%) declining altogether (Table 1).

The two most common reasons for preferring the paper survey to the online survey were: (1) no Internet/email access and (2) usually being "in the field" and away from computer access. A breakdown of the survey preferences by community type indicates that roughly the same proportion of rural and urban managers preferred an online version of the survey (79% and 78%, respectively). This indicates that Internet/email adoption is not a significant hurdle. We also found no statistically significant relationship between community type (rural/urban) and preferences for online, paper, or no survey (Fisher's Exact Test for a 2 by 3 matrix, p = 0.8017).

From the 419 agreeing to take the online survey, we selected a stratified sample according to community size (i.e., half rural, half urban) and region (i.e., northwest, southwest, northeast, and southeast). We emailed surveys to 88 water managers (67 rural and 21 urban) and achieved a 53% RR (57% rural and 43% urban), which is much higher than similar surveys of water managers (e.g., 18% RR found by Dickinson, Maddaus, & Madduas, 2000) and similar to end-user water surveys (e.g., Hurd, 2006).

Interestingly, the response rate by rural managers was much higher than that of urban managers, but given the small sample size, we did not find a statistically significant difference in proportions at the 95% confidence level. Whitacre (2007) argues that broadband Internet access is diffusing more slowly for some groups in Oklahoma (e.g., low education) than others (e.g., low income). In this context, we interpret our results to mean that water managers are one of the demographic groups in rural areas that may not face significant Internet/email connectivity issues. We suspect that water managers have more formal education than the general public. Similar groups, like government employees or those in certain fields of education, may also fall into this category.

Conclusions

We investigated whether an online survey of water system managers in Oklahoma would achieve adequate response rates as compared to those found in the literature for mail-based surveys. Previous research indicates that limited Internet/email access in rural areas is a significant hurdle to online surveys. However, we found wide acceptance of an online format in general and higher acceptance and response rate among rural managers. While coverage bias may still be a problem for the broader rural population, we found no significant coverage bias problems among rural water system managers in Oklahoma.

Our results suggest that online survey is a valid method to elicit information from this group and perhaps for government employees and other similar groups. We suggest that online surveys of some rural populations can produce RRs that are competitive to costly paper surveys, and Extension personnel should consider using this method to elicit information from certain stakeholders despite concerns about Internet/email access. Archer (2003) concluded that online surveys would soon be an acceptable method to collect data from the general public; we conclude that the time has come, at least for some rural populations.

References

Archer, T. M. (2003). Web-based surveys. Journal of Extension [On-line], 41(2) Article 4TOT6. Available at: http://www.joe.org/joe/2003august/tt6.php

Dickinson, M. A., Maddaus, L. A., & Madduas, W. O. (2000). Benefits of United States nationwide plumbing efficiency standards. Madduas Water Management.

Dillman, D. A., Smith, J., & Christian, L. (2008) Internet, mail, and mixed-mode surveys: The tailored design method, Third Edition. New York: John Wiley & Sons, Inc.

Hurd, B.H. (2006). Water conservation and residential landscapes: household preferences, household choices. Journal of Agricultural and Resource Economics 31(2):173 - 192.

O'Neill, B. (2004). Collecting research data online: Implication for Extension professionals. Journal of Extension [On-line], 42(3) Article 3TOT1. Available at: http://www.joe.org/joe/2004june/tt1.php

Whitacre, B. E. (2007). Factors influencing the temporal diffusion of broadband adoption: evidence from Oklahoma. The Annals of Regional Science 42:661 - 671.

Whitacre, B. (2008). Extension's role in bridging the broadband divide: focus on supply or demand? Journal of Extension [On-line], 46(3) Article 3RIB2. Available at: http://www.joe.org/joe/2008june/rb2.php