Evaluating Utah's Rural Online Initiative: Empowering Rural Communities Through Remote Work

Introduction and Conceptual Framework

Extension professionals fulfill the land-grant university mission by translating and disseminating research-based knowledge to communities through a wide range of nonformal educational programs (Franz & Townson, 2008; Garst & McCawley, 2015; Rasmussen, 1990). Extension programs are most effective when they specifically address identified problems faced by communities (Fetsch et al., 2012; Hetherington et al., 2019). As a result, there is increased pressure on Extension to provide relevant and impactful programs that address the critical needs of residents and communities (Braverman & Engle, 2009; Gagnon et al., 2015). The key issues affecting communities should inform Extension programming; Extension professionals should monitor community needs to design innovative, capacity-building educational programs where residents learn new knowledge and skills to respond to issues in their own communities (Graham et al., 2016).

Program planners at Utah State University (USU) Extension sought to understand the problems facing residents in urban and rural counties through a statewide needs assessment (Narine, 2019). A needs assessment is meant to guide relevant Extension programs (Garst & McCawley, 2015). Some needs identified in the assessment included well-paying jobs, steady jobs, and vocational skills programs. However, these needs are not unique to Utah. Thiede and Monnat (2016) explained that for decades jobs have been scarce and/or unstable in most rural communities in the West and Midwest. Additionally, Sherman (2006) found that residents from California's sparsely populated rural communities had to aggressively compete over the limited supply of mostly low-paying jobs. Pew Research reported lack of availability of jobs, lack of affordable housing, poverty, and unemployment as widespread problems across rural American communities (Parker et al., 2018). Because rural communities face low population density, residents more frequently deal with these issues as compared to residents in urban areas (Curtin & Cohn, 2015).

As a critical first step to program planning, we used the aforementioned recent statewide needs assessment (Narine, 2019) in our design and development of a new capacity-building program. Our intent was to address high-priority issues faced by residents of rural counties, namely lack of well-paying jobs, steady jobs, and vocational skills programs. Although Utah has experienced significant positive economic growth in recent years, including achievement of a statewide unemployment rate under 3% (Gochnour, 2019), many rural counties still experience increasing unemployment rates. Some rural counties have faced a 20% to 35% level of layoffs (Burkitt, 2017) due to automation and declines in extraction industries. Given increasing rural unemployment rates and the intergenerational poverty designation in 11 of Utah's 29 counties, USU Extension was appropriated $2.2 million in funding by the Utah Legislature to design and develop a pilot program to address declining economic growth in rural counties. USU Extension developed the Rural Online Initiative (ROI) to provide rural residents with the skills needed to engage in remote work opportunities. USU Extension recognized remote work as a solution to unemployment in rural communities as it facilitates job creation without rural–urban migration and facilitates additional income for rural residents (Greer & Payne, 2014).

Remote work opportunities in the United States have increased by approximately 159% since 2005 (Gardner & Lister, 2017). This change aligns with the decreasing number of employees assigned to physical workplaces (Brotherton, 2012). Similarly, 94% of the net employment growth in the United States from 2005 to 2015 "occurred in alternative work arrangements" (Katz & Krueger, 2016, p. 7). These trends suggest an opportunity to connect residents in rural Utah to remote work. By investing in remote work education, rural communities can become sustainable over the long term because compatible skills can be transferred to other jobs (Venkatesh & Speier, 2000). Remote jobs also are well paying and steady and save households the cost of travel to and from physical work locations (Bibby, 2019). However, there are some challenges associated with working remotely. These include workflow disruptions, isolation, burnout, and difficulties in seamless collaborations (Greer & Payne, 2014; Mahler, 2012; Pyöriä, 2011).

To address these challenges, we designed the Master Remote Work Professional (MRWP) certificate course under the ROI. The MRWP course provides specialized training and services for remote employees, online freelancers, and e-commerce entrepreneurs (Reese et al., 2018). We piloted this online course in 2018 (Gillmor, 2018; Noel & Hinkins, 2018), with the overall goal of reducing the social and economic problems of rural unemployment in Utah through remote work opportunities. Our main objective was to target the human capacity of rural communities by educating residents on securing well-paying and steady remote jobs. Through eight modules, participants learn the skills needed to transition from physical or on-site work to a virtual career. A summary of each module is provided in Table 1.

Table 1.
Module Summary of the Master Remote Work Professional Course

Module	Description
Workday	Describes components of a typical workday for remote workers and highlights expectations
Communication	Explains effective communication using online tools such as Zoom and Slack
Workflow	Defines workflow best practices that allow remote workers to identify priorities and create effective schedules
Productivity and time management	Explains how to maintain productivity by implementing schedules, identifying type of work tasks, and time tracking
Teams	Describes virtual team collaboration, trust among coworkers, and challenges of working in virtual teams
Compliance	Outlines best practices for information security compliance and laws and regulations that protect sensitive data
Critical thinking	Defines work ethic, working independently, and thinking ahead; highlights tips on how to problem solve and deal with situations as they arise in a remote work environment (e.g., working with team members in different time zones)
Remote job development	Focuses on remote work opportunities, the application process, and credible remote work companies

Legislatively funded programs must demonstrate public value. Therefore, we evaluated the short-term outcomes of the MRWP. In this regard, short-term outcomes refer to participants' knowledge, attitudes, skills, and aspirations (KASA) toward securing remote employment following completion of the MRWP certificate course. The targeting outcomes of programs (TOP) model (Rockwell & Bennett, 2004) guided our evaluation plan and served as a road map in planning the intervention to address the problem of rural unemployment. A key assumption of the TOP model is that the program development process can be reflected in the evaluation process. While the TOP hierarchy describes seven levels in program planning and evaluation, our results focus on short-term program outcomes (i.e., KASA).

The TOP model identifies seven steps in program planning. It begins with a description of the social, economic, and environmental (SEE) conditions to assess gaps between "what is" and "what should be." Desired practices are then determined; these are the actions or behaviors that play a role in closing the gap in SEE conditions. The next level, and most applicable to our study, details the KASA of participants, in our case, after completing the MRWP certificate course. The TOP model assumes that a positive change in KASA leads to adoption of desired actions or behaviors that closes the gaps in SEE conditions (Rockwell & Bennett, 2004). Following the TOP model, we theorized that if MRWP participants increase their knowledge and skills related to remote work, hold positive attitudes toward the benefits of remote work, and exhibit strong intentions to look for remote work, they are more likely to find remote work opportunities (i.e., medium-term outcome). Sequentially, short-term outcomes would likely lead to the realization of the medium-term outcomes of participants' finding and securing remote work. Next, if participants can maintain their remote employment over time, they are likely to benefit from increases in income and job satisfaction and an improved quality of life. In the long run, the social and economic problem of rural unemployment would decline, narrowing the relevant SEE gap. Other Extension professionals could consider Rockwell and Bennett's (2004) TOP model for planning and evaluating nonformal educational capacity-building programs in their own communities.

Methodology

We attempted to collect data from a census of participants from the May to August 2019 cohorts of the MRWP course (N = 1,025). We used two methods for data collection: A mandatory pre-/posttest design measured knowledge gain (N = 1,025), and an exit questionnaire assessed participants' attitudes toward remote work, perceived skill acquisition, and aspirations for gaining remote employment (n = 455). The pre- and posttests consisted of exam-type questions based on the eight MRWP course modules. We used a paired-samples t test to measure knowledge change from before to after the course. The null hypothesis for the t test was rejected at p < .05. The tailored design method (Dillman et al., 2014) informed the exit questionnaire. The exit questionnaire assessed perceived skill improvements via multiple ordinal items related to seven operational constructs: work–life balance, productivity, problem solving, digital communication, online technology, teamwork, and career management. We used a 5-point Likert-type scale to assess individual items under each construct, and we interpreted overall mean scores for each construct using the following improvement scale: 1.00–1.49 = much worse, 1.50–2.49 = somewhat worse, 2.50–3.49 = stayed the same, 3.50–4.49 = somewhat better, and 4.50–5.0 = much better (Scales et al., 2009). The exit survey questionnaire was reviewed by a panel of experts for validity. Although we did not conduct a pilot test, we used a post hoc Cronbach's alpha to determine internal consistency reliability of operational constructs. An alpha value of 0.7 and above was considered acceptable (Field, 2006). All construct variables had acceptable internal consistency (see Table 3 in the Findings section). Descriptive statistics were reported for attitudes, skills, and aspirations.

Findings

Although all participants were required to complete an attendance survey and pre- and posttests, only 455 responded to the voluntary exit questionnaire. Therefore, demographic data and knowledge gain were obtained for all participants enrolled in the course (N = 1,025), and data from the exit questionnaire reflected 455 participants (n = 455). Most MRWP course participants (74%) were female, and 37% were over 45 years old. About 16% had a high school diploma, 14% had an associate's degree, and 28% listed some college as their highest level of education. Even though most participants (71%) did not have remote work experience, a large proportion (92%) indicated that it was important or very important to them to acquire remote work skills. Regarding knowledge gain, results showed that there were statistically significant differences between pretest and posttest scores for all eight modules. These results indicated significant increases in participants' knowledge after completing all modules of the MRWP course (see Table 2).

Table 2.
Paired-Samples t-Test Results for Assessing Changes in Knowledge

Module	M	SD	t	df	p (one-tailed)
Communication	1.50	2.83	7.88	222	.000
Compliance	4.22	2.96	19.61	188	.000
Critical thinking	2.80	2.40	15.81	185	.000
Productivity	3.12	2.83	15.58	198	.000
Remote job development	1.52	2.86	7.16	181	.000
Teams	1.08	2.01	7.55	196	.000
Workday	3.67	3.94	14.47	240	.000
Workflow	0.76	3.07	3.49	197	.000

With respect to skills, on average, participants had high overall mean scores for work–life balance, productivity, problem solving, digital communication, online technology, teamwork, and career management. These results indicate that after completing the MRWP course, participants perceived that they had somewhat better abilities to balance their professional and personal lives, manage their professional and personal productivity, solve problems related to working remotely, communicate digitally, use online technology, contribute as a team member in their professional and personal lives, and manage their careers (see Table 3). When asked about attitudes toward their current jobs, on average, participants felt neutral toward their current job positions and desired higher incomes. Nearly all participants (96%) indicated feeling that their value as a remote worker was improved after completing the MRWP course, and 96% also said they felt empowered to seek remote work after completing the MRWP course.

Table 3.
Descriptive Statistics and Internal Consistency Results for Skill Constructs

Skill construct	M	SD	Cronbach's α
Work–life balance	4.04	.63	.89
Productivity	4.22	.60	.88
Problem solving	4.13	.64	.88
Digital communication	4.06	.61	.74
Online technology	4.01	.64	.88
Teamwork	4.10	.66	.94
Career management	4.14	.65	.84

Conclusions, Recommendations, and Implications

The long-term goal of the MRWP course is to reduce the social and economic problems related to rural unemployment in Utah. Preliminary results from our study showed that desired short-term outcomes from the course were realized, which, over time, may translate into an increase in remote employment in rural communities. There was an increase in knowledge among the participants for all the online modules, and there were increases in participants' perceptions regarding their abilities to perform remote work skills. Additionally, our evaluation suggests that most participants had strong intentions and motivations related to seeking remote employment. As a capacity-building effort, the ROI program demonstrated how traditional rural economic development strategies can be adapted to meet community needs. Compared to traditional efforts that depend on job creation through business recruitment, the ROI program provides people in rural communities with the knowledge and skills they need to respond to challenges in their own communities, namely the key issues of well-paying and steady jobs.

On the basis of our study, we recommend formative evaluation for continuous improvement of the MRWP course and the ROI. We also recommend implementation of follow-up evaluations to measure participants' success in securing remote employment. To inform future efforts of the ROI, we plan to conduct a descriptive study on employers' knowledge and willingness to provide remote employment opportunities to qualified rural residents. Their perceptions of any skill gaps would provide us with vital information to refine the current intervention to facilitate greater long-term impacts.

Extension professionals can plan and evaluate their programs using the framework we have presented herein.

Author Note

Correspondence concerning this article should be addressed to Paul Hill. Email: paul.hill@usu.edu

References

Bibby, A. (2019, December 13). 18 high paying remote jobs with $100K+ salaries. Flexjobs. https://www.flexjobs.com/blog/post/top-paying-remote-jobs/

Braverman, M. T., & Engle, M. (2009). Theory and rigor in Extension program evaluation planning. Journal of Extension, 47(3), Article v47-3a1. https://joe.org/joe/2009june/a1.php

Brotherton, P. (2012). Alternative workforce programs are on the rise. TD Magazine, 66(3), 28.

Burkitt, B. (2017, August 4). Gov. Herbert urges rural leaders to take action at summit. The Spectrum. https://www.thespectrum.com/story/news/local/cedar-city/2017/08/04/gov-herbert-urges-rural-leaders-take-action-summit/541621001/

Curtin, L., & Cohn, T. J. (2015, January). Overview of rural communities: Context, challenges and resilience. Psychology and AIDS Exchange Newsletter. https://www.apa.org/pi/aids/resources/exchange/2015/01/rural-communities

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). John Wiley & Sons Inc.

Fetsch, R. J., MacPhee, D., & Boyer, L. K. (2012). Evidence-based programming: What is a process an Extension agent can use to evaluate a program's effectiveness? Journal of Extension, 50(5), Article v50-5a2. https://www.joe.org/joe/2012october/a2.php

Field, A. (2006). Reliability analysis. https://www.discoveringstatistics.com/repository/reliability.pdf

Franz, N. K., & Townson, L. (2008). The nature of complex organizations: The case of Cooperative Extension. New Directions for Evaluation, 120, 5–14.

Gagnon, R. J., Garst, B. A., & Franz, N. (2015). Looking ahead: Envisioning the future of the Extension program development model. Journal of Human Sciences and Extension, 3(2). https://www.jhseonline.com/article/view/691

Gardner, K., & Lister, K. (2017). 2017 State of telecommuting in the U.S. employee workforce. https://www.flexjobs.com/2017-State-of-Telecommuting-US/

Garst, B. A., & McCawley, P. F. (2015). Solving problems, ensuring relevance, and facilitating change: The evolution of needs assessment within Cooperative Extension. Journal of Human Sciences and Extension, 3(2). https://www.jhseonline.com/article/view/684

Gillmor, L. (2018, December 20). Rural initiatives to change the landscape of economic development. Utah Governor's Office of Economic Development. https://business.utah.gov/news/rural-initiatives-to-change-the-landscape-of-economic-development/

Gochnour, N. (2019, May 29). Utah's economic commonwealth. Utah Business. https://www.utahbusiness.com/economies-utah/

Graham, D. L., Arnold, S., & Jayaratne, K. S. U. (2016). Research priority 6: Vibrant, resilient communities. In T. G. Roberts, A. Harder, & M. T. Brashears (Eds.), American Association for Agricultural Education national research agenda: 2016–2020 (pp. 1–66). University of Florida Department of Agricultural Education and Communication.

Greer, T. W., & Payne, S. C. (2014). Overcoming telework challenges: Outcomes of successful telework strategies. Psychologist-Manager Journal, 17(2), 87–111. https://doi.org/10.1037/mgr0000014

Hetherington, C., Eschbach, C., & Cuthbertson, C. (2019). How evaluation capacity building grows credible and actionable evidence for Cooperative Extension programs. Journal of Human Sciences and Extension, 7(2). https://www.jhseonline.com/article/view/825

Katz, L., & Krueger, A. (2016). The rise and nature of alternative work arrangements in the United States, 1995–2015. National Bureau of Economic Research. https://doi.org/10.3386/w22667

Mahler, J. (2012). The telework divide: Managerial and personnel challenges of telework. Review of Public Personnel Administration, 32(4), 407–418. https://doi.org/10.1177/0734371X12458127

Narine, L. K. (2019). Situational needs assessment of Utah. Utah State University. https://extension.usu.edu/employee/program-evaluation/statewide-needs-assessment

Noel, M. E., & Hinkins, D. P. (2018). H.B. 327 Rural Online Initiative. https://le.utah.gov/~2018/bills/static/HB0327.html

Parker, K., Horowitz, J. M., Brown, A., Fry, R., Cohn, D., & Igielnik, R. (2018). Views of problems facing urban, suburban and rural communities. Pew Research Center. https://www.pewsocialtrends.org/2018/05/22/views-of-problems-facing-urban-suburban-and-rural-communities/

Pyöriä, P. (2011). Managing telework: Risks, fears and rules. Management Research Review, 34(4), 386–399. https://doi.org/10.1108/01409171111117843

Rasmussen, W. D. (1990). Taking the university to the people: Seventy-five years of Cooperative Extension. American Journal of Agricultural Economics, 72(1), 253–254. https://doi.org/10.2307/1243178

Reese, J., Hill, P., & Swadley, E. (2018, September 7). Utah State University Extension focuses on strengthening Utah's rural economies. Utah State University. https://www.usu.edu/today/index.cfm?id=57945

Rockwell, K., & Bennett, C. (2004). Targeting outcomes of programs: A hierarchy for targeting outcomes and evaluating their achievement. University of Nebraska–Lincoln. https://digitalcommons.unl.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1047&context=aglecfacpub

Sherman, J. (2006). Coping with rural poverty: Economic survival and moral capital in rural America. Social Forces, 85(2), 891–913. https://doi.org/10.1353/sof.2007.0026

Thiede, B. C., & Monnat, S. M. (2016). The great recession and America's geography of unemployment. Demographic Research, 35, 891–928. https://doi.org/10.4054/DemRes.2016.35.30

Venkatesh, V., & Speier, C. (2000). Creating an effective training environment for enhancing telework. International Journal of Human-Computer Studies, 52(6), 991–1005. https://doi.org/10.1006/ijhc.1999.0367