Multiple Year Extension Program Outcomes & Impacts Through Evaluation

Introduction

Shrinking federal, state, and local funding has created numerous challenges for Extension, not the least of which is an increased demand for accountability and evaluation of programming. Grant funders are requiring more accountability as well. This need and demand for evaluation and accountability is nothing new to Extension (Bennett, 1975; Bush, Mullis, & Mullis, 1995; Arnold, 2002; Radhakrishna, & Bowen, 2010; Lekies, & Bennett, 2011).

Part of the challenge is that Extension educators continue to struggle with the concept of program evaluation—how to define it, how to focus it to garner intended outcomes and impacts, how to accomplish it, how to report results, etc. Many feel it takes too much time and requires too many resources, or they grudgingly do it as a program add-on or after thought (Sims, Dobbs, & Hand, 2002; Lyons, Palmer, Jayaratne, & Scherpe, 2006).

Very simply evaluation is defined as "the process of determining the merit, worth, or value of something or the product of that process" (Scriven, 1991, p. 139). Substantive Extension program evaluation resulting in measurable outcomes and impacts reported to stakeholders is essential to Extension's financial and long-term organizational success (Davis, 2003; Stup, 2003; Bravermann, & Engle, 2009; Morse, 2009). Evaluative outcomes and impacts can be used to inform curriculum development and delivery (Kelsey, Schnelle, & Bolin, 2005; Taylor, 2008; Duerden, & Witt, 2012).

The objective of this article is to highlight a multi-phase, multiple program year evaluation process resulting in substantive program outcomes and impacts. The Extension program focus is a business transition estate planning program effort for years 2004 through 2011 including data from Minnesota, Montana, North Dakota, Iowa, South Dakota, and Wisconsin.

Program Development, Content, and Delivery

Program curriculum was developed by University of Minnesota Extension. Program goals included (1) introducing farm/ranch families to transition and estate planning laws, rules, and guidelines; (2) outlining potential strategies and procedures for developing a business transition and estate plan; (3) identifying key risk factors associated with the lack of planning, lack of goal setting, and lack of intergenerational communication; (4) participants' increased knowledge and understanding of the key topics in the program curriculum; and (5) program participants developing and implementing a business transition and personal estate plan.

Program key educational points included personal, family, business, and retirement goal setting; intergenerational communication; business entity and transition strategies; tax issues; personal estate planning techniques; treatment of heirs; and completing a written transition plan outline.

Program curriculum was delivered via face-to-face workshops using PowerPoint© slides. Educational material consisted of a participant workbook that contained all PowerPoint© slides, all chapter worksheets, listings of Internet resources, and 27 information sheets. The workshop is 5 hours in length.

Business transition is an appropriate program platform for this evaluative process. It is an ongoing issue and has been a crucial programming issue, as shown by a plethora of professional literature, programs and professional presentations (Lobley, Baker, Whitehead, 2010; Bradley & Jenkins, 2009; Heleba, Parsons, & Sciabarrasi, 2004; DeVaney, 2004; Van der Hoeven, 2003; Fetsch, 1999; Weigel, Blundall, & Weigel, 1986). The program format, key educational points, and topics included in the program were designed to increase participant knowledge and lessen the complexity of the process, enabling them to move forward with the transition and personal estate planning process.

Evaluative Methodology

Stage 1 evaluation involved participants completing an end-of-meeting evaluation immediately following each workshop. Participants self-reported any increase in knowledge and understanding around the key educational points of the workshop, status as to their business transition and personal estate plans, and demographics about themselves and their business. Participants were asked to list the workshop topics they felt would be the most beneficial to them.

Stage 2 evaluation involved a follow-up evaluation sent to each participant business approximately 6 months following the last workshop of the year. Participants were asked to report their progress toward the development and implementation of a business transition and personal estate plan, challenges or obstacles they encountered, and the usefulness of the workshop and workbook in the process.

Participant Demographics

Table 1 illustrates the demographics of participants and the status of their business. The information includes program years 2004 through 2011 for participants from Minnesota, Montana, North Dakota, Iowa, South Dakota, and Wisconsin.

One hundred seven workshops involving 165 local business sponsors were presented in four states over the 7-period being evaluated. A total of 3,705 farm and ranch family members attended. Ages ranged from 22 to 89 years of age, with 60.5% male, 39.5% female, and 57.2% over the age of 55.

Many program participants had no business transition or estate plan (Table 1). Over 83% had no business transition plan or business successor. A total of 69.8% had no personal estate plan.

Table 1.
Participant Demographics: 2004 Through 2011, All States

Number of Workshops	107
Total Attendees	3,705
Number Farms/Ranches Represented	1,801
Local Communities Represented	1,166
Local Business Sponsors	165
Participant Age Range	22 to 89 years
Participants Over Age 55	57.2%
Participants Male	60.5%
Participants Female	39.5%
Participants Without Business Transition Plan	83.7%
Participants Without Personal Estate Plan	69.8%

Evaluation Results

Program outcomes were the primary focus of the end-of-meeting evaluation process (Table 2). The goal was to determine what, if any, increase in knowledge and understanding of the educational points of the program were achieved by participants.

Program impacts were the primary focus of the follow-up evaluative process (Table 3). The evaluation was intended to determine how many participants acted on their new knowledge and understanding of the program topics. That is, did they indeed develop and implement a business transition and personal estate plan as a result of attending the program.

End-of-Meeting Evaluative Results: Documenting Knowledge Change

Participants were asked to self-report any increase in knowledge and understanding of the key educational points in the program. Participants were asked to respond using a simple five-point Likert Scale with 1 = strongly disagree and 5 = strongly agree.

Median scores were all 4 or greater on the evaluation Likert Scale, indicating that participant responses were grouped near the high end of the scale (Table 2). Mean scores were also above 4. Standard deviations were small, indicating scores were grouped closely together.

Most participants stated that their knowledge and understanding of the educational points in the program increased due to attending a workshop. The percent indicating they agreed or strongly agreed that their knowledge and understanding increased ranged from 88.7% to 98.9% (Table 2).

Table 2.
Participants' Self-Reporting Whether Their Knowledge and Understanding of Educational Points Increased Due to Attending a Workshop

Educational Points	Median Score	Mean Score	Standard Deviation	% Indicating "Strongly Agree" Or "Agree"	N
Understanding of the need for clear goals & communication in the transition process.	5	4.53	.52	98.9%	2059
Greater understanding of strategies available for use in the transition process.	4	4.27	.54	95.2%	2059
Importance of assessing the financial strength of the business before the transition.	4	4.37	.58	95.1%	2059
Greater understanding of tax issues related to the transition process.	4	4.20	.63	88.7%	2059
Greater understanding of personal estate planning issues.	4	4.29	.59	93.1%	2059

Participants were asked to list, in order of importance, topics from the workshop they felt would be the most valuable to them as they began the transition and estate planning process. The top five topics were: the discussion on trusts as an estate planning tool (n=504), the need for goals and good communication (n=296), the discussion of wills (n=254), the workbook and curriculum materials (n=235), and tax issues (n=69).

Post-Meeting Evaluative Results and Program Impact

A follow-up evaluation was implemented to determine program impacts. The intent was to determine whether program participants acted on their new knowledge, resulting in the development and implementation of a business transition and personal estate plan. Six months following the final workshop each year, a mail-out survey was sent to all farm and ranch units with a current mailing address for years 2004 through 2011. A total of 1,828 surveys were mailed, with 702 returned for a 38.4% return rate.

As few as 37 and as many as 85 of the participating farm/ranch units each year started on the development of their business transition plan (Table 3). A total of 468 business units began the process over the 7-year period. Of that group, as few as seven and as many as 27 actually completed the process by implementing their business transition plan, for a total of 116 business units (24.8%) completing the process.

For the same period, as few as 35 and as many as 92 business units each year started their personal estate planning process, for a total of 476 business units over the 7-year period. As few as four and as many as 26 actually completed and implemented their personal estate plans, or a total of 112 business units. All this occurred within 6 months of the final workshop for each year represented (23.5%).

Table 3.
Number/Percent Farm/Ranch Units Starting/Completing Business Transition & Personal Estate Plans

Task	Years/States							Total Businesses
	2004-05 (MN)	2005-06 (MN)	2006-07 (MN)	2007-08 (MN, ND, IA, WI)	2008-09 (MN, ND, IA)	2009-10 (MN, ND, IA)	2010-11 (MN, ND, MT, SD)
Started Business Transition Plan	37 (58.8%)	57 (59.4%)	84 (61.8%)	77 (72.9%)	65 (67.2%)	63 (68.5%)	85 (78.0%)	468
Completed & Implemented Business Transition Plan	7 (11.6%)	12 (12.5%)	16 (12.2%)	16 (15.7%)	21 (22.2%)	17 (27.0%)	27 (31.8%)	116
Started Personal Estate Plan	35 (56.9%)	55 (57.3%)	92 (67.2%)	83 (79.2%)	68 (70.2%)	60 (65.2%)	83 (76.1%)	476
Completed & Implemented  Personal Estate Plan	4 (6.5%)	7 (7.3%)	19 (14.4%)	18 (17.1%)	22 (23.4%)	16 (26.7%)	26 (31.3%)	112

Participants were asked to list, in priority, the challenges and obstacles they encountered as they began the business transition and estate planning process. Obstacles in order of frequency reported were: finding time to address the process (n=141), difficulty in developing a list of goals for the process (n=79), difficulty in deciding how to divide assets among heirs (n=53), the owner generation not yet ready to retire and transition the business to the next generation (n=51), and finding competent professionals they felt comfortable working with (n=46).

A total of 99.3% of attendees stated the workbook was a useful reference tool to them as they worked toward development and implementation of their business transition and estate plan.

As further testimony to the usefulness and success of the program, a farmer from south central Minnesota who attended a workshop wrote:

We attended the workshop which prompted us to put together our farm business transition plan and succession plan. Within six months after getting the plans in place we were forced to act on them due to a farm accident that killed my brother who was the majority owner of the farm. Having gone to the workshop and getting a plan in place saved our family business from potential chaos. We are extremely grateful for the help and prompting we received by attending the workshop.

Summary and Conclusions

Data from the end-of-meeting evaluations highlighted program outcomes. A large percentage of participants self-reported an increase in knowledge and understanding of the educational points in the program. That would indicate the program effort was a success. Given the complexity of business transition and personal estate planning, the percentage agreement or strong agreement in excess of 88% is a strong verification of program success.

Topics reported by participants as being most valuable to them in the planning process have been informative to the program team. The list of topics guides what is included in the curriculum materials, at each update, as the key educational points for the program.

The follow-up evaluative data shows that participants did indeed act on their new knowledge in the form of starting and completing the transition and estate planning process. For the farm and ranch units returning follow-up evaluations over the 7-year period, 66.7% had started the business transition process. Of those starting the process, a total of 116 (16.5%) returning surveys completed the business transition process. A total of 67.8% of those returning surveys started the personal estate planning process. Of those, 112 (16.0%) actually completed the personal estate planning process.

The list of obstacles reported was also informative to the program team. The curriculum was modified during each of 12 revisions to address some of the obstacles. A list of websites where families can locate attorneys has been included in the materials.

The development team learned that starting with end in mind was crucial. That is, what do you want to impact, change, and evaluate can greatly inform how a program is developed and delivered. Given the complexity of transition and estate planning issues, a clear and concise program focused on addressing these issues can be successful. Counseling with an attorney and estate planner gave the team beneficial background on the issues and questions facing the farm and ranch clients they deal with. Using this information enabled the team to focus on what change or impacts would help farm and ranch families through the planning task. This focus then informed program curriculum development.

Because the evaluative process was an integral, ongoing part of program development and delivery, the team was able to glean the evaluative data planned for. The data gleaned has shown that the program was successful and achieved the desired outcomes and impacts over a 7-year period. Evaluative data also informed the program team, resulting in improved curriculum. Given the continued requests for the program, the complexity of the issues involved, increasing asset values requiring planning to avoid unnecessary tax issues, an aging farm/ranch owner population and the evaluative results, the team will continue to offer the program.

 

References

Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension [On-line], 40(3) Article 3FEA4. Available at: http://www.joe.org/joe/2002june/a4.php

Bennett, C. (1975). Up the hierarchy. Journal of Extension [On-line], 13(2). Available at: http://www.joe.org/joe/1975march/1975-2-a1.pdf

Bradley, D., & Jenkins, J. (2009). The importance of succession planning to the family farm in a volatile environment. Small Business Institute National Proceedings, 33(1), 254-268.

Bravermann, M. T., & Engle, M. (2009). Theory and rigor in Extension program evaluation planning. Journal of Extension [On-line], 47(3) Article 3FEA1. Available at: http://www.joe.org/joe/2009june/a1.php

Bush, C., Mullis, R., & Mullis, A. (1995). Evaluation: An afterthought or an integral part of program development? Journal of Extension [On-line], 33(2) Article 2FEA4. Available at: http://www.joe.org/joe/1995april/a4.php

Davis, G. A. (2003). Using a retrospective pre-post questionnaire to determine program impact. Journal of Extension [On-line], 41(4) Article 4TOT4. Available at: http://www.joe.org/joe/2003august/tt4.php

DeVaney, S. A. (2004). A Web site to help farm families communicate about farm transfer. Journal of Extension [On-line], 42(4) Article 4TOT5. Available at: http://www.joe.org/joe/2004august/tt5.php

Duerden, M. D., & Witt, P. A. (2012). Assessing program implementation: What it is, why it's important, and how to do it. Journal of Extension [On-line], 50(1) Article 1FEA4. Available at: http://www.joe.org/joe/2012february/a4.php

Fetsch, R. F. (1999). Some do's and don'ts for successful farm and ranch family estate transfers. Journal of Extension [On-line], 37(3) Article 3IAW2. Available at: http://www.joe.org/joe/1999june/iw2.php

Heleba, D., Parsons, R., & Sciabarrasi, M. (2004). Minimizing farm business succession risk in New England: Delivery of transferring the farm workshops. Journal of Extension [On-line], 42(6) Article 6RIB6. Available at: http://www.joe.org/joe/2004december/rb6.php

Kelsey, K. D., Schnelle, M., & Bolin, P. (2005). Increasing educational impact: A multi-method model for evaluating Extension workshops. Journal of Extension [On-line], 43(3) Article 3FEA4. Available at: http://www.joe.org/joe/2005june/a4.php

Lekies, K. S., & Bennett, A. M. (2011). The evaluation attitudes and practices of 4-H educators. Journal of Extension [On-line], 49(1) Article 1RIB2. Available at: http://www.joe.org/joe/2011february/rb2.php

Lobley, M., Baker, J. R., & Whitehead, I. (2010). Farm succession and retirement: Some international comparisons. Journal of Agriculture, Food Systems and Community Development, 1(1), 49-64.

Lyons, A. C., Palmer, L., Jayaratne, K. S. U., & Scherpf, E. (2006). Are we making the grade? A national overview of financial education and program evaluation. The Journal of Consumer Affairs, 40(2), 208-235.

Morse, G. W. (2009). Program quality results. In George W. Morse (Editor), The Minnesota Response: Cooperative Extension's Money and Mission Crisis (pp. 217-247). Bloomington, IN: iUniverse Publishing.

Radhakrishna, R., & Bowen, C. F. (2010). Viewing Bennett's hierarchy from a different lens: Implications for Extension program evaluation. Journal of Extension [On-line], 48(6) Article 6TOT1. Available at: http://www.joe.org/joe/2010december/tt1.php

Scriven, M. (1991). Evaluation Thesaurus. Newbury Park: Sage Publications.

Sims, R., Dobbs, G., & Hand, T. (2002). Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation. Distance Education, 23(2), 135-148.

Stup, R. (2003). Program evaluation: Use it to determine value to potential clients. Journal of Extension [On-line], 41(4) Article 4COM1. Available at: http://www.joe.org/joe/2003august/comm1.php

Taylor, G. D. (2008). Better Extension programming through statistics: Using statistical analysis of program evaluations to guide program development. Journal of Extension [On-line], 46(5) Article 5RIB4. Available at: http://www.joe.org/joe/2008october/rb4.php

Weigel, D. J., Blundall, J. S., & Weigel, R. R. (1986). Keeping peace on the farm: Stresses of two-generation farm families. Journal of Extension [On-line], 24(2) Article 2FEA1. Available at: http://www.joe.org/joe/1986summer/a1.php

Van der Hoeven, G. (2003, August 10). Estate planning and farm transfer in a changing legislative environment: North Carolina, U.S.A. an example. Paper presented at the 14th Congress of the International Farm Management Association, Perth, Western Australia. Retrieved from: http://purl.umn.edu/24400