Growing Our Own: A Longitudinal Evaluation of a Professional Development Program for Early-Career 4-H Professionals

Introduction

Within Extension, many states and regions have moved to competency-based professional development systems as means to train and retain Extension personnel. Preparing Extension educators on the basis of a competency model during their first 3 years on the job provides optimum learning at the most crucial stage (Brodeur, Higgins, Galindo-Gonzalez, Craig, & Haile, 2011). Providing face-to-face, small-group training workshops is a highly effective educational delivery method. However, increased workload, lack of time, and lack of funding are barriers to Extension educators' pursuing training opportunities to acquire competencies (Lakai, Jayaratne, Moore, & Kistler, 2012).

The Western Region 4-H Institute (also referred to herein as the Institute) was created as a means to provide early-career 4-H Extension educators—those having 5 years' experience or less—with an intense, competency-based, face-to-face training to strengthen Extension support for the community club program; introduce key competencies; model a variety of presentation, teaching, and evaluation methods; and provide an opportunity for networking among peers throughout the region.

Relevant Literature

Professional Development for Positive Youth Development

Professional development programs in Ohio, Louisiana, California, North Carolina, and Texas are examples of other states' offerings of competency-based professional development systems in Extension (Conklin, Hook, Kelbaugh, & Nieto, 2002; Fox, Sasser, & Arcemont, 2013; Heck, Subramaniam, & Carlos, 2009; Lakai et al., 2012; Stone & Coppernoll, 2004). This trend is not unique to Extension. Other youth development organizations are finding value in competency-based training as well (Diem, 2009). National organizations, including the American Camp Association, the National Collaboration for Youth, Boys & Girls Clubs of America, Girl Scouts, and the National Afterschool Association, also have developed competency models for youth workers (Garst, Baughman, & Franz, 2014).

A 4-H that interconnects the concept of positive youth development (Lerner & Lerner, 2013) with more diverse content opportunities has replaced the technical, subject matter–focused 4-H of the past (e.g., animal science, family and consumer science). The body of knowledge related to positive youth development continues to evolve (Fox et al., 2013), strongly influencing contemporary professional development agendas.

The 4-H youth development profession continues to be on the cutting edge of professional development. The 4-H professional research and knowledge base was the foundation for the 4-H youth development profession from 1985 until 2004 (Stone & Rennekamp, 2004). In 2004, an updated conceptualization, referred to as 4-H professional research, knowledge, and competencies (4-H PRKC), was unveiled as the road map for the 4-H workforce of the future (Stone & Rennekamp, 2004). Competencies in this model are described as the proficiencies, specific skills, and best practices needed to be an effective 4-H youth educator.

The 4-H PRKC model is an important resource for individuals just entering the 4-H workforce and those designing trainings for 4-H educators. 4-H PRKC involves four levels: domain, topic, component, and competency. This structure connects the dots between the essential subject matter necessary in the field of youth development and how to prepare and develop people to be effective in the field. At the domain level are the following six areas:

• youth development;
• youth program development;
• volunteerism;
• equity, access, and opportunity;
• partnerships; and
• organizational systems (Stone & Rennekamp, 2004).

Program Evaluation in Extension

Generally, those working in Extension are proficient at evaluating outputs (e.g., audience demographics and participation numbers) and immediate participant reactions. More effort is required to evaluate the outcomes, or the impact of a program, or to measure short-, medium-, and long-term results (Taylor-Powell, Steele, & Douglah, 1996). Workman and Scheer (2012) conducted a study to explore impact-level evidence collected through program evaluation by Extension personnel, as published in the Journal of Extension. Their study revealed that almost two thirds of the pertinent articles documented program outcomes. However, only 5.6% documented long-term outcomes, suggesting a need for more longitudinal research of Extension programs.

The Western Region 4-H Institute—A Competency-Based Approach

The Western Region 4-H Institute is a 5-day training program sponsored by the Western States 4-H Program Leaders. The Institute was first offered in 2002 to advance early-career 4-H youth development professionals' skill sets and abilities in developing and delivering 4-H club–based programming (Astroth & Lindstrom, 2008). The Institute is conducted every 3 or 4 years. A planning and teaching team (referred to herein as the planning team) plans and conducts each cycle of the Institute. The planning team comprises experienced 4-H professionals, the majority of whom have county-level field assignments. Special invited presenters augment the diversity of ideas and perspectives at the Institute.

The planning team uses 4-H PRKC as the primary driver of content selection for the Institute. The series of topics necessary for professional growth of 4-H educators, as elucidated by Rennekamp and Nall (1994), provides a general context. The topics include (a) developing a deeper understanding of the 4-H organizational structure and culture, (b) growing beyond base-level technical skills, (c) amplifying relevancy to previous training and education, (d) exercising creativity and initiative, (e) becoming more autonomous, (f) exploring personal/professional dynamics, and (g) building relationships with peers.

The emphasis of the Institute is that of a competency-based approach. The Institute remains a dynamic solution to the need for professional development for 4-H professionals in the Western Region, evolving with each cycle. That said, the mere size of the Western Region creates a challenge related to bringing people together for professional development. When coupled with the challenges of diminishing budget allocations (Astroth & Lindstrom, 2008), the Western Region leadership has found the 3-year delivery cycle to be practical and effective.

Institute Objectives

The objectives of the Western Region 4-H Institute are

1. to strengthen Extension support for the community club program in the Western Region,
2. to introduce early-career Western Region 4-H professionals to key elements of the 4-H PRKC model,
3. to model a variety of presentation and teaching methods and tools,
4. to model a variety of evaluation methods, and
5. to provide an opportunity for early-career 4-H professionals to network with peers throughout the region.

Need and Purpose: A More Comprehensive Evaluation for the 2012 Institute

Institute participants' experiences and related outcomes have been explored through the use of a variety of evaluative techniques. Evaluations have ranged from descriptive reports (Astroth, 2002) to retrospective "post-then-pre" surveys based solely on 4-H PRKC assessment indicators (unpublished reports by previous planning teams). Also, Astroth and Lindstrom (2008) conducted a detailed follow-up of participants of the first Institute, in 2002, on the topic of "a viable 4-H workforce" (p. 8). That study considered longer-term aspects related to the 4-H workforce, including the factors of turnover and retention, skill development, job satisfaction, and organizational loyalty.

The research reported here—the evaluation of the 2012 Institute—was timely and needed given the barriers to face-to-face, competency-based training opportunities and the lack of longitudinal research of Extension programs. The evaluation subcommittee for the 2012 Western Region 4-H Institute chose to employ evaluation techniques that would assess short- and medium-term outcomes, integrating previous Institute evaluation approaches into a more focused and comprehensive evaluation for 2012. The focus on 4-H PRKC was narrowed from the full set of nearly 300 competencies to a more defined subset of 68 competencies, which guided all programmatic and workshop content choices.

The 2012 Institute

The 2012 Institute was the fourth to be held and included approximately 36 hr of formal professional development, involving a wide variety of teaching styles and methods. The planning team identified the 4-H PRKC domains of youth program development, volunteerism, and partnerships as the primary professional development topics for 2012. The domains of youth development and organizational systems were included as well but received less emphasis (i.e., less session time). The domain of equity, access, and opportunity was not identified as a priority for the 2012 Institute. The topics "vibrant 4-H clubs" and "grant writing" were included, and expert practitioners and planning team members presented on effective use of social media and video production.

The 2012 planning team divided into subteams, each of which focused on one or more of the domains for 1 full day of the Institute. The subteams identified specific competencies in each domain to teach and evaluate and then chose complementary teaching and evaluation methods. Comparison of methods across all planned workshops assured that a wide variety of teaching and engagement approaches were modeled.

Evaluation Design and Methods

The 2012 Western Region 4-H Institute evaluation subcommittee—represented by the authors of this article—was tasked with the design of the evaluation and the collection and analysis of the evaluation results. We chose a two-part approach to gathering data. The first used open-ended and retrospective post-then-pre (Colosi & Dunifon, 2006; Schaaf, Klatt, Boyd, & Taylor-Powell, 2005) scaled questions to measure participants' knowledge growth and opinions of the elements of the Institute and to solicit recommendations for future cycles of the Institute. For the scaled questions, we used a 5-point rating scale: 1 = no knowledge; 2 = limited knowledge; 3 = know and use; 4 = know, use, and explain; and 5 = know, use, and teach. All questions were referenced to the Institute in general, specific workshops, and the aspects of 4-H PRKC that were addressed (i.e., the subset of 68 competencies). The data were gathered electronically after the conclusion of the Institute.

A follow-up conducted approximately 12 months after the Institute concluded comprised the second part of the data gathering. We identified key elements (topic areas) of the program and asked respondents to rate their relative values. The key elements were emphasis on 4-H PRKC, video experience (a separate, open-ended evening activity interconnected to Institute content and lessons learned), youth program development, youth development, capacity and partnerships, grant writing, volunteerism, a "Revolution of Responsibility" workshop, networking, and the overall Institute experience. Respondents rated the relative value of the elements by using a 5-point scale having the following anchors and midpoint: 1 = limited value, 3 = valuable, and 5 = great value. Reponses to similar open-ended questions asked immediately after the Institute concluded and 12 months after the Institute were coded and compared on the basis of five categories. Each category was referenced to one of the five 4-H PRKC domains targeted a priori by the planning team. Comments that did not fit into the established framework were identified as "not categorized." Networking emerged from the comments and open-ended responses as a sixth category. Responses to an additional open-ended question were coded (Ryan & Bernard, 2000) through the use of a set of key concepts agreed on by the evaluation subcommittee.

All data were collected through the use of Survey Monkey.

Results and Outcomes

For the first part of the evaluation, conducted at the conclusion of the Institute, 84% of the population (n = 58, N = 69) responded to the retrospective post-then-pre statements referenced to 4-H PRKC. For the second part of the evaluation, approximately 64% of the pool of those still having the same email address (i.e., 42/66 having active email accounts 1 year later) responded to the 12-month follow-up.

Post-Then-Pre Results for 4-H PRKC

An analysis of the retrospective post-then-pre responses indicated that participants demonstrated consistent positive changes across all categories of the 68 specific 4-H PRKC competencies identified as teaching/learning priorities for 2012. Those positive changes ranged from .67 to 1.14 for the 68 competencies. Using the 5-point scale and consolidating the data by 4-H PRKC domain, the results demonstrated positive changes of

• .98 for volunteerism,
• .97 for partnerships,
• .82 for youth development,
• .78 for youth program development, and
• .77 for organizational systems.

To further illustrate and highlight the positive changes described above, we combined the responses connoting the greatest understanding and confidence—that is, ratings of 4 (know, use, and explain) and 5 (know, use, and teach)—for each 4-H PRKC domain. We then calculated the percentage of respondents who retrospectively reported having this greatest level of understanding and confidence preprogram and postprogram for each domain relative to the total number of responses for the domain (Figure 1). The percent change preprogram to postprogram ranged from 32% (for the domain of organizational systems) to 41% (for the domain of partnerships), indicating consistent and positive short-term gains in participants' self-reported understanding of and confidence in the abilities to know, use, explain, and teach.

Figure 1.
Percentage Comparison of Retrospective Post-Then-Pre Data Combining Levels 4 and 5 on a 5-Point Likert Scale (n = 58)

Twelve-Month Follow-Up

As mentioned previously, as part of the 12-month follow-up, Institute participants rated the relative value of key elements of the Institute, using a 5-point scale. Thirty-nine of the 42 individuals responding to the follow-up completed the ratings. Figure 2 summarizes the average ratings of the key elements. The only element that had an average rating of less than 3 was the video experience. One possible explanation is that the average young professional's technical skills are more advanced than the planning team members assumed. Every other area received acceptable to very strong recognition related to value. According to those responding, networking was a critical element: It was the element of the Institute rated most highly by participants 12 months after they had experienced the program. It is important to note that the overall 4-H Institute experience was the second most highly rated element, suggesting that the whole experience was greater than the sum of its parts.

Figure 2.
12-Month Follow-Up Summary of Average Response Ratings by Participants on the Key Elements of the Western Region 4-H Institute, Using Predetermined Categories (n = 39)

Another area of the follow-up solicited authentic (i.e., open-ended) responses to the question ". . . how did the Institute help you strengthen your county level program?" On analysis of the responses (n = 35), several keywords/key concepts emerged. We coded all responses according to the concepts of

• confidence and renewal,
• networking,
• club/program improvement, and
• youth and volunteers.

Data are summarized in Figure 3. Among the statements, 19 responses were specific to one concept, whereas the other 16 were richer and more complex, referencing two or three of the concepts. Participants independently reidentified the relative value of core areas of the Institute knowledge base a posteriori (e.g., youth, volunteers, and program improvement) and concluded that their confidence in the ability to be successful had grown (i.e., confidence and renewal). For example, Institute participants noted that they "[experienced] renewal," "[were more] confident," and "[finally had the] confidence" to grow their programs, empower their youths and volunteers, and break into new valuable program areas. Among the responses from the 35 responding participants, three were categorized as "neutral or nonpositive." Relative to those three responses, one individual did not have a good experience, another felt advanced enough that the Institute was "mainly review," and a third felt overwhelmed by the intensity/volume of information and experiences.

Figure 3.
Content of the Institute Related to the Value for Strengthening Local Programs, Measured by Individual Responses (n = 35)

Comparison of Immediate Post-Then-Pre Responses and 12-Month Follow-Up Responses

We compared responses to two similarly worded questions from the data gathered at the conclusion of the Institute and as part of the 12-month follow-up for insight on the relative value of key elements (see Figure 4). The question asked immediately after the Institute was "Which elements of the Institute were most helpful to you . . . ?" The similar question asked in the follow-up was "Please list three of the most memorable/valuable workshops from your perspective. . . ." Fifty-six of the 63 participants who responded to the post-then-pre survey at the conclusion of the Institute answered the question. Thirty-three of the 42 respondents to the follow-up survey answered the question. Most notable are (a) the increase in participants' value of networking expressed in the follow-up survey and (b) the consistently high rankings of youth development and youth programs in both the post-then-pre survey and the follow-up survey.

Figure 4.
Comparisons of Identified Valuable Elements at the Conclusion of the Institute and 12 Months After the Institute

Discussion

Using a competency-based approach for professional development is highly effective. The identified competencies help focus the planning process, allowing the training to be delivered topically, and provide a consistent referent and order for participants. Additionally, a competency-based approach provides a basis for program evaluation to document both immediate and longer term outcomes. For example, the greatest emphases on elements of the 2012 Institute revolved around the 4-H PRKC domains of youth development and youth program development. It is encouraging, from a process evaluation perspective, that the most valuable elements identified by the participants of the 2012 Institute revolved around youth.

Impact is indicated on two levels. The comparison of evaluation results over time reinforces the notion that a strong theoretical base is influential on participant knowledge and retention. The participants' recognition of the personal and professional value of confidence and renewal and networking in the follow-up data suggests intrapersonal changes that can drive changes and improvements in practice. That is, participants independently verified that they were tapping their newly discovered confidence and renewal. They reported using their newly established networks of colleagues to implement the concepts and practices learned at the Institute for local program improvement.

A successful professional development program, such as the Western Region 4-H Institute, reaches a population beyond just those who attend. Each participant returned to his or her assignment with a broader conceptual base, an improved tool kit for facilitating learning, and a feeling of empowerment for moving forward with confidence. When the Institute participant's home audience (i.e., those affected by the work of the Institute participant within his or her assignment) is factored in, the Institute provided a benefit to—as reported by the 42 individuals who completed the 12-month follow-up—nearly 4,000 volunteers and over 16,000 youths in the Western Region.

Conclusions and Implications

We felt it was important to implement a longitudinal evaluation to reach beyond documentation of only immediate program effectiveness, which was shown consistently in previous iterations of the program (i.e., Astroth & Lindstrom, 2008). As suggested by Workman and Scheer (2012), it is crucial that Extension professionals move beyond measuring knowledge and skills gained and seek to measure substantial and true impact. The results of the study reported here document the value-laden outcome when program participants move beyond just improved knowledge and skills to a clearer focus on priorities and best practices applied to effect change at the local level. Identifying the Institute's impact on and personal value to the early-career professionals who attend is requisite for ensuring continuous program improvement and justifying continued investment by the regional leadership into this collaborative professional development initiative.

The 2012 Western Region 4-H Institute evaluation subcommittee was the first to set a goal of including a longitudinal assessment of the first objective of the Institute, which is to strengthen Extension support for the community club program in the Western Region. We believe that building a strong network of confident Extension professionals is an indicator of strengthened program support. Evaluation of future Western Region 4-H Institutes should continue to include longitudinal surveys to further measure the breadth and depth of long-term program support. Eventually, a large-scale evaluation of all past Institute participants, back to the 2002 session, may provide evidence of significant culture change resulting from program participants' Institute experiences. However, the varied methodologies and evaluative goals of the different program evaluation teams across cycles of the Institute limit the potential for comparisons. A more opportune practical extension of the study reported here might best be based on a follow-up to Astroth and Lindstrom's (2008) study on sustaining a viable 4-H workforce.

The Western Region 4-H Institute itself represents culture change in our system, as Extension searches for effective and parsimonious methods of reaching new audiences with expert human assets. Investing resources to bring professionals together from a large geographic area strengthens the network of Western Region 4-H professionals and is economically practical. The planning team model of each cycle of the Institute, which involves retaining at least 50% of the members for the next cycle, is a cultural value that ensures program consistency. Perhaps most important, enculturating early-career professionals by familiarizing them with a framework and experience that advances their abilities to employ best practices deepens their understanding of priorities, with the shared value having impact at multiple levels across the region.

Acknowledgments

The authors wish to express their appreciation to the Journal of Extension reviewers for their thoughtful and valuable comments and to P. Skelton of New Mexico State University for his constructive suggestions on a later iteration of the manuscript. We also wish to recognize and thank our colleagues on the 2012 Western Region 4-H Institute planning team, including D. Leidheiser (chair), J. Bunnell (cochair), A. Armajo, L. Hecimovich, C. Stone, and S. Vanderwey.

References

Astroth, K. A. (2002). Western 4-H Institute: Preparing the 4-H professional for success. Evaluation report I. Retrieved from ERIC database. (ED 469 010).

Astroth, K. A., & Lindstrom, J. (2008) Investing in professional development: Building and sustaining a viable 4-H youth workforce for the future. Journal of Youth Development, 3(2). Retrieved from http://www.nae4ha.com/assets/documents/JYD_080302final.pdf

Brodeur, C., Higgins, C., Galindo-Gonzalez, S., Craig, D., & Haile, T. (2011). Designing a competency-based new county Extension personnel training program: A novel approach. Journal of Extension, 49(3) Article 3FEA2. Available at: https://www.joe.org/joe/2011june/a2.php

Colosi, L., & Dunifon, R. (2006). What's the difference? "Post then pre" & "pre then post." Ithaca, NY: Cornell University Cooperative Extension.

Conklin, N. L., Hook, L. L., Kelbaugh, B. J., & Nieto, R. D. (2002). Examining a professional development system: A comprehensive needs assessment approach. Journal of Extension, 40(5) Article 5FEA1. Available at: https://www.joe.org/joe/2002october/a1.php

Diem, K. G. (2009). Preparing youth development professionals to be successful: How do the needs of Extension/4-H compare to those of other organizations? Journal of Extension, 47(1) Article 1RIB1. Available at: https://www.joe.org/joe/2009february/rb1.php

Fox, J. E., Sasser, D., & Arcemont, L. (2013). 4-H youth development professionals' perceptions of youth development core competence. Journal of Human Sciences and Extension, 1(1). Retrieved from http://media.wix.com/ugd//c8fe6e_38247af22331edc7c56098ece61d3b52.pdf

Garst, B., Baughman, S., & Franz, N. (2014). Benchmarking professional development practices across youth-serving organizations: Implications for Extension. Journal of Extension, 52(5) Article 5FEA2. Available at: https://www.joe.org/joe/2014october/a2.php

Heck, K., Subramaniam, A., & Carlos, R. (2009). Use of the PRKC tool in assessment of staff development needs: Experiences from California. Journal of Extension, 47(3) Article 3FEA7. Available at: https://www.joe.org/joe/2009june/a7.php

Lakai, D., Jayaratne, K., Moore, G., & Kistler, M. (2012). Barriers and effective educational strategies to develop Extension agents' professional competencies. Journal of Extension, 50(4) Article 4RIB1. Available at: https://www.joe.org/joe/2012august/rb1.php

Lerner, R. M., Lerner, J. V., & colleagues. (2013). The positive development of youth: Comprehensive findings from the 4-H Study of Positive Youth Development. Chevy Chase, MD: National 4-H Council. Retrieved from http://www.4-h.org/about/youth-development-research/positive-youth-development-study/

Rennekamp, R. A., & Nall, M. A. (1994). Growing through the stages: A new look at professional growth. Journal of Extension, 32(1) Article 1FEA2. Available at: https://www.joe.org/joe/1994june/a2.php

Ryan, G. W., & Bernard, H. R. (2000). Data management and analysis methods. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed.). Thousand Oaks, CA: Sage.

Schaaf, J., Klatt, J., Boyd, H., & Taylor-Powell, E. (2005). Analysis of retrospective post-then-pre data, Quick Tips #30. Madison, WI: Program Development and Evaluation, University of Wisconsin-Extension. Available in PDF format at http://www.uwex.edu/ces/pdande/resources/index.html or by contacting pdande.webmaster@ces.uwex.

Stone, B., & Coppernoll, S. (2004). You, Extension and success: A competency-based professional development system. Journal of Extension, 42(2) Article 2IAW1. Available at: https://www.joe.org/joe/2004april/iw1.php

Stone, B., & Rennekamp, R. (2004). New foundations for the 4-H youth development profession: 4-H Professional Research, Knowledge, and Competencies Study, 2004. Conducted in cooperation with the National 4-H Professional Development Task Force. National 4-H Headquarters, Cooperative State Research, Education, and Extension Service, U.S. Department of Agriculture.

Taylor-Powell, E., Steele, S., & Douglah, M. (1996). Planning a program evaluation. Madison, WI: Program Development and Evaluation, University of Wisconsin-Extension. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Workman, J. D., & Scheer, S. D. (2012). Evidence of impact: Examination of evaluation studies published in the Journal of Extension. Journal of Extension, 50(2) Article 2FEA1. Available at: https://www.joe.org/joe/2012april/a1.php