Evaluating for Impact: Professional Development Educational Content Delivery Through Learning Communities

Introduction

Recent increases in accountability and impact reporting have presented challenges for busy 4-H educators. In response to the increasing pressure for demonstrating programmatic results, Extension Services have begun to work towards increasing evaluation skills of their educators (Guion, Boyd, & Rennekamp, 2007). The National 4-H Learning Priorities team recently developed a professional development teaching and learning resource to build evaluation capacity among 4-H educators (Arnold et al., 2008). The resource includes seven modules; program planning for effective evaluations, focusing an evaluation, evaluation design, evaluation methods, collecting and handling data, analyzing and interpreting data, and communicating evaluation results. Each module includes competencies for "novice," "advanced beginner," and "practitioner" level evaluators and is designed for flexible delivery.

Learning Communities as Professional Development Delivery Modes

Three evaluators were recruited to pilot the National 4-H Learning Priorities: Evaluating for Impact Educational Content for Professional Development via learning communities. Extension specialists with evaluation expertise from Oregon, Vermont, and Virginia were selected for the pilot. Each state developed its own learning community tailored to needs of the participants. An advisory board of experts in curriculum development and Extension evaluation helped guide and evaluate the learning community process. For the purposes of the project, learning communities were defined as a group of people sharing common values and beliefs actively engaging in learning from each other.

Oregon Learning Community

The Oregon 4-H Youth Development program recruited 16 4-H educators to participate in an online learning community via Microsoft SharePoint, over a 10-month period. The project leader recruited educators to participate via email and subsequently led monthly sessions via conference call and Adobe Connect Web conferencing. Participants were given assignments to complete monthly and were encouraged to be involved through active participation in discussion boards and sharing documents and evaluation plans on the SharePoint site. In practice, the project leader found that participation was more static and task-oriented than expected, thus monthly contacts were initiated via conference call or Web conference.

The Oregon learning community completed each module of the curriculum and found it important to focus on basic and immediately relevant resources provided in the curriculum. Participants tended to be newer educators who appreciated the opportunity to learn more about evaluation. The learning community leader felt the pilot project successfully improved organizational evaluation capacity by increasing awareness of organizational evaluation expectations and contributing to the overall culture of appreciating and valuing program evaluation. The learning community format allowed for the practice and application of specific skills, although participants' time constraints limited their ability to do the activities. Oregon plans to continue to use the educational content in a slightly altered delivery format.

Vermont Learning Community

Vermont Cooperative Extension recruited 10 4-H Educators to participate in the learning community over a 6-month period. The learning community met monthly via conference call in addition to intensive one-on-one mentoring between the leader and community members. Each member worked independently on his or her learning action plan, using the educational content to gain knowledge and theory needed to achieve their action plan goal.

Participants recognized the importance of having a mentor with evaluation expertise to help them build skills and work on individual goals. They also expressed appreciation for the shared community for the opportunity to share personal struggles and gather feedback and ideas on new approaches to their evaluation challenges. Vermont plans to expand use of the content to build evaluation capacity with Extension professionals in other disciplines.

Virginia Learning Community

Virginia Cooperative Extension recruited 10 4-H professionals to participate in the learning community via phone conference over a 2-month period. Participants used the curriculum as a starting point for exploring specific evaluation topics and as a framework for building organizational evaluation capacity. Participants tended to be experienced 4-H educators with varying roles from agent to specialist to district administrator.

The Virginia participants reported positively on the curriculum as a framework for exploring different evaluation topics and resources. They engaged actively in the learning community and often shared conversations outside the organized conference calls. Most participants reported that they enjoyed the learning community and would participate in the future in a short-term format. Participants expressed some concern over the "academic" nature of curriculum and found it helpful to have a facilitator with extensive evaluation experience to help "de-mystify" some of the language and processes. As a result of the learning community experience, Virginia 4-H has created a State 4-H Evaluation plan of work.

Conclusions

The three pilot projects encountered successes and challenges using the learning community format for delivering evaluation professional development to 4-H educators. The learning communities were able to customize the Evaluating for Impact content to the needs of their learners and plans to continue using the resource. The learning community delivery format worked better for some states than others. Several themes emerged from the project and are summarized below.

• Evaluating for Impact curriculum was, overall, a good framework for states to begin work on increasing individual evaluation skills as well as organizational evaluation capacity.


• Evaluating for Impact curriculum can be delivered successfully in a learning community format with an experienced leader/facilitator.


• Learning communities can be "individualized" to meet the needs of the organization and its learners.


• Evaluating for Impact is comprehensive but may be challenging for 4-H educators with little evaluation experience.


• The evaluation and youth development expertise of the learning community facilitator is a key factor for success.


• An organizational evaluation culture that encourages evaluation for both program improvement and accountability was another factor influencing the success of the pilot project.


• Learning communities face challenges similar to other professional development formats, including educator time constraints and the technological capacity of the organization.

References

Arnold, M. E., Calvert, M. C., Cater, M. D., Evans, W., LeMenestrel, S., Silliman, B., & Walahoski, J.S. (2008). Evaluating for impact: Educational content for professional development. Washington, D.C.: National 4-H Learning Priorities Project, Cooperative State Research, Education & Extension Service, USDA.

Guion, L., Boyd, H. H., & Rennekamp, R. A. (2007). An exploratory profile of Extension evaluation professionals. Journal of Extension [On-line], 45(4). Article 4FEA5 Available at: http://www.joe.org/joe/2007august/a5.php