Evaluation of the Effectiveness of the Oregon Master Beekeeper Program

Introduction

The Oregon Master Beekeeper Program (OMBP) is a training and service program for beekeepers of all experience levels. A collaborative effort between Oregon State University and the Oregon State Beekeepers Association, the program is focused on goals that contribute to the health of honey bee colonies, integrity of the practice of beekeeping throughout the region, and awareness of pollinator health among the public. Since the OMBP started in 2012, we have enrolled 1,360 students in the apprentice level (beginning) training, 186 students in the journey level (advanced) training, and 14 in the master level training.

Certification at the apprentice level requires participants to attend classes, take an exam, keep hive records, and meet with a volunteer mentor at least once per season for hands-on training. OMBP volunteer instructors provide classroom training to apprentice level students. Instructors are provided with a series of eight 2-hr PowerPoint presentations, which they may edit according to their teaching style. Classes are held in 10 locations in Oregon and one in Idaho. The hands-on training is a unique aspect of the OMBP (Breece & Sagili, 2015). It was included as a component in the apprentice level training to increase the quality of education, as hands-on training has been found to be an effective teaching strategy in other Extension training events (Kane, 2002; Strong, Harder, & Carter, 2010). Upon apprentice-level certification, a participant may enroll in the journey level, and ultimately the master level, for advanced training. Much of the training at these levels is self-paced. Journey students learn by completing "guided studies," or worksheets requiring them to find answers using multiple resources. Master students learn by completing literature reviews and developing research projects. Journey and master students may attend special events for hands-on training in advanced beekeeping topics. Both levels include a community service component, in which students earn points by sharing beekeeping knowledge with others through teaching, writing, or developing projects.

The OMBP trains beekeepers through the application of approaches that allow for a variety of learning methods, thereby appealing to an audience diverse in backgrounds, life stages, learning styles, and areas of residence. This concentration on varied learning mechanisms could be applied in other Extension programs that provide long-term training in complex endeavors to similarly diverse audiences.

To assess the efficacy of the OMBP in training beekeepers, we distributed an online survey to all participants involved in the training from 2012 to 2015. We based the survey on the work of Swackhamer and Kiernan (2005), in which they developed a method of program evaluation for multiple-topic master gardener training. They found that many opinion-based Extension program surveys produce qualitative and anecdotal responses. Their proposed survey method encourages the collection of (a) a quantifiable measure of learning for each topic, (b) before-and-after data on confidence levels to ascertain a change resulting from training, and (c) qualitative feedback via a section for comments. The quantifiable information can be summarized to address program accountability and need for improvement.

Methods

In May 2016, we used online Qualtrics software to survey past participants (classes of 2012–2015) on their experience with the OMBP. We adopted several survey questions suggested by Swackhamer and Kiernan (2005). Our questions were designed to elicit information on participants' knowledge, colony management practices, and confidence levels in particular areas before and after training. We also provided adequate space in the survey for participant feedback as a means for identifying areas in which we could improve the program. Because we surveyed participants after training, we asked them to estimate their knowledge, behaviors, and confidence levels before training. Some participants had beekeeping experience prior to OMBP training. We asked this group specific questions to determine whether OMBP training had changed their beekeeping skills or behaviors.

The survey included scaled answer options for questions on specific topics. For example, when asked about how much they learned in classes covering honey bee biology, pests and pathogens, products of the hive, and seasonal colony management, respondents could answer "nothing new," "some new knowledge," "a lot," or "a great deal!" The ordinal responses may have differed from person to person; however, each answer beyond "nothing new" indicated an increase in knowledge gained. When asked about their confidence in hive management skills such as identifying common pests and diseases, respondents could answer "not confident," "somewhat confident," "confident," "very confident," or "extremely confident." Each answer beyond "not confident" indicated an increase in confidence gained.

Findings

Effectiveness of Classroom Training in Teaching Beekeeping Topics

Survey respondents reported that they learned the most in classes covering the following topics: pests and pathogens, seasonal colony management, and honey bee biology. The percentages of OMBP students who learned a lot or a great deal in all topics were substantial. For example, 85% of respondents learned a lot or a great deal about pests and pathogens. The topic for which students learned the least amount was products of the hive. Only 45% reported learning a lot or a great deal about this topic. Data are shown in Figure 1.

Figure 1.
Percentages of Survey Respondents (n = 137) Who Learned "A Lot" or "A Great Deal" About Class Topics

Changes in Colony Management After OMBP Training

We asked beekeepers with over 2 years of experience about changes in their colony management practices after OMBP training and the results of those changes. Honey production increased for 43% of the respondents. Many participants reported increasing efforts involved in swarm management (47%) and increasing apiary size by splitting existing colonies (47%). Most notably, 66% of these experienced beekeepers increased their Varroa mite monitoring efforts after OMBP training.

Changes in Confidence Regarding Beekeeping Skills After OMBP Training

Participants were asked to rate their confidence in their beekeeping skills by choosing from the following options: not confident, somewhat confident, confident, very confident, extremely confident. When asked to consider what their confidence levels had been prior to enrollment in the OMBP, substantial percentages of respondents reported feeling not confident about performing basic hive evaluations (52%), sampling for Varroa mites (68%), recognizing common honey bee diseases (72%), and recognizing starvation (60%). After OMBP training, only 5% or fewer respondents reported feeling not confident about these skills. Large majorities of respondents reported increases in confidence by at least one level regarding performing basic hive evaluation (Figure 2), sampling for Varroa mites (Figure 3), and recognizing common honey bee diseases (Figure 4).

Figure 2.
Changes in Confidence in Performing a Basic Hive Evaluation After Oregon Master Beekeeper Program Training (n = 124)

Figure 3.
Changes in Confidence in Sampling for Varroa Mites After Oregon Master Beekeeper Program Training (n = 122)

Figure 4.
Changes in Confidence in Recognizing Common Bee Diseases After Oregon Master Beekeeper Program Training (n = 122)

Effects of OMBP Training on Social Skills and Community Involvement

General confidence or self-esteem increased in 43% of the participants. Most respondents (70%) either began helping others solve beekeeping problems or increased their efforts in helping others. Many participants also experienced improvements in public speaking (30%), being willing to accept challenges (50%), taking on new leadership roles (36%), and feeling that they could make a difference in their communities (48%).

Participant Comments

We requested that participants include feedback for program improvement. Of the comments received, we judged 78% as positive. Many respondents commented on the mentoring aspect of the program and how beneficial it was for their beekeeping education. The following statements are representative of such comments:

• "I had a great mentor and that made all the difference in my continuing my beekeeper education. My mentor continues to be a great resource."
• "I probably wouldn't have my bees today if I didn't have a mentor years ago."
• "This was by far the most impactful portion of the program. My mentor was and still is an incredible support in patiently handling all of my questions and challenging me to critically think about management tools instead of giving me the answer."
• "Mentoring allowed for increased confidence in working with a colony of honeybees for the first time. Mentoring also allowed for easier identification of varying stages of hive activity/development through one-on-one access to the mentor, whether by phone or in person at your own hive/colony. The interaction between me and my mentor helped me tremendously throughout the year as I was learning what to do in each season."

The mentoring component of the program had a positive impact on most participants, although we also received some negative feedback regarding mentors (22% of comments received). The most common complaint was difficulty in scheduling meetings with mentors, as represented by this comment:

• "It's very difficult to get together with my mentor. He has not been out to see my bees. He's mainly available if you go to his house on Tuesday mornings. Because I work, I'm not able to make it. Trying to take a day off so I can meet with him."

Discussion

Survey Limitation

We received fewer completed surveys than expected. The survey was sent to over 621 participants of the OMBP, and we received 151 responses, even after a follow-up email. Therefore, the response rate was 24%. Although low response rates are common for web-based surveys (Monroe & Adams, 2012), this is well below the expected web-based survey response rate of 51% (Archer, 2008). We recognize that this low response rate may have resulted in findings not representative of the majority of participants in the program. Our plans to increase the response rate for future surveys include using innovative methods of surveying (Monroe & Adams, 2012), surveying within 4 months of the conclusion of training, and making additional contacts with participants.

Areas of Success

Participants showed an improvement in knowledge of many beekeeping topics, especially topics related to pests and pathogens. The Varroa mite is the most damaging pest of honey bees (Francis, Nielsen, & Kryger, 2013), and beekeepers cite Varroa mite infestation as one of the primary causes for colony loss (Seitz et al., 2016). Education on Varroa mite biology, symptoms of infestation, and treatment options is critical for beekeeping success, as is regular monitoring for Varroa mite levels. From our survey, we learned that our educational techniques were effective, as participants' levels of knowledge on Varroa as well as their Varroa monitoring efforts increased.

Participants also showed a marked increase in confidence in several beekeeping skills. Confidence in these skills may reduce beekeeper attrition and may encourage beekeepers to help others in the future (Swackhamer & Kiernan, 2005). Because community involvement is an important component in the advanced levels of the OMBP, participant confidence in beekeeping skills and topics is important and will help OMBP participants effectively convey information to others in their communities.

Areas for Improvement

The most common complaint was difficulty in scheduling meetings with mentors. This has been the top complaint since the inception of the OMBP. In an attempt to address the issue of scheduling difficulties, we now ask more questions about availability in the student and mentor applications. We provide an opportunity for mentors and students to meet at program orientation, and we suggest that they pencil in a few meeting dates for the year ahead. Further, we have recruited volunteer "regional representatives" to assist mentors and students with communication.

Conclusion

Our survey suggests that master beekeeper training benefited Oregon and Idaho participants by improving their beekeeping knowledge, management skills, confidence, and community involvement. The combination of classroom instruction and hands-on training with volunteer mentors has provided a quality experience for new beekeepers. We plan to continuously improve the training program on the basis of the survey results and comments provided by the respondents.

Evaluation of Extension training programs is necessary for identifying areas of success and areas for improvement. Applying a collection of quantifiable measures of learning including before-and-after data—as we did—can allow Extension personnel to measure a change after training. Other Extension program evaluators may experience a better response rate and more meaningful responses if participants are surveyed without delay through simple and effective survey software such as Qualtrics.

Like many Extension "master" programs, the OMBP is built on community involvement of trained participants. Diverse audiences in other master beekeeper programs or similar educational programs could benefit from a multifaceted approach to training, as we provide in the OMBP. Through hands-on training and quality classroom instruction, program participants may gain knowledge and confidence and become well-equipped to make meaningful contributions to their communities.

Acknowledgments

We thank the participants of the OMBP for completing our survey. We especially thank the volunteers of the OMBP. The program is successful entirely because of our volunteers and their countless hours of mentoring and instructing new beekeepers.

References

Archer, T. M. (2008). Response rates to expect from web-based surveys and what to do about it. Journal of Extension, 46(3), Article 3RIB3. Available at: https://joe.org/joe/2008june/rb3.php

Breece, C. R., & Sagili, R. R. (2015). Hands-on training emphasized in the Oregon Master Beekeeper Program. Journal of Extension, 53(3), Article 3IAW6. Available at: https://www.joe.org/joe/2015june/iw6.php

Francis, R. M., Nielsen, S. L., & Kryger, P. (2013). Varroa-virus interaction in collapsing honey bee colonies. PLOS ONE, 8(3), e57540.

Kane, P. N. (2002). A biosolids technician training course with a "hands on" team approach using professionals from the field. Journal of Extension, 40(2), Article 2IAW4. Available at: http://www.joe.org/joe/2002april/iw4.php

Monroe, M., & Adams, D. (2012) Increasing response rates to web-based surveys. Journal of Extension, 50(6), Article 6TOT7. Available at: https://www.joe.org/joe/2012december/tt7.php

Seitz, N., Traynor, K. S., Steinhauer, N., Rennich, K., Wilson, M. E., Ellis, J. D., . . . vanEngelsdorp, D. (2016). A national survey of managed honey bee 2014–2015 annual colony losses in the USA. Journal of Apicultural Research, 54(4), 292–304. doi:10.1080/00218839.2016.1153294.

Strong, R., Harder, A., & Carter, H. (2010). Agricultural Extension agents' perceptions of effective teaching strategies for adult learners in the master beef producer program. Journal of Extension, 48(3), Article 3RIB2. Available at: http://www.joe.org/joe/2010june/rb2.php

Swackhamer, E., & Kiernan, N. E. (2005). A multipurpose evaluation strategy for master gardener training programs. Journal of Extension, 43(6), Article 6FEA4. Available at: https://www.joe.org/joe/2005december/a4.php