Using an Innovative Multiple-Methods Approach to Evaluate Extension Conferences

Introduction

Extension professionals invest in evaluation planning to ensure that programs are achieving desired goals and objectives, thus making the best use of diminishing resources and limited time (Galloway, Peterson, & Dalton, 2006). Extension administrators promote the need to articulate impacts and encourage staff to identify public value talking points and stories resulting from successful programs (Franz, Arnold, & Baughman, 2014). As planners of a multistate 4-H volunteer conference, we needed an evaluation strategy that stretched beyond conference satisfaction to examine learning and communicate impact. As the longest running regional 4-H volunteer conference in the nation, the Southern Region's conference had undergone significant changes as a result of input from state program leaders. The conference evaluation needed to address both the objectives of the conference and the impact of the conference on participating 4-H volunteers. We needed options for obtaining greater 4-H volunteer voice and more in-depth data than had been gleaned through previous quantitative approaches. Therefore, we investigated and implemented multiple evaluation strategies in hopes of gathering more comprehensive and impactful data.

The use of a single data collection technique can be a weakness. Findings generated through reliance on one evaluation method are affected by the biases inherent in that method (Johnson, Onwuegbuzie, & Turner, 2007). Additionally, collecting evaluation data over time from multiple sources results in a more comprehensive assessment of program impact (Cronbach, 1982; Patton, 1982, 1990). Methods for assessing program outputs and outcomes can involve engaging participants in graffiti wall responses, reflective journaling, focus group conversations, roundtable discussions, interviews, storytelling, creative writing, artistic renderings, and end-of-program questionnaires, any or all of which can be combined for implementation across the duration of a conference or similar event. Perhaps the most important consideration in choosing an evaluation strategy is whether the selected methods will yield answers to the questions being considered (Thomas, 2003). Previously, our conference planning team had collected reactionary data through a written questionnaire. This approach had garnered a response rate of approximately 35%, and we were confident that alternatives would allow us to gather more robust input and more meaningful data from a larger sample of participants.

Through examination and use of various methods, we identified advantages and challenges of each. Ultimately, the results of applying a multiple-methods approach were positive, and our experience can serve to guide others. Understanding diverse evaluation methods can assist Extension professionals in determining which of the available options will most effectively meet their specific needs, and the strategies we implemented can be adapted for use with many single- or multiday Extension programs. Herein we present findings related to our examination and implementation of alternative evaluation methods and describe strategies we used in applying them.

Alternative Evaluation Strategies

In setting the stage for implementing an integrated multiple-methods approach, we adopted the theme "Sharing Our Stories" to foster a culture of reflection throughout the event. We chose evaluation strategies that lent themselves to storytelling, reflection, and qualitative assessment. The summary of strategies that follows serves as a guide other Extension professionals can use to adopt similar strategies in developing robust evaluation and impact reporting plans.

Graffiti Wall (Open-Ended) Questions and Sharing

Graffiti walls provide opportunities for participants to anonymously and creatively express ideas and build on the responses of others (Stuart, Maynard, & Rouncefield, 2015). Graffiti walls are informal assessment tools that provide boundary-less space for brainstorming and sharing to encourage participants to interact creatively (Cadieux, 2011).

Collecting evaluation information through graffiti walls allows respondents to return to a display to add information throughout an event. Although graffiti walls provide options for personal artistic contributions, challenges exist. Graffiti walls require accessible space and art supplies throughout an event. Also, analyzing the resulting data can be time intensive as multiple reviewers must code and compile graffiti responses.

We strategically placed graffiti walls in locations where people stood in line at different times. This format gave participants opportunities to react during what would have been idle time. To ensure that the walls stimulated engagement, volunteer conference advisory group members helped craft questions and invited responses (Culp, Edwards, & Jordan, 2015b). Because we varied the locations of the graffiti walls during the conference, participants who attended one activity but not another had opportunities to share. Conference participants were eager to respond to the graffiti wall questions; 82% responded to at least one graffiti wall question, and 69% responded to all of them. We captured data on the graffiti walls by recording responses electronically. We then categorized responses using a system of three raters, as outlined by Culp and Pilat (1998).

Reflective Journaling

Journaling is a process for deconstructing experiences, applying them to personal contexts, and productively moving forward (Lowe, Prout, & Murcia, 2013). Incorporating journaling as a conference evaluation tool provides participants greater freedom to creatively convey learning and assess how the conference is fulfilling personal needs. This strategy serves as a formative review of the participant's learning experience and helps with the selection of activities and presentations that support identified personal learning goals (Letch, 2012) while also placing learning in the context of the conference (Gulwadi, 2009). Journaling is a mechanism for considering both objective learning and emotional experiences and thereby generates greater involvement by the learner (Fortson & Sisk, 2007).

A challenging aspect of implementing reflective journaling is the intensive use of written communication. Including sections in a journal that encourage creative expression through poetry, drawing, or mind mapping increases interest among participants with artistic preferences and provides an alternative method for capturing thoughts and ideas for those who may struggle with reading and writing. Journaling creates a large volume of qualitative data for analysis, a trove of information that has depth but is time intensive to summarize and report.

At our conference, each attendee received an eight-page reflective journal. The journal included guiding questions, open space, and sentence prompts for completion. Participants journaled throughout the conference and received rewards when "caught in the act" of reflecting in their journals. At the conclusion of the conference, attendees submitted their journals for analysis.

Our use of journals served a twofold purpose. Because journals are self-reflective, the act of creating a journal supported transformative learning for participants (Fortson & Sisk, 2007). Each participant spent time considering the fundamental learning and the incidental learning occurring during the event. Additionally, as participants determined goals and articulated personal changes based on conference participation, they could make meaning of the experience and provide valuable input for future conferences. Forty-eight percent of conference participants completed journals for evaluation.

Focus Group Research and Roundtable Discussions

Focus group research is a qualitative assessment that captures the depth and breadth of a topic without strictly quantifying it (Washington & Fowler, 2005). Focus group sessions generate data through the give-and-take of group discussion (American Statistical Association, 1997), with respondents adding to others' comments to enhance their responses. Conducting focus group sessions using open-ended questions provides an opportunity for researchers to listen to and gather information from participants to assess the level of success in accomplishing program objectives, to probe subjects or topics more deeply, and to elicit program insights at a richer level (Nordstrom, Wilson, Kelsey, Maretzki, & Pitts, 2000).

Extension professionals can assess program effectiveness through the use of focus groups (Ansay, Perkins, & Nelson, 2004) to help identify specific questions and key issues. The technique is particularly useful when issues are underresearched, responses are unpredictable, or potential bias needs to be reduced (Culp & Pilat, 1998).

The use of focus group discussions to evaluate conferences provides an efficient means of exchanging information and reinforces a stated desire for networking (Culp, Edwards, & Jordan, 2015a). Other assessments may fail to identify needs and issues that are more likely to arise in the structure of a focus group (Gamon, 1992; The Community Tool Box, 2014). Data gathered through focus group research are important for planners in making informed choices for subsequent conferences (Krueger & Casey, 2009). Analyzing this qualitative information requires that a note taker accurately capture the discussion, generating a transcript that researchers must then analyze to identify themes and classify information for reporting.

For our event, conference volunteers were trained in conducting focus group discussions. Training included instruction as well as role play. These facilitators played a pivotal role in conducting successful focus group interviews (Culp et al., 2015a). Focus group discussions occurred over lunch in small groups. The first day's discussion involved questions addressing individuals' motivations for serving as 4-H volunteers. The second day's questions related to evaluating the conference. Because the focus group discussions were during a meal, every individual at lunch (100%) participated.

Interviews

Researchers may use interviews in evaluation to gather both quantitative and qualitative data (Kaplan, 2016). Evaluative interviews can increase the depth of information provided by participants and provide researchers with opportunities to ask clarifying questions to ensure that findings accurately reflect intended messages (Holm-Hansen, 2007).

Interviewing as an evaluation method can be challenging in that conducting interviews and analyzing the resulting data are time-consuming endeavors and interviewer bias may affect reporting or interpretation of the data (Holm-Hansen, 2007). Training interviewers and developing the interview questions in advance helps the researcher manage for biases and ensures consistency in the process. The method is useful for individuals who are comfortable speaking and responding to a series of questions on camera.

Before implementation of interviews at our event, conference volunteers learned effective interviewing skills. A staff member recorded the interviews, which were conducted during free time and facilitated by the trained conference volunteers. Twenty-two attendees were interviewed and responded to questions previously developed by the conference volunteers and faculty collaboratively. Questions were designed to elicit input on specific topics related to 4-H volunteer experiences and the conference experience. Interviews provided a more structured means of gathering information than storytelling.

Storytelling

Incorporating stories into learning and discovery is a valuable approach in both teaching and research precisely because it is not scientific (Bruner, 1986). Stories are compelling tools for finding meaning in a conflicted and contradictory world (Cronon, 1992). People ascribe meaning to events by forming the events into narratives (Green, 2003) used to share information, emotions, and ideas. Stories bring into focus aspects that numbers alone cannot illuminate (Franz, 2003; Peters, 2002, 2010; Peters & Franz, 2012), helping program evaluators discover, understand, and assess the meaning, importance, and value of Extension work in ways that are not accomplished through using numbers, reports, and statistics.

Sukop, Tobin, and Fischman (2007) cited many benefits to storytelling, including understanding a project from the participants' viewpoint, allowing for the identification of unintended consequences, and providing a means for engaging participants in the evaluation process. Storytelling is especially useful when implemented in combination with other assessment tools. Snowden (2012) suggested that creative response, storytelling, and graphic design are three ways to encourage creativity as an active component of evaluation assessments.

Storytelling as an assessment tool presents challenges. Experienced reality is never objective; stories are told from single or multiple points of view (Simmons, 2001). Stories reflect the perspective of the storyteller, and personal experiences influence not only how narrators share stories but also how stories are interpreted (Denning, 2011). It is important to take steps to improve both the registering of the initial story and its later recall. At knowledge-sharing sessions, things typically happen quickly. A person taking notes cannot entirely follow the discussion. A person fully engaged in the conversation does not have time to take notes. When feasible, storytelling should be audio-recorded, provided that the recording does not hamper the interchange (Denning, 2011).

During our conference, stories from 18 attendees were recorded. Although this was the smallest number of participants in any selected conference evaluation strategy, this method provided the richest data. The stories included personal vignettes about previous regional conferences and 4-H experiences in general. Participants were encouraged to focus their stories, keeping each story centered on a single idea. This evaluation strategy resulted in both anecdotal information useful for analyzing conference impact and video resources useful for marketing and recruitment.

Creative Writing and Artistic Renderings

Engaging participants in evaluation activities that appeal to their creativity can break down barriers and allow participants to provide unfiltered responses (Snowden, 2012). Creative opportunities allow freedom for self-expression and encourage participants to individualize experiences as they create artistic responses. By definition, art involves consciously choosing to use skill and imagination ("Art," n.d.); when this creative approach is applied to evaluation, the data are qualitative and open to interpretation.

Although this evaluation strategy involves a less scientific method for collecting information, the data obtained can help us more effectively tell Extension's stories to funders and decision makers (Layne, 1998). Providing resources to allow participants to draw, doodle, write, or perform creatively opens an organization to more diverse feedback regarding the effectiveness of instruction and the experience as a learning environment.

Creative evaluation endeavors are subject to interpretation and can be challenging to apply directly to stated objectives or outcomes (Layne, 1998). In instructing respondents, program planners must be intentional and thoughtful to ensure necessary boundaries while honoring artistic license. Using creative writing and artwork in evaluation can be difficult, as additional supplies and resources are needed. These strategies also require extra time, not only for planning and preparation but also for participants to create their responses and for researchers to compile resulting data.

We provided multiple ways for participants to express themselves creatively. On two consecutive days, tables at mealtime included scrapbook paper and art supplies. One day attendees were asked to share their individual conference stories, and the other day they were invited to respond to the prompt "4-H is . . ." Their artistic endeavors included drawings, poems, and word art. Participation levels of 74% and 71%, respectively, for the two activities provided ample data for examination.

Integration of Evaluation Throughout a Conference

Because we incorporated evaluation throughout the conference, participants shared formative assessment data at different points in time and provided summative data as well. Our use of multiple methods to collect evaluative data provided participants with opportunities to contribute to conference assessment via methods that appealed to individual needs and preferences. By ensuring that at least one strategy would be appealing to every attendee, we enabled participants to be more authentic.

So What?

Our initial response was amazement. We were excited at the enthusiasm with which participants embraced different evaluation methods and the higher levels of engagement. Participants wrote hesitantly on graffiti walls initially but then began looking for them as they entered buildings. Participants willingly conversed in roundtable discussions and focus groups and worked on their reflective journals each day. They lined up to tell their stories and to be interviewed. Every person participated in multiple evaluation methods. The response rate was higher than we anticipated, leading to increased volumes of data for analysis and examination and for use in making programming decisions.

We learned that attendees liked some aspects of the new conference model and missed some of the old components. We also realized that many people were more comfortable speaking than writing. Data generated via Likert-type scales are easy to average but had not been telling the whole story of the conference. Participants at this recurring conference are 4-H volunteers; they have stories to tell and experiences to share. We learned that the truest impacts of the conference and of participants' volunteer experiences could not be quantified.

Conclusions and Implications for Extension

Our multiple-methods strategy for evaluating a 4-H volunteer conference constituted a varied and diverse approach to determining the impact of a conference, assessing attendee satisfaction, and describing the experience of conference participation. Multiple methods appealed to the attendees, providing them with a variety of platforms for sharing input and reflections while also building a sense of belonging. Extension professionals using multiple methods to evaluate events will gain access to volumes of data while creating positive experiences for participants engaged in the process. The multiple-methods approach, although requiring additional time for planning and for compiling and analyzing data, results in stronger data and relevant stories to share with stakeholders.

The primary objectives of the conference were to deliver 4-H volunteer education and to provide networking opportunities. The evaluation opportunities we presented enabled participants to share their perceptions about gained knowledge and skills and encouraged them to engage with others in creative assessment activities. Participants further solidified learning while we received valuable data for future programming.

The use of multiple evaluation methods, conducted throughout the event, yielded a richer data set than had been collected previously. Attendees embraced different evaluation strategies and shared feedback freely. They were forthcoming and candid in responding. We had not anticipated the valuable testimonials captured through the recorded interviews and storytelling vignettes. We subsequently used those to promote the conference and to market 4-H volunteer opportunities throughout the region.

Gathering a variety of data also enabled us to better plan for future conferences. By examining workshops and activities the attendees found most useful, we have been able to make meaningful changes. Additionally, the stories and artistic materials created by participants are being used for sharing the impact of the event with decision makers, stakeholders, and potential funders.

The creation of shareable products was an added benefit of the multiple-methods approach. Not only were photographs of participants engaged in the activities shared, but the resulting graffiti walls, journals, videos, and artwork also comprised descriptive information of interest to diverse audiences. These creative pieces were useful for enhancing social media presence for the conference. This by-product of the evaluation strategy met another conference planning goal: sharing authentic, engaging, and timely information with 4-H volunteers not attending the conference.

Our intent was to incorporate 4-H volunteer voice in designing an evaluation for a multiday 4-H volunteer conference. To this end, we trained conference volunteers and provided them with opportunities to accept ownership of evaluation tools and strategies. Ultimately, keywords from the graffiti walls and journals became the conference logo (Figure 1). Participants' illustrations became art in printed programs. Their stories became encouragement for other 4-H volunteers. Their discussions led to focused workshop tracks at subsequent conferences. Their feedback influenced the conference footprint. Their participation in our multiple-methods evaluation process improved the conference.

Figure 1.
Conference Logo Developed from Attendee Input

The 4-H emblem represents fourfold, holistic youth development. A quantitative assessment focuses on the head; however, it can overlook the impact on the heart, hands, and health of the individual. A mixed-methods evaluation strategy honors our holistic approach.

Author Note

Author Jenny W. Jordan's affiliation has changed. She is Navy 4-H program specialist at North Carolina State University in Raleigh, North Carolina.

References

American Statistical Association. (1997). What are focus groups? Alexandria, VA: Section on Survey Research Methods, American Statistical Association. Retrieved from http://www.prm.nau.edu/prm447/asa%20brochures/focusgroups.pdf

Ansay, S. J., Perkins, D. F., & Nelson, J. (2004). Interpreting outcomes: Using focus groups in evaluation research. Family Relations, 53(3), 310–316.

Art [Def 2]. (n.d.) In Merriam-Webster Online. Retrieved from http://www.merriam-webster.com/dictionary/art

Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.

Cadieux, C. (2011). Top ten assessment strategies for K-12 teachers. Retrieved from http://info.marygrove.edu/MATblog/bid/75035/Top-Ten-Assessment-Strategies-for-K-12-Teachers

Cronbach, L. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.

Cronon, W. (1992). A place for stories: Nature, history, and narrative. The Journal of American History, 78(4), 1347–1376.

Culp, K., III, Edwards, H. C., & Jordan, J. W. (2015a). The use of focus groups to evaluate the Volunteer Conference of Southern States. Journal of Extension , 53 (4), Article 4FEA6. Available at: http://www.joe.org/joe/2015/august/a6.php

Culp, K., III, Edwards, H. C., & Jordan, J. W. (2015b). Utilizing an advisory group to obtain volunteer perspective for regional programming. Journal of Extension , 53 (5), Article 5FEA9. Available at: http://www.joe.org/joe/2015october/a9.php

Culp, K., III, & Pilat, M. (1998). Converting qualitative feedback into quantifiable categories. Journal of Extension, 36(5), Article 5IAW3. Available at: http://www.joe.org/joe/1998october/iw3.php

Denning, S. (2011). The leader's guide to storytelling. San Francisco, CA: Jossey-Bass

Fortson, A., & Sisk, R. (2007, May). Reflective journaling as assessment and teaching. From the REAP International Online Conference on Assessment Design for Learner Responsibility. Retrieved from https://www.reap.ac.uk/reap/reap07/Portals/2/CSL/feast%20of%20case%20studies/Reflective_journaling_as_assessment_and_teaching.pdf

Franz, N. (2003). Transformative learning in Extension staff partnerships: Facilitating individual, joint, and organizational change. Journal of Extension, 41(2), Article 2FEA1. Available at: http://www.joe.org/joe/2003april/a1.php

Franz, N., Arnold, M., & Baughman, S. (2014). The role of evaluation in determining the public value of Extension. Journal of Extension, 52(4), Article 4COM3. Available at: https://www.joe.org/joe/2014august/comm3.php

Galloway, R., Peterson, K., & Dalton, M. (2006). Student focus groups reveal impacts of 4-H program. Journal of Extension, 44(4), Article 4TOT2. Available at: http://www.joe.org/joe/2006august/tt2.php

Gamon, J. A. (1992). Focus groups: A needs assessment tool. Journal of Extension, 30(1), Article 1TOT2. Available at: http://www.joe.org/joe/1992spring/tt2.php

Green, A. (2003). Difficult stories: Service learning, race, class and whiteness. College Composition and Communication, 55(2), 276–301.

Gulwadi, G. B. (2009). Using reflective journals in a sustainable design studio. International Journal of Sustainability in Higher Education, 10(2), 96–106.

Holm-Hansen, C. (2007). Conducting interviews: Tips for conducting program evaluation. Retrieved from http://www.gao.gov/special.pubs/pe1015.pdf

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133.

Kaplan, J. (2016). Interviews. Retrieved from http://betterevaluation.org/en/evaluation-options/interviews

Krueger, R. A., & Casey, M. A. (2009). Focus groups: A practical guide for applied research (4th ed.). Thousand Oaks, CA: Sage Publications, Inc.

Layne, J. (1998). An artful approach to program evaluation. Optimum, The Journal of Public Sector Management, 29(2/3), 1–9. Retrieved from http://www.optimumonline.ca/pdf/29-2/program_evaluation.pdf

Letch, N. (2012). Using reflective journals to engage students in learning business process management concepts. Knowledge Management & E-Learning: An International Journal, 4(4), 435–454.

Lowe, G. M., Prout, P., & Murcia, K. (2013). I see, I think I wonder: An evaluation of journaling as a critical reactive practice tool for aiding teachers in challenging or confronting contexts. Australian Journal of Teacher Education, 38(6). http://dx.doi.org/10.14221/ajte.2013v38n6.6

Nordstrom, P. A., Wilson, L. L., Kelsey, T. W., Maretzki, A. N., & Pitts, C. W. (2000). The use of focus group interviews to evaluate agriculture education materials for students, teachers and consumers. Journal of Extension, 38(5), Article 5RIB2. Available at: http://www.joe.org/joe/2000october/rb2.php

Patton, M. J. (1982). Practical evaluation. Beverly Hills, CA: Sage.

Patton, M. J. (1990). Qualitative evaluation and research methods. Beverly Hills, CA: Sage.

Peters, S. J. (2002). Citizens developing a voice at the table: A story of educational organizing in contemporary Extension work. Journal of Extension, 40(4), Article 4FEA1. Available at: http://www.joe.org/joe/2002august/a1.php

Peters, S. J. (2010). Democracy and higher education: Traditions and stories of civic engagement. East Lansing, MI: Michigan State University Press.

Peters, S., & Franz, N. K. (2012). Stories and storytelling in Extension work. Journal of Extension, 50(4), Article 4FEA1. Available at: http://www.joe.org/joe/2012august/a1.php

Simmons, A. (2001). The story factor: Secrets of influence from the art of storytelling. New York, NY: Basic Books.

Snowden (2012, May 4). How art can strengthen evaluation [Web log post]. Retrieved from http://blog.americansforthearts.org/2012/05/04/how-art-can-strengthen-evaluation

Stuart, K., Maynard, L., & Rouncefield, C. (2015). Evaluation practice for projects with young people: Creative research. Los Angeles, CA: SAGE Publications.

Sukop, S., Tobin, J., & Fischman, G. E. (2007). Storytelling approaches to program evaluation: An introduction. Retrieved from http://www.racialequitytools.org/resourcefiles/calendow.pdf

The Community Tool Box (2014). Qualitative methods to assess community issues. Retrieved from http://www.ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/qualitative-methods/main

Thomas, R. M. (2003). Blending qualitative & quantitative: Research methods in theses and dissertations. Thousand Oaks, CA: Corwin Press, Inc.

Washington, R., & Fowler, S. (2005). Systematic assessment of resistance to Extension organizational change: Evidence from the Alabama Cooperative Extension System. Journal of Extension, 43(2), Article 2RIB2. Available at: http://www.joe.org/joe/2005april/rb2.php