Using Youth Participatory Evaluation to Improve a Bullying Prevention Program

Introduction

Extension employees typically use established curricula when planning their youth programs. Often, it is necessary to modify and adapt curriculum packets to meet implementation time lines, budgets, and particular organizational needs. If a curriculum has to be modified, faculty, regional educators, and/or county educators generally make unilateral decisions concerning the activities and lessons to be included, taking into account time constraints and specific audience needs. Although Extension faculty and staff have research-based knowledge about youth, those with the greatest local knowledge about youth interests are youths themselves. One way to utilize this knowledge is through participatory approaches that involve youth. These approaches, usually referred to as youth participatory evaluation (YPE), have the potential to positively affect the relevance and effectiveness of programs.

The use of participatory approaches is still a relatively new way to involve youths in improving programs that serve them (Checkoway, Dobbie, & Richards-Schuster, 2003). Involving youths in the program planning process provides Extension personnel with fresh perspectives on how to meet youths' needs. Youths are able to give feedback concerning various aspects of the program being implemented—including feedback about both content and process—and these assessments can help improve the program's design and daily operation (London, Zimmerman, & Erbstein, 2003; Walker, 2007). Capitalizing on the expertise and perspectives of youths at the beginning of the planning process can be beneficial to program implementation. Accordingly, methods such as YPE can serve as models for youth involvement in a variety of Extension programs.

Participatory evaluation is grounded in the philosophy that program participants should play an active role in an evaluation that is practical, useful, and empowering (Nieto, Schaffner, & Henderson, 1997). Participatory evaluation that involves youth focuses on utilizing youths' insights to improve the programs and organizations designed to serve them (Sabo Flores, 2008). YPE is predicated on the assumption that youth are capable of making meaningful contributions to their organizations and communities when given the opportunity (Eccles & Gootman, 2002; Scales, 1990). Youths' contributions can include presenting ideas, problem solving, organizing groups, conducting research and evaluation, and working to develop new services (Checkoway & Richards-Schuster, 2004). Furthermore, YPE projects create opportunities for youths to generate evaluation questions, develop their own evaluation instruments, and analyze their own data to form conclusions about the effectiveness of programs (Sabo, 2001). With this progressive approach to program evaluation, youth are repositioned from passive receivers of program curricula to active decision makers in the program planning and evaluation processes. YPE capitalizes on the diverse assets, experiences, and expertise of youth and creates a context for leadership and meaningful contributions to program development and evaluation.

In an exemplar YPE project, Lifting New Voices, Checkoway and Richards-Schuster (2004) implemented a 5-year program designed to create community change through a collaboration between community-based organizations and youth participants in a statewide project. Lifting New Voices provided youths opportunities to contribute throughout the program planning process. Youths also helped develop the evaluation instruments used to collect and analyze data about the project's effectiveness. Although this is an exemplary model of YPE, the process of involving youth in program development and evaluation can vary. Other ways to partner with youth using the YPE principles of youth voice and participation involve youths in less extensive ways; these include forming learning communities and engaging in youth consultation.

Forming learning communities is a hybrid YPE approach that Extension faculty and staff have used to engage youth. The process starts with Extension faculty's introducing Extension county educators to research-based tools and ideas. Afterward, county educators and youths design and pilot test individual programs to meet local needs, using their newly developed skills and knowledge (Barnaba, Krasny, Kasperek, Hoskins, & Hope, 2000; Krasny & Doyle, 2002).

Extension faculty and staff also have used YPE principles in youth consultation, another model that involves youth in the evaluation of programs. With this model, adults initiate a research project and consult with young people to make its operations more effective, thus utilizing their special knowledge about themselves as a group (Checkoway et al., 2003). The Children's Garden Consultants Program is an example of youth consultation in an Extension context. The program was a 3-day workshop that involved 4-H youths in the evaluation of a garden-based learning program (Lekies, Eames-Sheavly, Wong, & Ceccarini, 2006). Over the 3 days, youths listened to presentations, went on garden tours, and watched videos concerning children's gardening programs. Subsequently, the youths created presentations that highlighted their recommendations for improving garden design and garden programming. YPE was used in this context to engage youth in program planning and to improve a program that was currently being offered in communities.

The project discussed in this article is similar to the consultation model. However, our project explored the process of including youths in assessing a program before it was implemented in communities. The bullying prevention curriculum titled Be SAFE was evaluated by youths and adults. Be SAFE is for youths aged 11–14 and has 26 lessons organized in 10 focus areas. Our goal with this article is to discuss how we used YPE principles to reduce the number of lessons and activities that would be implemented and to ensure that the activities implemented across our state would have the greatest impact possible. Although having several lessons and activities can be a strength of a program, Extension educators have many programs to implement in their assigned communities and must be sure that program time is used effectively and that the activities are likely to be impactful. We undertook our YPE project to help agents achieve this goal.

Method

Participants

The curriculum evaluation team had 12 members: seven girls, aged 13–15 and in the ninth grade, and five adults. The youth partners were participants in the Auburn University Young Women Leaders Program (YWLP), which is modeled on a similar program developed at the University of Virginia. YWLP is a university-led, afterschool mentorship program at the local junior high school that is structured to facilitate leadership, relational skills, and empowerment. The girls in YWLP, who are referred to as Little Sisters, were asked to participate in this project because they are "leaders in training." The Little Sisters represent voices of youths who have been historically marginalized because of gender, racial/ethnic background, and/or socioeconomic status. The five adult partners were women who represented various positions in higher education: an Extension specialist, the faculty advisor to YWLP, two graduate students, and an undergraduate student.

Procedure

Our evaluation process occurred over 3 months. The meetings were held at an eighth-and-ninth-grade junior high school after school each week for an hour and a half. A graduate student led the start of each meeting by asking each girl about the highs and lows of her week. Then we transitioned into the Be SAFE lesson for the day by passing out handouts for the activities and introducing the topic. Girls were reminded at each meeting that although they were participating in the activities, they also would be evaluating the activities after their participation. After we completed all the lessons in the curriculum, the youths also read through the evaluation survey tool that accompanied the curriculum and generated questions and concerns.

Data Collection

Data about each lesson were collected from the girls through the use of a sticker survey method adapted from the work of Sabo Flores (2008). The star-sticker method was used to make completion of the surveys more enjoyable, encourage participation, and reduce social desirability bias through anonymity. The survey questions were developed by the authors and were designed to determine age appropriateness of the activity, helpfulness of the information, and participant enjoyment of the activity. In each category, 5 stars indicated excellent, and 1 star indicated poor. At the bottom of the survey, there was a space in which the girls could give written feedback related to what they liked and disliked about the lesson. The girls were given a star-sticker survey to complete after each lesson. The girls rated the lesson using stickers and filled out the qualitative section of the survey form. They were reminded not to write their names on the surveys to ensure anonymity.

Observational data were collected by the adults who were not implementing the curriculum that day. Two adult partners were responsible for writing observational field notes in three domains: levels of youth engagement in the lesson, interactions between adults and youths during the lesson, and interpersonal interactions among youths during the lesson. Field notes were written during each meeting.

Data Analysis

Each lesson's score was calculated by totaling the number of stars it received from youth participants. The lessons were then sorted from highest score to lowest score, and the result was recorded in a table (Table 1). Afterward, the written qualitative responses about each lesson were recorded in the table in a corresponding column. To further rank the activities, we created another column in which we recorded written observation data from the adults, specifically related to communication during the lessons. The lessons with the highest rated scores and most positive comments were included in the final modified curriculum. This process allowed us to reduce the curriculum from 26 lessons to eight lessons.

Table 1.
Youths' Rankings of Lessons, Highest to Lowest

Rank	Lesson
1	Take a Stand!
2	Who Am I?
3	Seeing Is Believing—Or Is It?
4	Speaking Up and Standing With: Skills for Being an Ally
5	The Relationships Continuum
6	Clear Mind Mud Mind: Understanding State of Mind
7	Creating Space for Feelings
8	The Mood Elevator
9	What's the Difference?
10	What Makes Bullying Real for You?
11	Taking Action to Stop Cyberbullying
12	And Words (and Images) Can Hurt Forever
13	Standing Up: Assertive Versus Aggressive Responses
14	Creating Personal Respect Posters
15	And the Beat Goes On—How Does Music Move You?
16	Contributing to the Circle of Courage
17	Steps to Cyber Safety
18	Moving to Circles of Support
19	Wake Up and Smell the Messages
20	Exploring Spheres of Influence
21	Exploring Separate Realities
22	What's in a Name?
23	Gathering the Perspectives of Others
24	Everyday Mindfulness
25	Creating a Safety Plan
26	The Art of Listening

Results

Feedback relating to levels of fun and engagement helped us determine which lessons and activities would be most promising to implement in a condensed and modified version of the curriculum. The primary purpose for eliciting youth input during the evaluation process was to inform decisions about which lessons to retain. When the youths rated lessons on the basis of fun and enjoyment, the highest scored activities involved moving around the room, processing scenarios in small groups, and role playing. In contrast, activities that were mostly adult-led discussions were rated least desirable and generally were not included in the modified curriculum. Table 2 shows the activities that were included in the new version of the curriculum. Also, on the basis of the girls' written feedback, we changed two group activities that required participants to write and discuss at their desks so that they instead write on large, flip-chart paper placed in different parts of the room. With this active learning approach, youths participating in the revamped curriculum will be able to shift from the monotony of sitting at their desks to the action of moving around while engaging in discussions with one another.

Table 2.
Lessons Included in the Modified Curriculum

Lesson order	Lesson name
Introduction	Ground Rules
1	What Makes Bullying Real for You?
2	Taking Action to Stop Cyberbullying
3	Who Am I?
4	Take a Stand!
5	Clear Mind Mud Mind: Understanding State of Mind
6	The Relationships Continuum
7	Speaking Up and Standing With: Skills for Being an Ally
8	Standing Up: Assertive Versus Aggressive Responses

Also because of conversations with the girls about the activities, we decided to add summaries to the beginnings of the lessons. For example, the first lesson, What Makes Bullying Real for You, focused on three categories of bullying—verbal, physical, and indirect—and the behaviors that coincide with each category. The lesson started with a general definition of bullying, and then students were organized in small groups and asked to stand near one of three flip charts containing the words Verbal, Physical, and Indirect. Each group was responsible for writing down behavioral examples of the applicable type of bullying and then rotating to another station. As the activity progressed, each group added something to each list. Afterward, we discussed what the girls had written. Although the girls enjoyed the structure and content of the lesson, two of them suggested that there should be more discussion on why people do not stand up for the victims of bullying and why a bully may engage in bullying behaviors. These two topics were to be discussed in depth in subsequent lessons, but we had not taken the time to provide an overview of all the lessons we would be using in the curriculum. The adult facilitators decided it would be helpful to give students a preview with each lesson of what we would be discussing during that lesson and later in the curriculum, thereby making it easier to keep the conversations more focused during each lesson.

Changing the word choice and phrasing in some activities was another important area of improvement that was highlighted by working with the youths. Literacy levels are an important consideration for optimal understanding of curriculum activities. Although there is often a broad range of literacy levels in classrooms in which youths are the same age, curricula that target the wide age group of 11- to 14-year-olds must not include words that are too advanced. For example, the first activity in the Be SAFE curriculum asks youths to define the word affirming and give examples of what the word means to them. However, many of the girls in our study did not know the definition of affirming and had difficulty defining it without help from the facilitator. The girls suggested changing affirming to something more common and adding affirming later.

The girls also reviewed the curriculum's retrospective survey. As they answered questions on the survey, they noted that a few questions were phrased in ways that exceeded their literacy levels. For example, one question related to building emotional intelligence: "I understand that my internal experience of a moment will shift when my thinking settles." Although we went through the activity that directly corresponded to this question, we had to explain the meaning of the question in a way that compromised the survey. We asked the youths how we could make the idea easier to understand. After several iterations, the question was reworded as, "I know that once I calm down, my thinking will be clearer."

Although this project used a participatory approach, future projects that utilize YPE could be more inclusive if youths were provided with the curriculum to read and review before the program begins and involved in the actual evaluation process once it is implemented. This approach would provide more opportunities for youth to become stakeholders in their own development.

Conclusion

The YPE process that was implemented with the YWLP youths was helpful in reviewing and revising the bullying prevention program that had been selected for use in our state. The surveys, written feedback from youths, and adult facilitators' observations provided in the form of field notes helped us condense Be SAFE from 26 lessons to eight lessons and revise some of the lessons to make them more interactive for youths. Without youth feedback, curriculum choices would have been made solely by adults without the insights of youths. Youth input also helped the Extension specialist involved in the project proactively identify potential obstacles in the curriculum and include alternative words and approaches during curriculum trainings with county and regional Extension educators.

Although our utilization of YPE was beneficial to the program planning process, there is also real value for youths who are involved in YPE. Sabo Flores (2008) clarifies the multiple benefits of YPE:

The activity of engaging youth in the evaluation process is more than a means of improving the quality of the data or ensuring that particular constituencies are heard; it is a process that both mirrors and supports the tenets of positive youth development and leads to stronger and more appropriate programming. (p. 13)

By including youths as collaborators in the planning and evaluation processes for youth programs, Extension personnel can ensure that Extension programs and curricula can have a meaningful impact on the youths being served. Youth involvement in YPE efforts also can translate into an engaged audience for adult and family programs as youths mature. Research indicates that youths' participation in the planning and evaluation processes of programs that serve them presents an opportunity for long-term involvement and ownership (Brennan, Barnett, & Baugh, 2007).

References

Barnaba, E. M., Krasny, M. E., Kasperek, L. M., Hoskins, S., & Hope, J. (2000). Explorations from an aerial perspective educator's manual. Ithaca, NY: Cornell Media Services.

Brennan, M. A., Barnett, R. V., & Baugh, E. (2007). Youth involvement in community development: Implications and possibilities for Extension. Journal of Extension [online], 45(4) Article 4FEA3. Available at: http://www.joe.org/joe/2007august/a3.php

Checkoway, B., Dobbie, E., & Richards-Schuster, K. (2003). The wingspread symposium: Involving young people in community evaluation research. Community Youth Development Journal, 4(1), 7–11.

Checkoway, B., & Richards-Schuster, K. (2004). Youth participation in community evaluation research. American Journal of Evaluation, 24, 21–33.

Eccles, J. S., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Committee on Community-Level Programs for Youth. Washington, DC: National Academies Press.

Krasny, M., & Doyle, R. (2002). Participatory approaches to program development and engaging youth in research: The case of an intergenerational urban community gardening program. Journal of Extension [online], 40(5) Article 5FEA3. Available at: http://www.joe.org/joe/2002october/a3.php

Lekies, K. S., Eames-Sheavly, M., Wong, K. J., & Ceccarini, A. (2006). Children's garden consultants: A new model of engaging youth to inform garden design and programming. HortTechnology, 16, 139–142.

London, J. K., Zimmerman, K., & Erbstein, N. (2003). Youth-led research and evaluation: Tools for youth, organizational, and community development. New Directions for Evaluation, 98, 33–45.

Nieto, R. D., Schaffner, D., & Henderson, J. L. (1997). Examining community needs through a capacity assessment. Journal of Extension [online], 35(3) Article 3FEA1. Available at: http://www.joe.org/joe/1997june/a1.html

Sabo, K. (2001). The benefits of participatory evaluation for children and youth. PLA Notes, 42, 48–52.

Sabo Flores, K. (2008). Youth participatory evaluation: Strategies for engaging young people. San Francisco, CA: John Wiley & Sons, Inc.

Scales, P. (1990). Developing capable young people: An alternative strategy for prevention programs. The Journal of Early Adolescence, 10(4), 420–438.

Walker, K. (2007). Youth empowerment evaluation: Learning voice. American Journal of Evaluation, 28, 321–326.