Developing a Program Evaluation Instrument for Texas 4-H: A Work in Progress

In 1999, over one million youth participated in 4-H activities through a total of ninety-eight project categories in Texas (Texas 4-H, 2000). While these figures lead the nation in participation by state, it has become evident that the Texas 4-H and Youth Development Program must take an aggressive approach to evaluate county, regional, and state programs. It is imperative that state officials, Extension colleagues, and the general public have a full understanding of not only the unparalleled 4-H enrollment numbers in Texas, but also of the effectiveness and impact of 4-H programs.

The evaluation instrument is threefold. The first objective is to measure youth development skills outlined by the 4-H model of head, heart, hands, and health. The second objective evaluates the effectiveness of the program being measured by analyzing knowledge gained and overall satisfaction of the program. The third objective is to measure the demographics of the program participants to ensure the program is reaching an audience similar to the demographic make-up of Texas.

Evaluation Instrument

The development of an evaluation instrument to measure programs in Texas can be challenging because of the variation in programming and membership in the Texas 4-H Program. After a series of discussions by the evaluation committee, it was determined that the evaluation instrument should consist of five major sections and be Web accessible from the Texas 4-H Web page. These sections include universal youth development skills, program impact, program satisfaction, demographics, and an open-ended suggestion section. The program impact and program satisfaction sections will have a pull-down mechanism on the Web page to allow the program coordinators the ability to choose the appropriate statements to meet their specific needs for their program.

The youth development skills section is a set of statements that are relevant to all project experiences and to youth of all ages and backgrounds. It was developed according to the life skills outlined by the head, heart, hand, and health fourfold development model (Horton, Hutchinson, Barkman, Machtmes, & Myers, 1999). This set of 12 statements will be analyzed using a Likert Scaling system on a 1 to 5 rating (1 = Deceased Greatly, 2 = Decreased, 3 = No Change, 4 = Increased, 5 = Increased Greatly). Program participants will simply check the appropriate box according to their self perceptions pertaining to each statement.

The program impact section will measure the effectiveness of the knowledge objectives set forth by the program coordinator at the initiation of the program. Again, the program coordinator has the option to choose the statements from the Web page. This section evaluates skills gained from each program and will be used to evaluate the specific subject matter on a Likert-type Scale (1 = Decreased Greatly, 2 = Decreased, 3 = No Change, 4 = Increased, 5 = Increased Greatly).

The program satisfaction section asks the participant to evaluate the layout of the program to determine the effectiveness of speakers, location, and schedules. Similar to the program impact statement, the coordinator will decide which statements to use in this section. This portion of the survey will allow the program coordinator the ability to measure program environment and effectiveness of teaching materials. This section will also be measured using a Likert-type Scale (1 = Very Dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very Satisfied).

The demographic section and the open-ended suggestion section are self explanatory. The demographic section measures gender, race, community population, age, grade level, and school type attended by the youth participants. The open-ended suggestion section will be a qualitative section provided to simply offer verbal feedback for participants based on individual concerns not targeted within the other four sections of the evaluation instrument.

Testing

The evaluation instrument is currently being tested on several programs to determine correlation relationships between statements and totals by determining statement reliability, construct validity, and face validity. After analysis, modifications will be made with statistical support. Also, program coordinators have administered a verbal evaluation to participants to measure the youth's ability to understand the evaluation and the clarity of instructions.

After modifications of the instrument are complete, statistical summaries will be formulated for programs using descriptive techniques by investigating differences between medians of comparable groups as outlined by Mogay (1999). Also, responses to youth development skills, program impact, and program satisfaction will be analyzed using descriptive statistics by SPSS (2000). Suggestions and guidelines from the University Institutional Review Board will be considered related to administering the survey to minor children. These data and their results will assist in monthly and yearly reporting to stakeholders, legislators, and clientele in explaining the impact these programs are having on youth across Texas.

Challenges

Similar to other new ideas and techniques, this on-line evaluation program has challenges that must be addressed. The biggest stumbling block centers around computer access. This Web-based entry site must be compatible with the computer technology that is in the county offices for proper utilization. Also, trainings must be developed to provide insight to County Extension Agents on how to enter the data correctly and utilize these results properly and effectively to more accurately summarize these programs for their monthly and yearly narratives. Finally, for this on-line evaluation to be efficient, it is imperative that the measuring indicators reflect the original objectives of the program.

After approval of all modifications to the program evaluation instrument are complete, the instrument will be placed on the Web for program coordinators to use for evaluation. The goals of the instrument are to evaluate programs, allowing coordinators to make positive corrections to meet the needs of the youth audience and to allow for impact accountability of Texas 4-H Programs and activities.

References

Horton, R. L., Hutchinson, S., Barkman, S. J., Machtmes,K., & Myers, H. (1999). Developing experientially based 4-H curriculum materials. 1999 National Association of Extension 4-H Agents Association Preconference. October 23, Pittsburgh, PA.

Moray, N. (1999). So you want to use a Likert Scale? Learning Technology Dissemination Initiative. Retrieved September 8, 2000, from the World Wide Web: http://137.195.128.40/Itdi/cookbook/info_likert_scale/index.html

SPSS. (1999). SPSS. Applications guide (Base 10.0). SPSS Inc., Chicago, IL.

Texas 4-H. (2000). College Station, TX. Retrieved September 14, 2000, from the World Wide Web: http://resources.tamu.edu/scripts/es237_reports_form.asp