Using Pre- and Post- Tests to Evaluate the Achievement of Short Course Learning Objectives

Introduction

The Oregon Forest Institute for Teachers (OFIT), a 6-day in situ educational workshop, is designed to help K-12 classroom teachers understand the management of our forest resources and to share with them materials and activities they can use in their own classrooms. OFIT was modeled on a California program (Forestry Institute for Teachers) that began in 1995 and is based on a set of educational goals for teacher education institutes developed by a national taskforce of the Society of American Foresters (SAF).

Twenty-four teachers attended the first OFIT in August, 1999. The results of the trial run were very encouraging. Instructors and participants alike were pleased with the outcomes of the first program. However, as with all first attempts, there was room for improvement. The steering committee, an eclectic group of professional forestry educators, felt that we had not adhered to the educational goals in designing institute activities and that we had no real measure of how well our educational goals had been achieved.

Content Themes and Measurable Objectives

To address this issue prior to the 2000 program, the steering committee prioritized clear themes based on the SAF goals, developed measurable learning objectives, and designed learning activities to meet these objectives. Finally, the steering committee agreed that we needed an evaluation that would enable us to measure if our learning objectives have been achieved.

During the following year, the OFIT Steering Committee met with much success. A six-person curriculum sub-committee clarified the program content themes and learning objectives. The three major themes were defined as follows.

1. Forest Systems: Oregon's forests are complex, resilient, and varied ecosystems.
2. Forest Values: Competing demands for a wide variety of values influence the management of Oregon's forests.
3. Forest Management: Sustainable forest management can be achieved through diverse objectives, perspectives, techniques, and regulations.

Specific, measurable learning objectives were then developed under each theme. These objectives take the form of action statements prefaced by the phrase: "Upon completing OFIT, participants will:"

1. Forest Systems
   1. Identify major human and environmental components of the forest system.
   2. Illustrate changes in the system over time.
   3. Describe the interaction between a system's parts.
   4. Describe the variety of forests across the state.
2. Forest Values
   1. Identify and describe the wide variety of values placed on forests.
   2. Illustrate the competing values that underlie a forest issue.
   3. Compare and contrast the major types of forest landowners and managers.
3. Forest Management
   1. Identify and describe how changing science and technology are used in forest management.
   2. Identify various ways the public is involved in influencing forest management.
   3. Give examples of plans, policies, or rules that regulate forest practices.
   4. Identify and describe the work of a variety of professionals that are involved in managing forests.

These educational themes and learning objectives were used in designing OFIT 2000. The 6-day workshop was broken into three segments representing the themes. Within a theme, specific activities were selected to address each of the learning objectives. The underlying themes, learning objectives, and curriculum were thoroughly reviewed and accepted by the steering committee.

Results

Thirty-seven K-12 teachers attended OFIT 2000. Participants represented a diversity of institutions, community sizes, and opinions. During the program, OFIT participants were asked to participate in several forms of evaluation, including daily questionnaires, an end-of-week survey, and a pre- and post-test.

The pre- and post-course test was designed specifically to measure and document changes in a participant's understanding of forests and forestry. Participants put their names on the tests so we could measure specific changes in individual scores.

The pre-test and post-test consisted of identical questions. The nine test questions were fashioned directly from the 11 learning objectives. To answer each fill-in-the-blank question, participants were asked to list four items. Participants were awarded one point for each correctly filled-in blank. A perfect score was thus 36 points. Two learning objectives not covered in the pre-test and post-test were left out because they could not be easily measured using this type of question.

Participants were asked general questions like, "list four major human or environmental components of a forest system," "list four ways that a forest system changes over time," and "list four types of professionals that are involved in managing forests."

All tests were scored at one sitting, in as objective a manner as possible. Thirty-five of the thirty-seven students participated in both tests. One student left OFIT 2000 before the post-test was administered, and one student asked not to participate in the post-test.

Pre-test scores ranged from 10 to 35, with a mean of 23.0 out of 36. Post-test scores ranged from 23 to 36, with a mean of 31.7 out of 36.

A t-test for matched groups or paired t-test was used examined the null hypothesis that the pre-test and post-test score populations are the same. This hypothesis was rejected at the p=.01 level. The alternate hypothesis that the score populations are different was accepted. We conclude that the increase in the mean score from 23.0 to 31.7 is a statistically significant increase.

Conclusion

Paired t-tests and many other statistical tests useful in program evaluation are explained clearly by Fitz-Gibbon and Morris (1978). Fitz-Gibbon and Morris provide simple worksheets to calculate the paired t-statistic. Most statistics software packages also can perform paired t-tests. However, we have found it much simpler to develop a simple spreadsheet based on the formulae provided by Fitz-Gibbon and Morris.

This form of evaluation assures us that the OFIT 2000 learning objectives have been achieved. Participants showed increased knowledge and understanding in the three thematic areas of forest systems, forest values, and forest management.

Achieving these learning objectives was only one of the aims of OFIT 2000. Feedback through the other evaluations indicates that our other institute objectives (like creating a community of forest-literate K-12 educators) were also met.

This experience has shown that the success of a workshop or short course can more easily be ascertained by developing measurable learning objectives, teaching to the objectives, and measuring the learning with pre- and post-tests.

References

Fitz-Gibbon, C. T., & Morris, L. L.. 1978. How to calculate statistics. Sage Publications/Beverly Hill. 142 pp.