A Retrospective Pretest-Posttest Evaluation of a One-Time Personal Finance Training

Introduction and Review of Literature

A continuing concern among Extension educators is how to assess whether our educational initiatives result in positive outcomes. Indeed, attempts to measure the impact of Extension education programs have been implemented to varying effects for many years (Davis, 2003; Diem, 2003; Lyons, Palmer, Jayaratne, & Scherpf, 2006; Rockwell & Kohn, 1989). Despite countless examples of evaluation instruments from which Extension educators may seek guidance, there is little consensus when it comes to identifying the most effective evaluation method for use with one-time, short-duration consumer education programs or training efforts common in Extension personal finance programs.

Fortunately, a growing number of scholars are providing guidance on how best to evaluate the effectiveness of one-time financial education workshops, classes, or programs (Fox, Bartholome, & Lee, 2005; Lyons et al., 2006; O'Neill, 1998; Rockwell & Kohn, 1989). Given continued interest in improving the delivery and effectiveness of Extension education programs by those who fund, as well as those who implement, such programs (Diem, 2003; O'Neill, 1998), it is reasonable to suggest that Extension educators must continue to learn from, and adopt, evaluation strategies that have taken place in similar educational contexts. Toward this goal, this article reports on the use of a retrospective pretest-posttest in a one-time personal finance workshop taught by Extension faculty for high school teachers responsible for teaching six new state-mandated personal finance performance standards.

A retrospective pretest-posttest evaluation differs from a traditional pretest-posttest evaluation in that the retrospective evaluation instrument is administered once, rather than twice. To assess change in the constructs of interest, a retrospective pretest-posttest instrument is administered at the conclusion of an intervention or experience and asks respondents to refer back to a prior point in time. This retrospective response is accompanied by a contemporaneous response. By comparison, a standard pretest-posttest evaluation is administered twice: once prior to the event or experience and again at the conclusion (Posavac & Carey, 1997).

A retrospective pretest-posttest evaluation can be an effective evaluation strategy for one-time workshops because the additional costs (pecuniary and opportunity costs) of a traditional pretest-posttest evaluation design often are not warranted. Further, in some settings retrospective pretest-posttest evaluations have been shown to provide a more accurate assessment of program outcomes relative to traditional pretest-posttest assessments (Pratt, McGuigan, & Katzev, 2000). Retrospective pretest-posttest evaluations have also been shown to be useful for documenting self-assessed changes that occur as a result of the particular intervention, in part, because retrospective pretest-posttest evaluations are more sensitive to respondent change than traditional pretest-posttest evaluations (Skeff, Bergen, & Stratos, 1992).

Pratt et al. (2000) articulated three benefits of a retrospective pretest-posttest evaluation over traditional pretest-posttest designs that are particularly relevant for Extension faculty. First, a retrospective pretest-posttest evaluation reduces bias that may result if participants are not present at the very beginning and end of a training session. Second, a retrospective pretest-posttest instrument allows scarce educational time to be devoted to the delivery of the educational material rather than evaluation. Finally, the results from retrospective pretest-posttest evaluations may be better than traditional pretest-posttest methods if participants' perceived knowledge of a subject is based on incorrect information, a situation that may only be illuminated after the participant has participated in the workshop. This so-called "response shift bias," (Howard & Dailey, 1979) is an inaccurate self-report of one's knowledge on a pretest because one does not yet have enough information to understand that one's knowledge of a subject is not well developed and is particularly problematic when workshops address complex subjects that are clarified over the course of the training (Rockwell & Kohn, 1989).

As described in the next section, the retrospective pretest-posttest evaluation used in the study reported here was constructed to assess high school teachers' self-reported confidence teaching concepts related to Georgia's new personal finance performance standards before, and after, a one-time workshop conducted by Extension faculty.

Methods

Beginning in the 2007-08 academic year, Georgia mandated a new set of educational performance standards across all grades and curricula. One component of this sweeping revision of the state's educational standards was a set of six new social studies performance standards (SSEPF1- SSEPF6) related to the economics of personal finance for grades 9-12 (Georgia Department of Education, 2006). These new performance standards required that students demonstrate the ability to: 1) apply rational decision making to personal spending and saving choices, 2) explain that banks and other financial institutions are businesses that channel funds from savers to investors, 3) explain how changes in monetary and fiscal policy can have an impact on an individual's spending and saving choices, 4) evaluate the costs and benefits of using credit, 5) describe how insurance and other risk-management strategies protect against financial loss, and 6) describe how the earnings of workers are determined in the marketplace.

To help high school teachers become more comfortable teaching economic and personal finance principles that meet the new performance standards, University of Georgia Cooperative Extension faculty provided a 10-hour, in-person workshops in six locations between May and July 2007. Teachers who attended one of the workshops received copies of sample curricula from the National Council on Economic Education's Financial Fitness for Life (National Council on Economic Education, 2008), the National Endowment for Financial Education's High School Financial Planning Program (National Endowment for Financial Education, 2008), and related materials developed by Georgia Extension faculty. The workshops were also designed to familiarize teachers with numerous books, activities, and organizations available to help them prepare their students to meet the six new performance standards. As an added incentive for high school teachers to participate, attendees qualified for one Professional Learning Unit (PLU) necessary for career advancement.

The Financial Literacy for High School Students workshops were attended by 37 teachers. Of the 32 teachers who completed an evaluation, 26 (81%) were female, five (16%) were male, and one (3%) declined to answer. Most teachers (27) were White (84%); three (9%) were African American; one (3%) indicated "other"; and one (3%) declined to answer. Three (9%) indicated that they were middle school teachers, whereas 29 (88%) taught high school and one (3%) declined to answer. The most common subjects taught by the teachers included family and consumer sciences (28%) and economics (47%); the remainder (25%) taught some other subject. The teachers reported instructing an average of 197 students each year, with approximately 47% of their students living in low-income households. Teachers estimated that 42% of their students were from racial or ethnic minority groups.

To assess the extent to which the teachers thought their teaching skills related to each of the six performance standards had changed as a result of the workshop, a retrospective pretest-posttest evaluation instrument was administered at the conclusion of the workshops. As shown in Figure 1, at the conclusion of the workshop teachers were asked to indicate their level of agreement with each of the eight statements before and after they participated in the workshop, on a scale from 1 to 5 where 1=Strongly Disagree and 5=Strongly Agree. As constructed, the retrospective pretest-posttest instrument allowed the teachers to indicate how much their agreement with the statement changed as a result of the workshop.

Figure 1.
Retrospective Pretest-Posttest Evaluation Instrument

The primary goal of the workshops was to introduce teachers to curricula that reinforced the performance standards. Therefore, the first six items in Section 1 of the evaluation instrument (Figure 1) queried teachers' level of agreement with six statements about their teaching abilities that corresponded to the six performance standards. However, the workshop was also designed to provide hands-on personal finance activities (e.g., game show-style teaching aids, computer-based teaching aids) that could be used to reinforce middle and high school students' learning. As a result, the seventh item in Section 1 assessed the extent to which the teachers had access to personal finance educational activities before and after the workshop. Finally, the eighth item in Section 1 assessed the teachers' confidence in their ability to manage their own finances before and after the workshops.

Because this group of teachers was not a simple random sample selected from a larger known population, statistical inferences are not appropriate. Indeed, one should expect that teachers who self-select into a 10-hour workshop will be different on many dimensions from teachers who did not self-select into the workshop. Nevertheless, a careful review of the quantitative results is illustrative and follows in the next section.

Results

Notably, all but nine teachers indicated higher agreement with each and every statement after participating in the workshop. Further, the nine teachers who did not report higher agreement on each item indicated that there was either no change from "4=somewhat agree" on the retrospective pretest to "4=somewhat agree" on the posttest or "5=strongly agree" on the retrospective pretest to "5=strongly agree" on the posttest.

As shown in Table 1, the overall average for these eight items increased from 3.6 before the workshop to 4.8 after the workshop. The average score increased a full point or more for every item except "I am confident in my ability to manage my own finances." As one might expect from a group of teachers who teach economics and personal finance, their confidence in their ability to manage their own finances was the highest of all group mean scores before the workshop (4.2). As a group, the teachers' confidence in their own ability to manage their finances increased to 4.8 at the conclusion of the workshop.

Table 1.
Evaluation Results

	Group Mean
Evaluative Statement	Before Workshopa	After Workshopa	Differenced
I can teach others how to apply a rational decision making model when making spending and saving choices.b	3.6	4.9	+ 1.3
I can teach others the relationship between financial institutions and savers/investors.b	3.7	4.7	+ 1.0
I can teach others the role of monetary and fiscal policy in the U.S. economy.b	3.0	4.5	+ 1.5
I can teach others how to evaluate the costs and benefits of credit.b	3.9	4.9	+ 1.0
I can teach others how insurance and other risk-management strategies protect against financial loss.b	3.5	4.7	+ 1.2
I can teach others how the earnings of workers are determined in the marketplace.b	3.7	4.7	+ 1.0
I have access to many personal finance-related activities to use in the classroom.c	3.2	4.9	+ 1.7
I am confident in my ability to manage my own finances.c	4.2	4.8	+ 0.7
Average of all eight items	3.6	4.8	+ 1.2
Notes: N=32 a 1=Strongly disagree, 2=Somewhat disagree, 3=Neutral, 4=Somewhat agree, 5=Strongly agree b Reflects Performance Standards for Economic Understanding (Georgia Department of Education, 2006) c Reflects learning goal of workshop coordinators d Every participant indicated either no change (from "4=somewhat agree" on the retrospective pretest to "4=somewhat agree" on the posttest or "5=strongly agree" on the retrospective pretest to "5=strongly agree" on the posttest) or positive change (stronger agreement with each statement). Not surprisingly, dependent sample t-tests indicate that the difference between the pretest and posttest is significant for each item at a 99% confidence level. Identical conclusions were reached with a non-parametric alternative, the Wilcoxon rank sum test.

Prior to the workshops teachers were least confident in their ability to teach others the role of monetary and fiscal policy in the U.S. economy. This performance standard retained the lowest mean confidence after the workshop, but the mean level of agreement increased from 3.0 before the workshop to 4.5 afterward. After monetary and fiscal policy, the largest reported increase in the mean was reflected in their ability to teach others how to apply a rational decision-making model to spending and saving choices (from 3.6 to 4.9) and in teaching others how insurance and other risk-management strategies protect against financial loss (from 3.5 to 4.7).

The largest change in pre- and post-workshop mean score was found on the statement assessing teachers' access to personal finance-related materials that could be used in the classroom. The group's mean retrospective response was 3.2, which increased to 4.9 at the conclusion of the workshop. Finally, teachers indicated that prior to the workshop they agreed most strongly that they had an ability to teach others how to evaluate the costs and benefits of using credit. The teachers' mean agreement before the workshop was 3.9, which increased to 4.9 afterward.

Although the primary purpose of the instrument was to assess any changes in the teachers' agreement about their ability to teach the core performance standard after attending the workshop, the Extension educators who conducted the workshops were also interested in teachers' perceptions of quality of the instructors, materials, and the workshop overall. As shown in Figure 1, a second section of the instrument asked teachers to rate the quality of the instructors, the materials, and the workshop overall on a scale of 1 (Not helpful) to 4 (Very helpful). Responses were overwhelmingly positive, with group mean scores of 3.9 for instructors, 4.0 for materials, and 3.9 for the workshop overall. Not surprisingly, when asked if they would recommend this workshop to others all teachers indicated they would recommend this workshop to others.

Conclusion and Implications

Because of the wide range of educational efforts in which Extension educators are involved, it may not be surprising that there is no consensus on how best to evaluate any given educational effort. Fox et al. (2005) provide a good discussion of potential frameworks that may guide the process of developing a meaningful set of evaluation "best-practices" from which Extension educators may receive guidance. In particular, Fox and his colleagues (2005) urge educators to exploit an evaluation framework based on Jacob's flexible five-tiered approach to evaluation (Jacobs, 1988). Most relevant here is the "program impact" evaluation tier, which is characterized by, among other things, attempts to "provide information that contributes to an area of knowledge and/or evaluation" (Fox et al., p. 214). Similarly, Lyons et al. (2006) specifically call on personal finance educators to build their own, and collectively the nation's, evaluation capacity by sharing methods that lead to effective evaluation practices. Lyons and her colleagues argue that there is "a general lack of time, staff and financial resources to conduct program evaluations" (p. 209) that inhibits Extension faculty from establishing effective evaluation practices.

Because of these recent calls for more thoughtful, rigorous, and efficient evaluations of the full range of consumer education and financial education efforts, and because Extension educators have the responsibility to demonstrate the effectiveness of their programs to many constituencies, continued effort toward the development of effective evaluation methods and instruments remains vital. Extension programs are often characterized by one-time, short-duration trainings that, compared with longer-term multi-day courses, present challenging time, content, and evaluation constraints. Because of these constraints, retrospective pretest-posttest evaluations may provide an advantage over traditional pretest-posttest designs.

As evidenced by these particular workshops, a retrospective pretest-posttest evaluation may address the calls for more thoughtful and rigorous evaluations in a way that allows scarce educational time to be devoted to the delivery of the educational material rather than evaluation. In the case of the workshop discussed here, this benefit allowed for approximately 20 minutes of additional instruction. While not extraordinary, each minute of instruction during a one-time workshop is extremely valuable. Moreover, as demonstrated by the teachers' higher levels of agreement about their ability to teach complex personal finance concepts, one-time, short-duration workshops remain a viable educational tool for Extension personal finance educators.

All six content-focused evaluation items from these personal finance workshops suggest that participants found that the one-time workshops significantly increased their ability to teach relatively challenging personal finance concepts.

Finally, as noted by Pratt et al. (2000), a retrospective pretest-posttest evaluation is preferred over traditional pretest-posttest methods when participants' perceived knowledge is incorrect, and participants may not know this until after the workshop. As explained by Howard and Dailey (1979) and demonstrated by Howard, Schmeck, and Bray (1979), participants may overestimate their knowledge on traditional self-reported pretests. This is particularly problematic when workshops address complex subjects that are illuminated over the course of the workshop (Rockwell & Kohn, 1989), a challenge common to many Extension program areas, including personal finance.

In light of the potential advantages of the retrospective pretest-posttest evaluation method and the increasingly demanding reporting environment where Extension programs must document impact of all activities, Extension educators who teach complex topics in one-time educational sessions similar to those outlined here are encouraged to include retrospective pretest-posttest evaluation in their repertoire of evaluation methods.

Acknowledgements

This work was supported, in part, by a grant from the National Council on Economic Education (NCEE) and the University of Georgia Family and Consumer Science Cooperative Extension. The author thanks Michael Rupured, Joan Koonce, and the many county Extension personnel who conducted the training and evaluations. The author also thanks Angela Lyons for sharing her practical evaluation insights.

References

Davis, G. (2003). Using a retrospective pre-post questionnaire to determine program impact. Journal of Extension [On-line], 41(4) Article 4TOT4. Available at: http://www.joe.org/joe/2003august/tt4.php

Diem, K. G. (2003). Program development in a political world—It's all about impact! Journal of Extension [On-line], 41(1) Article 1FEA6. Available at: http://www.joe.org/joe/2003february/a6.php

Fox, J., Bartholome, S., & Lee, J. (2005). Building the case for financial education. Journal of Consumer Affairs, 39(1), 195-214.

Georgia Department of Education. (2006). Social studies standards. Retrieved May 24, 2010 from: http://www.georgiastandards.org/socialstudies.aspx

Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2), 144-150.

Howard, G. S., Schmeck, R. R., & Bray, J. H. (1979). Internal invalidity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement, 16(2), 129-135.

Jacobs, F. H. (1988). The five-tiered approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs (pp. 37-68). New York: Aldine DeGruyter.

Lyons, A. C., Palmer, L., Jayaratne, K. S. U., & Scherpf, E. (2006). Are we making the grade? A national overview of financial education and program evaluation. Journal of Consumer Affairs, 40(2), 208-235.

National Council on Economic Education. (2008). Financial fitness for life. Retrieved February 18, 2008 from: http://fffl.ncee.net/

National Endowment for Financial Education. (2008). High school financial planning program. Retrieved February 18, 2008 from: http://www.nefe.org/HighSchoolProgram/tabid/146/Default.aspx

O'Neill, B. (1998). Money talks: Documenting the economic impact of Extension personal finance programs. Journal of Extension [On-line], 36(5) Article 5FEA2. Available at: http://www.joe.org/joe/1998october/a2.php

Posavac, E. J., & Carey, R. G. (1997). Program evaluation methods and case studies. Upper Saddle River, NJ: Prentice Hall, Inc.

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341-349.

Rockwell, S. K., & Kohn, H. (1989). Post-then-pre evaluation. Journal of Extension [On-line], 27(2) Article 2FEA5. Available at: http://www.joe.org/joe/1989summer/a5.php

Skeff, K. M., Bergen, M. R., & Stratos, G. A. (1992). Evaluation of a medical faculty development program: A comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Evaluation and the Health Care Professions, 15(3), 350-366.