A User-Friendly Evaluation Resource Kit for Extension Agents Delivering Financial Education Programs

Introduction

Financial education programs play an important role in helping people achieve their financial well-being (Gorham, DeVaney, & Bechman, 1998). Cooperative Extension provides educational programs to address these issues. The Family and Consumer Sciences Extension agents deliver financial education programs at the local level. These programs vary with the financial needs of local citizens.

There are two main barriers agents face when trying to document the impact of financial education programs. The first is that many agents do not have the necessary evaluation knowledge and skills. The second is that user-friendly and reliable evaluation tools do not exist to help them assess the impact of their programs (Lyons, Palmer, Jayaratne, & Scherpf, 2006). To address these barriers, the National Endowment for Financial Education (NEFE) provided funding to create a resource kit that could be used to evaluate financial education programs.

An evaluation toolkit was subsequently developed by the authors. This tool kit was tested with Extension agents and found to be user-friendly and reliable for designing instruments to evaluate financial education programs.

Evaluation Resource Kit

The resource kit is available online to Extension agents at: <http://www.nefe.org/eval/license_agreement.html>. The toolkit has two components, an evaluation manual and an online database for designing customized evaluation instruments.

The Evaluation Manual

The purpose of the evaluation manual is to help Extension agents understand basic evaluation concepts and apply those concepts to document program impact. There are five parts to the manual. The first two parts provide an overview of financial education and the basics of program evaluation. The third part explains the evaluation planning process and the identification of impact indicators. The fourth part explains how to use the database to design evaluation instruments. The final part explains the processes associated with collecting, summarizing, and presenting evaluation data. The manual is available online and in printable form.

The Online Evaluation Database

The online database is a user-friendly resource for designing customized evaluation instruments. Users select from one of four specific evaluation formats as well as from a pool of impact indicators. Users can also add their own questions and statements and are not limited to the indicators provided. The indicators are categorized under six major financial themes. Within these six themes, the indicators are further categorized under sub-themes, and the user has the option to select questions from the topics of interest.

Use of the Evaluation Resource Kit

To access the toolkit, the user first needs to accept NEFE's license agreement. Once the user clicks on agree, a new Web page opens, and the user is directed to the evaluation manual and database. It is advisable to review the manual before using the database. The online version of the manual is ideal for quickly reviewing needed content.

When the user clicks on the database, the process for creating a customized evaluation instrument begins. There are six to 10 steps to designing an evaluation tool, depending on the type of options selected by the user. The first step is deciding which evaluation option is best. There are four choices, "post evaluation only" for one-time short presentations, "pre and post evaluation" for one-time relatively long workshops, "stages to change evaluation" for multi-session programs, and "train-the-trainer evaluation" for instructor trainings.

Once the appropriate evaluation option is selected, a new Web page opens, and the user is prompted to enter general program information such as the name of the program and organization. At this stage, the user also has the option to create an information-recording sheet for the program and a follow-up evaluation instrument.

The user then moves the curser to the "Next" button and clicks on it. From here, the subsequent steps vary with respect to the type of evaluation option selected. For example, the "post evaluation only" and the "pre and post evaluation" options have 10 steps. The "post evaluation only" option documents participants' levels of knowledge, skills, and aspirations at the end of the program. The "pre and post evaluation" option captures actual changes in participants' knowledge, skills, and intended behavior. If the follow-up option is selected, participants' actual behavior change can be documented.

The "stages to change evaluation" option has nine steps and is appropriate for documenting participants' progress towards actual behavior change. The "train-the-trainer evaluation" option has 10 steps and documents the impact that the training has on instructors' subject matter knowledge, training skills, personal financial management skills, and confidence levels in delivering the program.

For all of the above options, a user can add to an evaluation instrument questions that capture qualitative and demographic information. Once the user is familiar with the evaluation database, and has a clear understanding of the program objectives, he or she is able to design a customized and reliable evaluation instrument within about ten minutes.

Applications and Recommendations

Overall, this evaluation toolkit helps Extension agents delivering financial education create reliable evaluation instruments. In using this resource, agents can do a better job of evaluating their programs. The train-the-trainer evaluation option of the database is appropriate for financial education Extension specialists to assess their in-service training programs. The manual contains general evaluation information that can be used by any Extension professional who wants to develop a better understanding of program evaluation.

Acknowledgment

This project was funded by the National Endowment for Financial Education® (NEFE®). Project number 001-10-2003

References

Gorham, E. E., DeVaney, S. A., & Bechman, J. C. (1998). Adoption of financial management practices: A program assessment. Journal of Extension [On-line], 36 (2). Available at: http://www.joe.org/joe/1998april/a5.html

Lyons, A. C., Palmer, L., Jayaratne, K. S. U., & Scherpf, E. (2006). Are we making the grade? A national overview of financial education and program evaluation. The Journal of Consumer Affairs, 40 (2), 208-235.