Assessing Program Impact with the Critical Incident Technique

More than 30 years ago, several articles were published in the Journal of Extension (JOE) about the Critical Incident Technique (CIT), a method of qualitative data analysis that was used to study Extension staff job performance (Hampton, 1980; Santopolo & Kell, 1976). The CIT did not surface, however, in a search of recent published JOE manuscripts and may, thus, be unfamiliar to a new generation of Extension faculty. This article bridges that gap by describing recent use of the CIT in an Extension-led personal finance training program for library professionals.

Developed by Flanagan (1954), the CIT is a systematic, open-ended technique tor encouraging subjects to tell personal stories that provide descriptive data. In the case of the aforementioned program, the CIT was used to collect data about how library staff made use of information provided during a series of financial education classes. The focus of the class series was to build their skills, confidence, and capacity to help library patrons access timely, accurate, and trustworthy information and resource materials about personal finance topics.

The Critical Incident Technique

The CIT is a qualitative data technique. As such, the focus is on collecting open-ended descriptive data from respondents and converting it into meaningful categories that can be used to quantify the results (Culp & Pilat, 1998). It is commonly used in studies of nursing care, marketing efforts, and the effectiveness of employee training programs. Researchers who use the CIT employ a structured methodology to encourage respondents to share their experiences about a particular topic of study (e.g., program impact evaluation). As described by Flanagan (1954), there are five steps in the CIT procedure: 1. Determination of the objectives of the activity, 2. Development of plans and specifications for collecting factual incidents, 3. Collection of data, 4. Analysis of data, and 5. Interpretation and reporting of facts regarding reported incidents. While critical incidents represent only raw data and not automatic solutions or insights, the CIT can assist with collecting information that is directly relevant to important questions or problems (Flanagan, 1954).

Very simply, the CIT is a form of story-telling where respondents are asked to share their experiences about something (e.g., how they handled certain situations). For example, Hampton (1980) used the CIT to study Extension agents' effectiveness in carrying out their job responsibilities, and Santoplo & Kell (1976) conducted a similar analysis with EFNEP aides. Respondents are typically asked to describe a critical incident (e.g., work project, difficult situation, conflict/confrontation, client interaction, communication problem) and its outcome and their feelings about and/or perceptions of it. Incidents are considered effective/successful when something positive occurs: program objectives are met, staff and/or clientele are happy, effective solutions are found, and information from staff training is put into practice. On the flip side, incidents are considered ineffective/unsuccessful when there are negative results (e.g., failure to obtain objectives, make progress, or satisfy staff and/or clientele needs).

The CIT has advantages and disadvantages. On the plus side, it provides a methodology to collect data in respondents' own words and is fairly inexpensive to administer. Questions can be asked using personal interviews or written (or online) questionnaires and provide valuable insights that might be missed by forced choice responses. Disadvantages include respondents' difficulty recalling less recent incidents and unwillingness to take time to fully consider CIT questions and "tell their story" to researchers (Critical Incident Technique, 2012; Flanagan, 1954).

Application to a Personal Finance Program

The CIT was used as one of several evaluation methods in an Extension-led financial education training program for library staff. Staff completed a questionnaire before a series of 11 classes began and will complete another upon project completion. The four questions used to elicit respondents' critical incident stories were as follows.

1. Think about your experiences helping patrons with personal finance questions. Remember a time when you had a successful experience helping someone with these types of questions. Please write down what happened.
2. What made this a successful, positive experience?
3. Think about your experiences helping patrons with personal finance questions. Remember a time when you had an unsuccessful experience helping someone with these types of questions. Please write down what happened.
4. What made this an unsuccessful or challenging experience?

Following administration of a pre-training survey, the number of successful and unsuccessful incidents was recorded and tabulated. Successful incidents were defined as those where staff were able to assist patrons with personal finance questions. A sample of 73 library staff reported 40 successful and 39 unsuccessful critical incidents. These incidents were then sorted into quantifiable categories. Successful incident categories were: provided quality information resources, patron satisfied, provided referral, and enabled patron to solve problem. Unsuccessful critical incident categories were: lack of training/knowledge, patron wanted advice/more help than able to provide, lack of resources, and poor patron attitude (Radford, 2012).

Reporting Program Impact

CIT data can be used to evaluate program impact in several ways. First, differences in the number of successful and unsuccessful incidents can be compared using data collected before and after an educational program to determine if and how participants made use of the information provided. Ideally, there will be an increase in reported post-training successful incidents and fewer unsuccessful incidents versus pre-training results. Responses in specific categories of incidents can also be compared.

Questions about respondents' perceptions of critical incidents can also provide excellent quotes for reports to stakeholders and funders who may like to see case study examples in evaluation reports as well as quantitative data. The following quote is an example.

A patron needed information about financial aid for school. I was able to direct her to sites that were appropriate to her request. It was a positive experience because I actually knew the answer and the patron was satisfied with the site provided (Radford, 2012).

Summary

This article described the use and application of the critical incident technique (CIT) to collect qualitative program impact data. Extension faculty should consider adding the CIT to their "toolkit" of program evaluation methods.

References

Critical incident technique (2012). Wikipedia. Retrieved from: http://en.wikipedia.org/wiki/Critical_Incident_Technique 

Culp, K., & Pilat, M. (1998). Converting qualitative feedback into quantifiable categories. Journal of Extension [On-line], 36(5) Article 5IAW3. Available at: http://www.joe.org/joe/1998october/iw3.php 

Flanagan, J.C . (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-355.

Hampton, L. A. (1980). The Extension agent's job: Critical behaviors. Journal of Extension [On-line], 18(1). Available at: http://www.joe.org/joe/1980january/80-1-a2.pdf

Radford, M. (2012). Evaluation report 1: Money Matters financial education program. New York: New York Public Library.

Santopolo, F. A., & Kell, K. (1976). Paraprofessionals: Critical job requirements. Journal of Extension [On-line], 14(4). Available at: http://www.joe.org/joe/1976july/76-4-a1.pdf