Application of Item Analysis to Assess Multiple-Choice Examinations in the Mississippi Master Cattle Producer Program

Introduction

The Mississippi Master Cattle Producer program is a comprehensive training in major beef cattle production topic areas developed and administered by the Mississippi State University Extension Service. An Internet-based self-study version of the program is available for online completion at http://msucares.com/livestock/beef/mcp. It is designed for persons interested in learning at their own pace about improving production on beef cattle operations. The Internet-based training features streaming video of speakers and slides. Slides with scripts, reference publications, and certification examinations are available for online viewing or download.

The program consists of training modules in eight beef cattle production subject areas. Program participants complete about 24 hours of training across the following subject areas:

• Beef Cattle Nutrition
• Forage Systems
• Beef Cattle Reproduction
• Breeding and Genetics
• Economics and Marketing
• Herd Health and Handling
• Beef End Product
• Beef Quality Assurance

Participants must successfully review all course materials and pass the examinations for all subject areas to be eligible for program certification. Examinations consist of 15 multiple-choice items each. This format was selected for simplicity in program administration (Santos, 2000). Recognizing the importance of examination validity, it was determined that examinations needed review to ensure that they were functioning as intended (Norland, 1990; Radhakrishna, 2007). Therefore, the objective of the evaluation reported here was to perform an item analysis of Mississippi Master Cattle Producer examination responses to 1) determine the difficulty of individual examinations, 2) assess the effectiveness of distractors for individual items, and 3) identify specific topics appropriate for placing further educational emphasis.

Item Analysis

A post-hoc item analysis was performed to examine the program assessments to improve these tests by revising or eliminating ineffective items. This was done by evaluating item difficulty and ability to discriminate or separate test takers who show a high degree of skill or knowledge from test takers who do not show a high degree of skill or knowledge. Item analysis also identified specific items for review. The statistical software package, IBM SPSS Statistics 20.0.0, was used to analyze examination responses (SPSS, 2011). Because participants were allowed to retake the examinations until a passing score was achieved, only the initial examination responses were included in the analysis.

Item difficulty was defined as the proportion of participants answering the item correctly. Measures less than 0.30 indicated a relatively difficult item, whereas measures greater than 0.85 indicated a relatively easy item (Bailey, 1998). Item-total correlation was used as a measure of correlation between the item and overall examination. Item-total correlations less than 0.30 indicated items that were considered not to discriminate well. When an item discriminated well, a participant who earned a high score on the item would also earn a relatively high score on the overall examination and vice-versa. The discrimination index was used as a measure of how well the item separated high-scoring participants from low-scoring participants. Relatively difficult and easy items with discrimination indices less than 0.00, and moderately difficult items with discrimination indices less than 0.30, were determined to need review (Zurawski, 1998).

Distractors were incorrect responses meant to entice the participant to choose them in place of the correct response. In analyzing individual distractors within an item, examination scores were ranked into thirds. The percentage of participants choosing each response was calculated for these three score groups.

Item and Examination Difficulty

Examinations completed by 50 program participants were analyzed. Item difficulty classifications by examination subject matter area are presented in Table 1. Eighty-five percent of the items were classified as easy. Possible reasons for this include the following: 1) the items themselves were not very rigorous, 2) the program participants demonstrated a good mastery of concepts covered by the examinations, or 3) the reference materials were adequate to support test-takers in selecting the correct responses. The degree of item difficulty may have been due to the question wording, distractor effectiveness, or how well the material covered provided the information needed to correctly answer the questions asked.

Program participants were required to achieve a score of at least 80% correct responses to pass an examination as part of the program certification process. On all examinations except the breeding and genetics one, at least 94% of participants passed their initial examination attempt. Three out of every 10 persons failed their initial attempt on the breeding and genetics examination.

Problematic Item Identification

The discrimination index, item-total correlation, and item difficulty measure and classification are presented in Table 2 for the breeding and genetics examination. Item 15 was identified as needing review because it had an item difficulty of 0.43 (moderately difficult) and a discrimination index of 0.22, which was less than the 0.30 threshold for moderately difficult items to escape review. Items 3, 4, 6, 11, 14, and 15 had item-total correlation values below 0.30 and thus did not discriminate well.

Distractor Evaluation

Individual distractors were evaluated for items identified as needing review. For example, the correct response for item 15 on the breeding and genetics examination was chosen by only 40% of participants, ranking in the top third for overall score on that examination. One distractor was chosen by 60% of this top-scoring group. This suggests that the distractor could have been tricking participants who otherwise understood the material being tested. The two other distractors were chosen relatively infrequently by the middle- and bottom-scoring groups and never by the top-scoring group. They were ineffective, perhaps too obvious, distractors. It is possible that this problematic item could be salvaged by replacing the distractors, or the item may need rewording or complete replacement.

Conclusions

Item analysis can serve as a useful tool in improving multiple-choice questions used in Extension programming. It can identify gaps between instruction and assessment. Combined with direct feedback from program participants, it was determined that additional emphasis needs to be placed on breeding and genetics instruction in future beef cattle extension programming.

References

Bailey, K. M. (1998). Learning about language assessment: Dilemmas, decisions, and directions. Pacific Grove, CA: Heinle & Heinle Publishers.

Norland, E. V. T. (1990). Controlling error in evaluation instruments. Journal of Extension [On-line], 28(2) Article 2TOT2. Available at: http://www.joe.org/joe/1990summer/tt2.php

Radhakrishna, R. B. (2007). Tips for developing and testing questionnaires/instruments. Journal of Extension [On-line], 45(1) Article 1TOT2. Available at: http://www.joe.org/joe/2007february/tt2.php

Santos, J. R. A. (2000). Getting the most out of multiple response questions. Journal of Extension [On-line], 38(3) Article 3TOT2. Available at: http://www.joe.org/joe/2000june/tt2.php

SPSS. (2011). SPSS for Windows, Release 20.0.0. SPSS, Inc. Chicago, IL.

Zurawski, R. M. (1998). Making the most of exams: Procedures for item analysis. The National Teaching & Learning Forum. Retrieved from: http://cte.udel.edu/sites/udel.edu.cte/files/ntlf/v7n6/exams.htm