An Analysis of the Use of an Incomplete Sentences Test for Employment Selection of Cooperative Extension Agents and Educators

A harried Extension administrator in the midst of a critical hiring decision weighs the pros and cons of several candidates, and becomes increasingly frustrated with the lack of clarity and the conflicting advice being offered from various sources about whom to hire or promote. The administrator asks, "isn't there something like Ohms law that would apply here?" This understandable longing for regularity, predictability, and certainty in the social world is abundantly expressed in the long and venerable use of selection testing and procedures for hiring and promotion.

The use of testing for personnel selection came to prominence with the massive problems of personnel selection and classification for the Armed Forces in the second world war. Examples of instruments used in contemporary personnel processes include, but are not limited to: aptitude tests, test of general intelligence, mechanical and comprehension tests, weighted biographical inventories, and various verbal and quantitative performance tests.

The validity of employment selection testing remains in dispute (DeWolf & van den Bosch, 1984; Wigdor & Garner, 1982; Hattrup, Schmidt, & Landis, 1992). Perlman et Al. (1980) estimated that at least 60% of the variation in single predictor- criteria relationships could be accounted for by sampling error, predictor and criterion unreliability, and range restriction. The use of pre-employment testing has also been substantially restricted by legislation and regulation which seeks to avoid discrimination against members of protected classes (Douglas, Feld, & Asquith, 1989). However, despite these concerns and constraints, instruments intended to aid in selection and to predict future performance remain widely and variously used. Increasingly, those with hiring responsibility in Extension and other fields seek valid and reliable means to supplement their intuition in assessing the potential of future employees (Rudner, 1992).

This is a report on eleven years of experience utilizing an incomplete sentences test as a part of the employment selection process for Extension Agents/Educators in a state Cooperative Extension Division. The incomplete sentences test was developed for the Cooperative Extension Service (CES) by the United States Department of Agriculture (Hahn, 1979) as an aid to effective personnel practices. The instrument contains sixty items, consisting of the first few words of sentences which the respondent is asked to complete on the basis of one's feelings. An objective scoring mechanism was provided which yields a numerical score between 0 and 100.

The instrument is designed to provide a prediction of the overall presence and strength of five job-related skills, abilities, or other attitudes (SAOs), identified during an analysis of the job content of CES field agents, and associated with job performance. The SAOs addressed by the incomplete sentences test include: job commitment, communication skills, interpersonal skills, positive attitude, problem solving ability, and self confidence. A study of the concurrent validity of the incomplete sentences test and job performance was reported by the USDA, (Hahn, 1979). In that study, 234 Cooperative Extension Service field staff completed the instrument and were then rated on job performance by their supervisors. Comparisons of those scores with performance ratings, using various methods for scoring the incomplete sentences test, yielded correlation coefficients in the range from .20 to .35. Authors of the instrument recommended that the incomplete sentence test be used as "a scorable selection instrument." Despite the relatively modest relationship between the incomplete sentence test scores and performance in the validity studies, Hahn (1979) and the Extension Committee on Organization and Policy advised that "if used, the incomplete sentence test can help to select superior agents" (ECOP, 1980).

The following objectives were established for this study:

Objective 1: Determine the extent to which scores from the incomplete sentences test accurately predict selection of applicants for the position of Cooperative Extension agent/educator.

Objective 2: Determine the extent to which scores from the incomplete sentence test accurately predict the future performance of Cooperative Extension agents/educators.

Method and Results

Active and inactive application files and materials were maintained and available for all applicants and selectees for the position of Cooperative Extension agent/educator in a state CES for the years 1982 through 1993. This data set included usable incomplete sentence test scores for 76 individuals who were hired and 38 who applied but were not selected. Incomplete sentence scores ranged from 69 to 97 for selectees and from 72 to 97 for non-selects.

Performance data for those agent/educators who were hired were also available. As a part of an annual performance evaluation system, each agent/educator's performance was reviewed by his or her district director. An overall evaluation from unsatisfactory to outstanding was awarded. From 1982 to 1991 directors could choose one of four nominal categories to describe the overall performance of agent/educators working in their district. After 1991 a fifth choice was added to the performance evaluation system. An overall measure of job performance was created by converting these nominal descriptors to numerical scores, standardizing them to take into account the changes in the descriptors used across different versions of the performance evaluation system, and calculating an overall mean score.

Scores from the incomplete sentences test were examined to determine how accurately they predict selection of applicants for the position of Cooperative Extension agent/educator

As can be seen from the data in Table 1, the mean score for the selectees was slightly lower than that of the non-selects. There was no statistically significant difference between the selectees and non-selectees.

Among the population of Cooperative Extension agents who had taken the incomplete sentences test as a part of their employment process, an analysis of the relationship between incomplete sentence scores and job performance yielded a Pearson Product- moment correlation of -0.149. According to Guilford (1956), correlation coefficients less than .20 can be interpreted as "slight, almost negligible relationships;" therefore, making it difficult to support the proposition that the incomplete sentences test is a valid predictor of employment decisions.

If job performance, as described in the methods section, is considered a dependent variable and is entered into a bivariate regression model with the incomplete sentence test score as the independent variable, an R-squared of .0933, and F score of 1.41 with a statistical significance of .2547, is obtained.

On the basis of the data collected, it is possible to conclude that in this study the incomplete sentences test scores have not been useful in discriminating between those who would ultimately be selected as Cooperative Extension agent/educators and those who were not. Given that the aggregate incomplete sentence test scores of persons hired as agents or educators was lower than the scores of those who applied, but were not selected, it becomes apparent that other elements of the employment process, such as review of qualifications, input from referees, and selection interviews predict or explain employment selection decisions more accurately. These results particularly call into question the usefulness of the incomplete sentences test as a screening device to identify those applicants who are likely to be hired.

Given a weak negative relationship, it is clear that the incomplete sentences test is of no more use in predicting job performance than it is in regard to hiring decisions. The strength of the relationship between incomplete sentence test scores and actual job performance measures approaches neither the levels reported in the 1979 validation study nor critical levels for rejection of a null hypothesis.

In summary, a study of over a decade of experience with the use of an incomplete sentences test in the employment processes of a state Cooperative Extension Division provides no support for its validity as either an employment screening device or as a predictor of future performance as an agent/educator. The results of this study are limited to this incomplete sentences test, in this CES setting. Objective tests and assessment instruments may be a useful component of an employment screening and selection process; however, their validity must be carefully and accurately assessed in the context in which they will be utilized.

DeWolf, C.J., & van den Bosch, R. (1984). Personnel selection. In P.J.D. Drenth et Al., (Eds.) Handbook of Work and Organizational Psychology. New York: Wiley.

Douglas, J.A., Feld, D.E., & Asquith, N. (1989). Employment testing manual. (pp. 2-1 - 2-39) Boston: Warren, Gorham & Lamont.

Extension Committee on Organization and Policy (1980). Personnel selection & performance appraisal workshop: 1980. Participant workbook. Author.

Gaugler, B.B., Rosenthal, D.B., Thornton, G.C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology. 72, 493-511.

Guilford, J.P. (1956). Fundamental Statistics in Psychology and Education. (p. 145) New York: McGraw Hill.

Hahn, C.P. (1979). Development of performance evaluation and selection procedures for the cooperative extension service: summary report. (Contract 12-05-300-372) . Washington D.C. American Institutes for Research.

Hatrup, K., Schmidt, N. & Landis, R.S. (1992). Equivalence of constructs measured by job-specific and commercially available aptitude tests. Journal of Applied Psychology. 77, 298-308.

Perlman, K. Scmidt, F.L., & Hunter, J.E. (1980). Validity generalization results for tests used to predict training success and job proficiency in clerical occupations. Journal of Applied Psychology, 65, 373-406.

Rudner, L.M. (1992). Pre-employment testing and employee productivity. Public Personnel Management. 21, 133-150.

Wigdor, A.K., & Garner, W.R., (Eds.). (1982). Ability Testing; Uses, Consequences, and Controversies. Committee on Ability Testing, Assembly of Behavioral and Social Sciences. Washington, D.C., National Academy Press.