Instructional Video Evaluation Instrument

A problem facing many Extension educators is how to evaluate videos for instructional content. This was apparent when the advisory committee for the High Plains Intermountain Center for Agricultural Heath and Safety (HI-CAHS) and Extension agents strongly recommended videos be used in the education programs developed by the HI-CAHS and several videos were identified that might be adapted for use in educational programs. That need resulted in this research and development of an evaluation instrument used by the center and its advisory committee.

For this project, videos are considered "products and product evaluation methods are needed to help extension agents and video producers arrive at better decisions based on reliable, accurate, and complete information (Krink & Gustafson, 1986). From idea inception through final product and beyond, evaluation needs to be incorporated into the production and viewing process. Formative and summative evaluation should "have a place all through the production process: before, during and after" (Hausman, 1990, p. 124). Evaluation is needed to provide the industry with more direction about what constitutes effective, high-quality educational agricultural video products.

Until evaluation becomes an integral part of the video design and production process, there will be no experienced-based knowledge of what video approaches work, with what type of audiences, under what kinds of conditions, and in what type of content areas (Sneed, 1991). Evaluation of products, including formative and summative procedures, is an important aspect of production, with the aim of "making administrative decisions and decisions about course improvement" (Krink & Gustafson, 1986, p. 217).

The lack of quality evaluation techniques, the emphasize on the negative, and too many variables to measure are barriers to evaluation reaching its intended goals. The lack of quality evaluation techniques is partly responsible for the misconception about what evaluation is and how to achieve it. In the real world, it is impossible to "isolate the effects of a video program on a particular outcome, like increased sales or improved job performance" (Sneed, 1991, p. 5). From its formative and summative roots in the design and production process, evaluation can be accomplished by the Extension agent to decide whether a tape is suitable for their particular instructional program. The instructional video product "should be a realization that there is a need for moving visual material of this type in a particular instructional situation either in a supporting role or as the main vehicle of instruction and that no suitable material is already available" (Ellington, 1985, p. 176).

Compiling the suggestions for quality indicators for videos found in books, articles, and forms by various authors (Dube, 1980; Ellington, 1985; Handbook of Forms, 1985; Hart, 1984; Hausman, 1990; Hunter, 1990; Hutton, 1984; Krink & Gustafson, 1986; National Career Development Association (NCDA), 1992; Pett, 1989; Sneed, 1991) and synthesizing the instructional design methodology of Brookfield (1985, 1986), Friere (1970), Galbraith (1991, 1992), and Seels and Glasgow (1990), an instrument was developed to aid the reviewer in evaluating a video for its instructional quality (Appendix A). This instrument is divided into four general areas of importance--Content, Instructional Plan, Technical Considerations, and Supplemental Materials.

Content is a prime concern in an instructional setting. The video must be accurate, useful, and free from bias. If the content is not correct and up-to-date, then the video is not ideally usable for learning. The content must be accurate and current (NCDA, 1992). The video must portray current and useful situations in today's world.

The content of the video must be useful. The video should stimulate, motivate and inform the learner to act on the information. Ideally, learners should consider and incorporate the ideas presented (Krink & Gustafson, 1986; NCDA, 1992).

The video should be bias-free, avoiding stereotyping because of age, gender, ethnicity, race, physical impairment, values, dress, language, or social class (Krink & Gustafson, 1986; NCDA, 1992). If the video is not free from bias, the educational objectives may be greatly effected or compromised. Individuals depicted in the video should not be shown as a role stereotype for the task being enacted or illustrated. "A video lacking a progressive social orientation would also be deficient in objectivity and accuracy of information" (NCDA, 1992, p. 6).

Instructional design models are used to control the design process. These models generically include five steps: analysis, design, development, implementation, and evaluation. During analysis the instructional designer might perform a needs assessment and create a problem statement. The design entails creating a plan of operation that would guide the designer in setting competencies and outcomes, writing objectives, creating assessment strategies and a selection of the proper media (videotapes, texts, facilitation aids, etc.). Development means turning that plan into reality, creating the necessary session plans, study guides, workbooks, job aids, etc. that are needed for delivering the instructional program. When the program is ready it is implemented on a trail basis and evaluated so improvements can be made (Seals & Glasgow, 1990).

Our concern with the selection of the proper video to use in a learning activity is with this design phase. During the design, a plan must be established that results in the learners' needs being met through the use of the video. This plan can be generically outlined as having an introduction, a body, and closure. The introduction should include the objectives of the session, benefits that will be derived from the session, and some sort of "attention getter." The main body should have a presentation of the content and some demonstration or application of the content. It should also allow time for learner reflection on the content and application. Continue this presentation- application-reflection cycle until all the objectives expressed in the introduction are met (Brookfield, 1985, 1986; Friere, 1970; Galbraith, 1991, 1992). Closure should review what has been learned and motivate the learner to apply the content to their lives. Quality indicators for instructional design are organized around this structure and are considered important when evaluating a video's worth for instructional purposes.

The technical quality indicators are derived from good design practices of the producers of video products for instruction. The video industry is charged with producing materials that give quality methods to the instructional process. Characteristics inherent to the medium are: (a) products foster unification and involvement between the viewer and the subject matter, (b) video viewing provides one-way communication that transcends space and time, (c) the viewer is enveloped with sound with visual perspectives, (d) video viewing involves all of the senses simultaneously, and (e) video demands participation from the viewer (Dube, 1980). Hunter (1990) identifies specific characteristics of a quality video product as transcendence, attention manipulation, detail, special effects, economy, independence, and interdependence.

The information accompanying a video is important in the instructional design of the product. It must be accurate and useful to the learner and the facilitator. It must state the purpose of the video, give a summary of the content of the video, clarify any terms or procedures that may not be clear from the video, and provide the learner and facilitator with a guide to using the video (NCDA, 1992).

Video is not a "magic bullet" (Hart, 1984, p. 87). For the product to be of high quality and effective, a program can require considerable facilitation and hard-copy support materials. "Programs that achieve the most successful educational results are known to have supplemental materials that correspond to the [video]" (Hunter, 1990, p. 20).

The main purpose of the evaluation instrument is to allow Extension agents and other reviewers to establish baseline information related to a video before a decision is made to recommend the product for inclusion in a learning event. The instrument can be copied and distributed to several Extension agents, subject matter experts, instructors, and end users for independent review of the video. Each quality indicator is normally weighted the same, however, an evaluation coordinator could weight higher one or more indicators to add emphasis. With this information, the value of the video for instructional purposes can then be quantified.

Brookfield, S. D. (1985). A critical definition of adult education. Adult Education Quarterly, 36, 44-49.

Brookfield, S. D. (1986). Understanding and facilitating adult learning. San Francisco: Jossey-Bass.

Dube, C. (1980). A study in instructional television. Performance and Instruction, 24(5), pp. 24-29.

Ellington, H. (1985). Producing teaching materials. New York: Nichols.

Friere, P. (1970). Pedagogy of the oppressed (M. B. Ramos, Trans.). New York: Herder & Herder. (Original work published 1968).

Galbraith, M. W. (1991). The adult learning transactional process. In M. W. Galbraith (Ed.), Facilitating adult learning: A transactional process (pp. 1-32). Malabar, FL: Krieger.

Galbraith, M. W. (1992). Nine principles of good facilitation. Adult Learning, 3(6), 10-11, 20.

Handbook of Forms. (1985). Irving, TX: International Television Association.

Hart, I. (1984). Video and the control of knowledge. In O. Zuber Skerritt (Ed.), Video in higher education. London: Kogan Page.

Hausman, C. (1990). Institutional video: Planning, budgeting, production and evaluation. Belmont, CA: Wadsworth.

Hunter, P. E. (1990). A designer's guide to script writing: Video capabilities and limitations. Performance and Instruction, 29(3), 18-22.

Hutton, D. (1984). Video technology in higher education: State of the art? In O. Zuber Skerritt (Ed.), Video In Higher Education. London: Kogan Page.

Krink, F. G., & Gustafson K. L. (1986). Instructional technology: A systematic approach to education. New York: Holt, Rinehart, & Winston.

National Career Development Association. (1992). Guidelines for the preparation and evaluation of video career media. Alexandria, VA: Author.

Pett, D. W. (1989). Design of the audio track for slide sets and filmstrips. Performance and Instruction, 28(9), 1-4.

Seels, B., & Glasgow, Z. (1990). Exercises in instructional design. Columbus, OH: Merrill.

Sneed, L. (1991). Evaluating video programs: Is it worth it? White Plains, NY: Knowledge Industry.

Author Notes: This research has been made possible through a grant from the U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, and the National Institute for Occupational Safety and Health.

              Instructional Video Evaluation Instrument

Video Title: ___________________________________________________

Name of Evaluator: _____________________________________________

Phone: ________________________  Date Viewed: __________________

Please rate the video according to the following quality
indicators by CIRCLING one response for each item (1 equals Poor
and 5 equals Exceptional).  Give comments where appropriate.

Poor-----Exceptional

Content

1.  Accurate                                     1  2  3  4  5

       Was the content of the video accurate and up-to-date? If
not, then the video is not ideally suitable for learning. There
may be portions of the content that should NOT be used, as well
as sections that are usable. Please note unusable content in the
space provided or on a separate attachment.

Comments:______________________________________________________

2.  Useful                                        1  2  3  4  5

       Was the content of the video generally useful? The video
should stimulate, motivate and inform the learner to act on the
information that was being presented. Will you incorporate the
ideas presented into your life?

Comments:______________________________________________________

3.  Bias-Free                                     1  2  3  4  5

       Was the video bias-free, including stereotyping with
regard to age, sex, ethnicity, race, physical impairment, values,
dress, language, or social class?

Comments:______________________________________________________

Instructional Plan

4.  Stated the Objectives                         1  2  3  4  5

       Did the video begin with a motivating introduction to
stimulate interest? Were the objectives or key elements made
clear in the introduction?

Comments:______________________________________________________

5.  Content Presentation                          1  2  3  4  5

       Was the content detail controlled to promote
understanding? Did the video simplify complex tasks and avoid
introducing extraneous information? Did it try to cover too much
material or introduce too much detail?

Comments:______________________________________________________

6.  Learner Application                            1  2  3  4  5
       Did the video suggest methods for the learner to apply the
newly acquired knowledge? Were suggestions for practice of what's
being discussed considered? Practice can be designed into the
overall program design as well as into the video itself.

Comments:______________________________________________________

7.  Learner Reflection                            1  2  3  4  5

       Did the video allow for learner reflection? Was
reflection, silence, or time allowed for the learners to react to
a scene or statement? It is also important for the facilitator to
interact with the student to provide feedback on the learner's
application of the material.

Comments:______________________________________________________

8.  Met the Objectives                            1  2  3  4  5

     Did the video meet the learning objectives and needs of the
learner?  Did what was being visually depicted fit the learning
objectives?  As in the introduction, people also remember the
last things that are presented in a program, therefore, did the
video have the key learning elements repeated in the summary or
conclusion.

Comments:______________________________________________________

9.  Learner Interaction                           1  2  3  4  5

       Was the video conducive to learner interaction? Videos can
often be used to promote active learning.

Comments: _____________________________________________________

10. Integration into the Learning Environment     1  2  3  4  5

       Can the video be easily integrated into the learning
environment by adding emphasis to or supplementing more
traditional methods? Did the video bring remote experiences and
places to the learner?

Comments:______________________________________________________

Technical Production

11. General Video Design Characteristics          1  2  3  4  5

       Was the video well planned, organized, and structured?
Was the technology transparent and non-threatening to the
learner? Did the video demonstrate its ability to transcend space
and time? The camera can go where the learner cannot and the
video is an excellent media for presenting information or
demonstrations that are timely, however, care must be taken to
prevent giving a false idea of reality.

Comments:______________________________________________________

12. Focused on Intended Content                   1  2  3  4  5

       Did the video avoid content not related to the subject
matter stated in the introduction? Digressions could lead to
confusion and may be a waste of video time.

Comments:______________________________________________________

13. Visual Quality                                1  2  3  4  5

       Is the camera looking at the scene from the learners'
point of view? This is especially important when psychomotor
skills are being taught. Did the scene changes appear to be
appropriate? Were special effects used to enhance learning by
drawing attention to specific attributes of what is being seen?
Were varying types of camera shots, close-ups to long shots, used
to provide variety in the video?

Comments:______________________________________________________

14. Audio Quality                                 1  2  3  4  5

       Was the vocabulary of the narration appropriate for the
intended audience? Was the speed of the narration slow enough to
be understood? Was the music fitting for the visual affects or
audio narration? Were background noises used that were conducive
to learning? Were sound effects used to add emphasis to the
visual tract of a video to enhance learning?

Comments:______________________________________________________

15. Audio-Visual Relationship                     1  2  3  4  5

       Was the audio-visual combined well? The audio and visual
components should not contradict one another but complement each
other. Were there a variety of differing types of sounds and
visuals to attract and hold attention?

Comments:______________________________________________________

Included Supplemental Materials

16. Provided Introductory Information             1  2  3  4  5

       Did the included supplemental materials include the
purpose and objectives of the video? Did the video accomplish
what is stated in the supplemental materials?

Comments:______________________________________________________

17. Clarifies and Summarizes Content              1  2  3  4  5

       Were job aids or diagrams provided to help in
understanding the material? Were terms defined? Were sources for
further investigation included? Are there suggested activities in
the materials to aid in understanding? Such as, discussion
questions, role plays or simulation exercises. Is the summary
useful in understanding the nature of the video and does it match
what is on the tape?

Comments:______________________________________________________

                    Total (Sum the Scores, 85 Max.) ___________

Additional Comments:---------------------------------------------
-----------------------------------------------------------------
-----------------------------------------------------------------
-----------------------------------------------------------------
------------------------------------------------------