The Systematic Screening and Assessment Method: An Introduction and Application

Increasingly, Extension professionals are attempting to implement evidence-based practices (Dunifon, Duttweiler, Pillemer, Tobias, & Trochim, 2004). The problem, however, is that in certain areas, including health, programs with evidence of impact are limited (Flynn et al., 2006; The Guide to Community Preventive Services, 2014). Programs are continually being developed by Extension with the intention of finding feasible and effective approaches to preventing negative health outcomes. Extension programs that have demonstrated impact, using appropriate evaluation techniques, should be considered for dissemination, replication, and more thorough evaluation (Fetsch, MacPhee, & Boyer, 2012).

An Introduction to the Systematic Screening and Assessment Method

The Systematic Screening and Assessment Method (SSA) could be a cost-effective and rapid approach to identifying programs with at least preliminary evidence of achieving intended outcomes (Leviton, Khan, & Dawkins, 2010). SSA is an approach that scans the professional landscape, in this case Extension, for promising programs and then assesses these programs using pre-determined criteria. This effort should include programs across numerous sites, such as states, and result in the identification of a subset of programs poised for further evaluation (Leviton & Gutman, 2010).

SSA is a sequential approach that includes the following steps:

1. Solicit a topic;
2. Scan for programs that relate to the topic;
3. Screen the programs identified using pre-determined criteria;
4. Conduct an evaluability assessment of those programs that pass the initial screen;
5. Review and rate interventions for promise/readiness for evaluation; and
6. Use information to inform program evaluation (Leviton & Gutman, 2010).

An Application of the Systematic Screening and Assessment Method

4-H programs in the areas of science, citizenship, and healthy living offer a wealth of research-based, ready-to-use curricula on a variety of topics (National 4-H Council, 2013). The 4-H Healthy Living Mission Mandate involves a holistic approach encompassing the following domains: healthy eating; physical activity; social-emotional health; alcohol, tobacco, and other drug (ATOD) use prevention; and injury prevention. As described below, SSA was implemented by Extension evaluation specialists at Mississippi State University to identify 4-H programs in the areas of nutrition; physical activity; and ATOD prevention that are being developed by Extension and gauge the merit of those programs.

SSA Step 1—Solicit a Topic

In 2013, in collaboration with 4-H National Headquarters, U.S. Department of Agriculture (USDA), National 4-H Council requested proposals to support a project documenting programs that met the following criteria:

• Target 4-H youth, ages 9-19;
• Include a youth development program with an organized, purposeful set of activities designed to achieve positive youth development outcomes;
• Include activities congruent with the 4-H Healthy Living mission as presented in the healthy eating; physical activity; and ATOD use prevention logic models; and
• Were developed and implemented by Cooperative Extension faculty and staff.

SSA Step 2—Scan for Programs Related to the Topic

To identify programs that met the aforementioned criteria, an invitation to participate in a survey was sent to all State 4-H Program Leaders and 4-H Healthy Living liaisons. This environmental scan also included a document review of grantee reports of 4-H Healthy Living projects, as well as a 4-H Healthy Living Literature Review (Hill, McGuire, Parker, & Sage, 2009), in an effort to capture information about noteworthy programs not identified through the survey. As needed, the evaluation specialists verified or clarified reported information with 4-H Program Leaders.

SSA Step 3—Screen the Programs Identified Using Predetermined Criteria

The evaluation specialists leading the project reviewed information collected during the environmental scan to determine which most clearly met the following criteria:

• Specified goals, objectives, activities, and outcomes are realistic, measurable, and logical;
• Outcomes are tied to 4-H Healthy Living logic models;
• Outcomes are assessed with at least a pretest/posttest; and
• Evaluation results are reported.

SSA Step 4—Conduct an Evaluability Assessment of Those Programs That Pass the Initial Screen

An evaluability assessment is used to determine the nature of an evaluation, likelihood of useful results, and opportunities for program improvement (Wholey, 1987). Table 1 reflects a checklist that was used to complete this project's evaluability assessment. Some information for the checklist was collected through the scan. Additional information (as needed) was collected through interviews or email with a representative of programs that met the criteria.

Table 1.
A Checklist Used in Step 4 of the Systematic Screening and Assessment of 4-H Healthy Living Programs

Evaluability Parameters	Key Questions
Program design	Does the program clearly define the problem that it aims to change?
	Are the health needs clearly and explicitly identified? In which categories do the programs fall (Healthy Eating; Physical Activity; and/or Alcohol, Tobacco, and Other Drug Use Prevention)?
	Has the beneficiary population of the program been determined? What is the population?
	Does the program have a clear theory/logic model? What is the theory?
	Is the results framework of the program coherently articulated? Are the outputs, outcomes, and goals logically connected?
	Are the objectives clear and realistic? Are they measurable? Do they respond to the needs identified?
Availability of information	Does the program have the capacity to provide data for evaluation?
	Does the program have SMART (specific, measurable, attainable, realistic, and timely) indicators on key areas of interest?
	Does baseline information exist? If so, what information?
	Does the program have a monitoring system to gather and systematize the evaluation data with defined responsibilities, sources, and periodicity?
	What are the likely costs of such data collection and analysis?
	What kind of information do the key stakeholders request?
Conduciveness of the context	Is the context conducive to conduct the evaluation, both external and internal to the program?
	What resources are available to undertake the evaluation, such as well trained staff, financial resources, equipment?
	What evaluation capacities and expertise exist to undertake the evaluation from the 4-H Healthy Living perspective?
Note: This checklist was adapted from a checklist developed by the United Nations Development Fund for Women (2009).

SSA Step 5—Review and Rate Interventions for Promise/Readiness for Evaluation

Programs that met the criteria for inclusion in the evaluability assessment were classified as having preliminary, moderate, or strong evidence of replicability using definitions provided by the Corporation for National and Community Service (2012). Specifically, classification was documented on the following briefly-explained evaluation design:

Preliminary evidence—outcome studies using a pretest/posttest study design;

Moderate evidence—at least one experimental/quasi-experimental study design, or correlational research that controlled for selection bias; or

Strong evidence—numerous quasi-experimental studies, or one large, well-designed and well-implemented randomized controlled trial implemented over multiple sites.

SSA Step 6— Use Information to Inform Program Evaluation

Findings and program summaries developed through the project will be disseminated through professional development conferences/workshops held for Extension leadership, specialists, and agents/educators. Additionally, USDA and National 4-H Council could work together with program representatives to identify 4-H "'signature" programs. Evaluation specialists on this project will offer technical assistance on evaluation to representatives of programs identified to enhance the skills of Extension personnel and to strengthen evidence of program replicability so other promising 4-H Healthy Living programs can be disseminated for utilization throughout the nation.

Conclusions

As demonstrated through our application, SSA is a clear, straightforward technique. The approach can be tailored to the specific context and resources available for conducting SSA. Like any evaluation method, SSA has limitations:

• The list of programs identified in an environmental scan might not be exhaustive;
• Program information collected through the scan could be incomplete or incorrect;
• It is difficult to obtain parallel program information when using different screening methods; and
• SSA requires a certain level of evaluation expertise that some Extension services lack.

Despite these limitations, SSA can yield meaningful results on emerging programs that are worthy of more rigorous evaluation.

Acknowledgements

This work was financially supported by National 4-H Council and Molina Healthcare. Special thanks to State 4-H Program Leaders and 4-H Healthy Living staff for submitting information about programming; and National 4-H Healthy Living Management Team members for input on development of the environmental scan survey.

References

Corporation for National and Community Service. (2012). Overview of funding opportunity: Social innovation fund [Notice of federal funds availability]. Retrieved from: http://www.nationalservice.gov/pdf/12_0210_sif_nofa.pdf

Dunifon, R., Duttweiler, M., Pillemer, K., Tobias, D., & Trochim, W. M. K. (2004).Evidence-based Extension. Journal of Extension [On-line], 42(2) Article 2FEA2. Available at: http://www.joe.org/joe/2004april/a2.php

Fetsch, R. J., MacPhee, D., & Boyer, L. K. (2012). Evidence-based programming: What is a process an Extension agent can use to evaluate a program's effectiveness? Journal of Extension [On-line], 50(5) Article 5FEA2. Available at: http://www.joe.org/joe/2012october/a2.php

Flynn, M.A., McNeil, D.A., Maloff, B., Mutasingwa, D., Wu, M., Ford, C., & Tough, S.C. (2006). Reducing obesity and related chronic disease risk in children and youth: A synthesis of evidence with 'best practice' recommendations. Obesity Review, 7 (Suppl. 1), 7–66. doi:10.1111/j.1467-789X.2006.00242.x

The Guide to Community Preventive Services. (2014). The community guide. Retrieved from: http://www.thecommunityguide.org/index.html

Hill, L., McGuire, J., Parker, L., & Sage, R. (2009). 4-H healthy living literature review and recommendations for program planning and evaluation. Retrieved from: http://www.4-h.org/resource-library/professional-development-learning/healthy-living-training-guides-resources/

Leviton, L. C., & Gutman, M. A. (2010). Overview and rationale for the systematic screening and assessment method. New Directions for Evaluation, 2010 (125), 7–31. doi:10.1002/ev.318

Leviton, L. C., Khan, L. K., & Dawkins, N. (2010). Editors' notes. New Directions for Evaluation, 2010 (125), 1–5. doi:10.1002/ev.317

National 4-H Council. (2013). 4-H youth development programs. Retrieved from: http://www.4-h.org/youth-development-programs/

Peterson, D. J., Baker, B. A., Leatherman, J., Newman, M. E., & Miske, S. (2012). Engaging youth, serving community: Social change lessons from a 4-H rural youth development program. Retrieved from: http://cyber.law.harvard.edu/sites/cyber.law.harvard.edu/files/KBWLessonsfrom4H2012.pdf

United Nations Development Fund for Women. (2009). Guidance notes on carrying out evaluability assessment. Retrieved from: http://erc.undp.org/unwomen/resources/guidance/Guidance%20Note%20-%20Carrying%20out%20an%20Evaluability%20Assessment.pdf

Wholey, J. (1987). Evaluability assessment. New Directions for Program Evaluation, 33, 77-92.