April 2015 // Volume 53 // Number 2 // Ideas at Work // v53-2iw2
The Systematic Screening and Assessment Method: An Introduction and Application
The Systematic Screening and Assessment method (SSA) could be a useful approach to identifying Extension programs being implemented, to describing the nature of those programs, and to highlighting those that have proven effective. After a general discussion of SSA, we provide an application of SSA as recently used to identify 4-H Healthy Living programs with evidence of improving diet; physical activity; and alcohol, tobacco, and other drug use prevention outcomes. Our goal is to increase awareness about SSA and to demonstrate that it can be used to expand the knowledge and reach of Extension programs with evidence of impact.
Increasingly, Extension professionals are attempting to implement evidence-based practices (Dunifon, Duttweiler, Pillemer, Tobias, & Trochim, 2004). The problem, however, is that in certain areas, including health, programs with evidence of impact are limited (Flynn et al., 2006; The Guide to Community Preventive Services, 2014). Programs are continually being developed by Extension with the intention of finding feasible and effective approaches to preventing negative health outcomes. Extension programs that have demonstrated impact, using appropriate evaluation techniques, should be considered for dissemination, replication, and more thorough evaluation (Fetsch, MacPhee, & Boyer, 2012).
An Introduction to the Systematic Screening and Assessment Method
The Systematic Screening and Assessment Method (SSA) could be a cost-effective and rapid approach to identifying programs with at least preliminary evidence of achieving intended outcomes (Leviton, Khan, & Dawkins, 2010). SSA is an approach that scans the professional landscape, in this case Extension, for promising programs and then assesses these programs using pre-determined criteria. This effort should include programs across numerous sites, such as states, and result in the identification of a subset of programs poised for further evaluation (Leviton & Gutman, 2010).
SSA is a sequential approach that includes the following steps:
- Solicit a topic;
- Scan for programs that relate to the topic;
- Screen the programs identified using pre-determined criteria;
- Conduct an evaluability assessment of those programs that pass the initial screen;
- Review and rate interventions for promise/readiness for evaluation; and
- Use information to inform program evaluation (Leviton & Gutman, 2010).
An Application of the Systematic Screening and Assessment Method
4-H programs in the areas of science, citizenship, and healthy living offer a wealth of research-based, ready-to-use curricula on a variety of topics (National 4-H Council, 2013). The 4-H Healthy Living Mission Mandate involves a holistic approach encompassing the following domains: healthy eating; physical activity; social-emotional health; alcohol, tobacco, and other drug (ATOD) use prevention; and injury prevention. As described below, SSA was implemented by Extension evaluation specialists at Mississippi State University to identify 4-H programs in the areas of nutrition; physical activity; and ATOD prevention that are being developed by Extension and gauge the merit of those programs.
SSA Step 1—Solicit a Topic
In 2013, in collaboration with 4-H National Headquarters, U.S. Department of Agriculture (USDA), National 4-H Council requested proposals to support a project documenting programs that met the following criteria:
- Target 4-H youth, ages 9-19;
- Include a youth development program with an organized, purposeful set of activities designed to achieve positive youth development outcomes;
- Include activities congruent with the 4-H Healthy Living mission as presented in the healthy eating; physical activity; and ATOD use prevention logic models; and
- Were developed and implemented by Cooperative Extension faculty and staff.
SSA Step 2—Scan for Programs Related to the Topic
To identify programs that met the aforementioned criteria, an invitation to participate in a survey was sent to all State 4-H Program Leaders and 4-H Healthy Living liaisons. This environmental scan also included a document review of grantee reports of 4-H Healthy Living projects, as well as a 4-H Healthy Living Literature Review (Hill, McGuire, Parker, & Sage, 2009), in an effort to capture information about noteworthy programs not identified through the survey. As needed, the evaluation specialists verified or clarified reported information with 4-H Program Leaders.
SSA Step 3—Screen the Programs Identified Using Predetermined Criteria
The evaluation specialists leading the project reviewed information collected during the environmental scan to determine which most clearly met the following criteria:
- Specified goals, objectives, activities, and outcomes are realistic, measurable, and logical;
- Outcomes are tied to 4-H Healthy Living logic models;
- Outcomes are assessed with at least a pretest/posttest; and
- Evaluation results are reported.
SSA Step 4—Conduct an Evaluability Assessment of Those Programs That Pass the Initial Screen
An evaluability assessment is used to determine the nature of an evaluation, likelihood of useful results, and opportunities for program improvement (Wholey, 1987). Table 1 reflects a checklist that was used to complete this project's evaluability assessment. Some information for the checklist was collected through the scan. Additional information (as needed) was collected through interviews or email with a representative of programs that met the criteria.
|Evaluability Parameters||Key Questions|
|Program design||Does the program clearly define the problem that it aims to change?|
|Are the health needs clearly and explicitly identified? In which categories do the programs fall (Healthy Eating; Physical Activity; and/or Alcohol, Tobacco, and Other Drug Use Prevention)?|
|Has the beneficiary population of the program been determined? What is the population?|
|Does the program have a clear theory/logic model? What is the theory?|
|Is the results framework of the program coherently articulated? Are the outputs, outcomes, and goals logically connected?|
|Are the objectives clear and realistic? Are they measurable? Do they respond to the needs identified?|
|Availability of information||Does the program have the capacity to provide data for evaluation?|
|Does the program have SMART (specific, measurable, attainable, realistic, and timely) indicators on key areas of interest?|
|Does baseline information exist? If so, what information?|
|Does the program have a monitoring system to gather and systematize the evaluation data with defined responsibilities, sources, and periodicity?|
|What are the likely costs of such data collection and analysis?|
|What kind of information do the key stakeholders request?|
|Conduciveness of the context||Is the context conducive to conduct the evaluation, both external and internal to the program?|
|What resources are available to undertake the evaluation, such as well trained staff, financial resources, equipment?|
|What evaluation capacities and expertise exist to undertake the evaluation from the 4-H Healthy Living perspective?|
|Note: This checklist was adapted from a checklist developed by the United Nations Development Fund for Women (2009).|
SSA Step 5—Review and Rate Interventions for Promise/Readiness for Evaluation
Programs that met the criteria for inclusion in the evaluability assessment were classified as having preliminary, moderate, or strong evidence of replicability using definitions provided by the Corporation for National and Community Service (2012). Specifically, classification was documented on the following briefly-explained evaluation design:
Preliminary evidence—outcome studies using a pretest/posttest study design;
Moderate evidence—at least one experimental/quasi-experimental study design, or correlational research that controlled for selection bias; or
Strong evidence—numerous quasi-experimental studies, or one large, well-designed and well-implemented randomized controlled trial implemented over multiple sites.
SSA Step 6— Use Information to Inform Program Evaluation
Findings and program summaries developed through the project will be disseminated through professional development conferences/workshops held for Extension leadership, specialists, and agents/educators. Additionally, USDA and National 4-H Council could work together with program representatives to identify 4-H "'signature" programs. Evaluation specialists on this project will offer technical assistance on evaluation to representatives of programs identified to enhance the skills of Extension personnel and to strengthen evidence of program replicability so other promising 4-H Healthy Living programs can be disseminated for utilization throughout the nation.
As demonstrated through our application, SSA is a clear, straightforward technique. The approach can be tailored to the specific context and resources available for conducting SSA. Like any evaluation method, SSA has limitations:
- The list of programs identified in an environmental scan might not be exhaustive;
- Program information collected through the scan could be incomplete or incorrect;
- It is difficult to obtain parallel program information when using different screening methods; and
- SSA requires a certain level of evaluation expertise that some Extension services lack.
Despite these limitations, SSA can yield meaningful results on emerging programs that are worthy of more rigorous evaluation.
This work was financially supported by National 4-H Council and Molina Healthcare. Special thanks to State 4-H Program Leaders and 4-H Healthy Living staff for submitting information about programming; and National 4-H Healthy Living Management Team members for input on development of the environmental scan survey.
Corporation for National and Community Service. (2012). Overview of funding opportunity: Social innovation fund [Notice of federal funds availability]. Retrieved from: http://www.nationalservice.gov/pdf/12_0210_sif_nofa.pdf
Dunifon, R., Duttweiler, M., Pillemer, K., Tobias, D., & Trochim, W. M. K. (2004).Evidence-based Extension. Journal of Extension [On-line], 42(2) Article 2FEA2. Available at: http://www.joe.org/joe/2004april/a2.php
Fetsch, R. J., MacPhee, D., & Boyer, L. K. (2012). Evidence-based programming: What is a process an Extension agent can use to evaluate a program's effectiveness? Journal of Extension [On-line], 50(5) Article 5FEA2. Available at: http://www.joe.org/joe/2012october/a2.php
Flynn, M.A., McNeil, D.A., Maloff, B., Mutasingwa, D., Wu, M., Ford, C., & Tough, S.C. (2006). Reducing obesity and related chronic disease risk in children and youth: A synthesis of evidence with 'best practice' recommendations. Obesity Review, 7 (Suppl. 1), 7–66. doi:10.1111/j.1467-789X.2006.00242.x
The Guide to Community Preventive Services. (2014). The community guide. Retrieved from: http://www.thecommunityguide.org/index.html
Hill, L., McGuire, J., Parker, L., & Sage, R. (2009). 4-H healthy living literature review and recommendations for program planning and evaluation. Retrieved from: http://www.4-h.org/resource-library/professional-development-learning/healthy-living-training-guides-resources/
Leviton, L. C., & Gutman, M. A. (2010). Overview and rationale for the systematic screening and assessment method. New Directions for Evaluation, 2010 (125), 7–31. doi:10.1002/ev.318
Leviton, L. C., Khan, L. K., & Dawkins, N. (2010). Editors' notes. New Directions for Evaluation, 2010 (125), 1–5. doi:10.1002/ev.317
National 4-H Council. (2013). 4-H youth development programs. Retrieved from: http://www.4-h.org/youth-development-programs/
Peterson, D. J., Baker, B. A., Leatherman, J., Newman, M. E., & Miske, S. (2012). Engaging youth, serving community: Social change lessons from a 4-H rural youth development program. Retrieved from: http://cyber.law.harvard.edu/sites/cyber.law.harvard.edu/files/KBWLessonsfrom4H2012.pdf
United Nations Development Fund for Women. (2009). Guidance notes on carrying out evaluability assessment. Retrieved from: http://erc.undp.org/unwomen/resources/guidance/Guidance%20Note%20-%20Carrying%20out%20an%20Evaluability%20Assessment.pdf
Wholey, J. (1987). Evaluability assessment. New Directions for Program Evaluation, 33, 77-92.