The Children, Youth, and Families at Risk (CYFAR) Evaluation Collaboration

Overview

In 1991, USDA-CSREES (Cooperative State Research Education and Extension Service) began funding the Children, Youth, and Families at Risk National Initiative (CYFAR). The initiative emphasizes a holistic approach to preventive educational programs that address risk and resilience factors in children, youth, families, and communities. The goal of the initiative is to facilitate the development and maintenance of healthy environments that enable those at-risk to develop life skills necessary for contributing, fulfilling lives. Community programs are developed based upon ecological principles. They are also based upon the philosophy that community based programs which are developed with active citizen participation in all phases of programming, planning, implementation, and evaluation, will best meet the needs of their communities and will have a higher probability of being sustained (USDA, 1997).

The CYFAR initiative funded a total of 94 community-based projects (with 163 community-based sites) in 49 states and 3 territories beginning in 1991. These Youth-at-Risk projects served approximately 99,000 youth and 17,000 parents. Over 20,000 youth and adult volunteers in communities contributed their time to these programs (CYFAR Annual Report, 1997). The vast majority of these projects (n = 67) reached the end of their granting period in 1996, 25 during 1997 and 2 during 1998, and are no longer directly funded through the initiative. Beginning in 1994, State Strengthening Projects were designed and funded to improve statewide capacity to support as well as improve the quality and quantity of community-based programs for children, youth, and families at-risk. In 1999, 38 State Strengthening projects were renewed and additional ones will be funded through CYFAR.

Another major component of the CYFAR initiative focuses on sustaining and expanding these community programs through a broad variety of supports and technical assistance. The CYFAR Initiative has funded five National Children, Youth, and Family Networks, which merge program and technology resources of the Land Grant universities to provide technical assistance to communities. National Networks focus on Child Care, Family Resiliency, Science and Technology, Collaboration, and Health. CYFERNet, the electronic information infrastructure which links and supports the five networks and assists communities with computer and technology issues, was also funded by the CYFAR Initiative.

Beginning in 1996, the CYFAR Initiative provided computers and electronic access to resources and training in essential technology skills for staff, youth, and adults in communities. CYFAR is in the process of expansion by establishing strategies and objectives essential to sustaining the nation-wide, collaborative, technology-based network of technical assistance to community projects for at-risk children and families (CYFAR Annual Report, 1998).

In an effort to help sustain the work of the CYFAR initiative, a team of Extension professionals and evaluation researchers from universities involved in CYFAR programs were brought together to work with CSREES staff to assess program impact and organizational change, to build the evaluation capacity of Extension, and to provide information on how community-based projects can become institutionalized within CES and sustained within their communities. The CYFAR Evaluation Collaboration assists states and communities in assessing the impact of their prevention education programs for children, youth, families, and communities in order to build the evaluation capacity for each state.

Evaluations are conducted with the following assumptions in mind: evaluation is ongoing, is developmental, matches program complexity with research approaches, is flexible, helps programs to conduct further evaluations, is broad-based, can be conducted internally, is collaborative, and in itself is an intervention. Outcome indicators and assessment instruments for community projects are being developed as well as information concerning the current status of Youth-at-Risk sites and future efforts toward program sustainability.

The work of the CYFAR Evaluation Collaboration focuses on programs within communities and on providing information to states and Cooperative Extension on sustainability and organizational changes. A description of how the Evaluation Collaboration addresses these two areas, through its three components (State Strengthening Evaluation Project, NetCon, and the National Youth at Risk Sustainability Study) is provided.

Supporting Programs in Communities

State Strengthening Evaluation Project

The State Strengthening Evaluation Project works collaboratively with all State Strengthening funded states to enhance the quality of evaluations of community-based programs for children, youth and families. Several evaluation resources were developed as part of this on-going effort and can be accessed at http://ag.arizona.edu/fcr/fs/cyfar/cyfernetsite.htm. The State Strengthening Evaluation Guide (Callor, Betts, Carter, & Marczak, 1997) was developed to help community-based programs evaluate the processes of program development and plan for outcome and impact evaluations.

The Guide makes links between theory and research, program goals and objectives, measures, variables, and program implementation and offers a framework for developing evaluations by presenting a modified version of Jacobs' (1988) five tiered approach to program evaluation. The guide is currently being used by participating states as a step by step tool for conducting program evaluation, as an evaluation training manual, and as a textbook for an undergraduate course on program development and evaluation.

The Evaluation Project team also works with project evaluators in all State Strengthening Projects to develop resources to support evaluations of four key domains of the CYFAR Initiative including: children, youth, parents/families and community. State Strengthening project evaluators join workgroups based on the outcome area(s) that most closely parallels the central intent of their state project. Each of the four national outcome workgroups then relies on the expertise of its members to: identify critical indicators under their respective outcome areas; provide research base/supporting literature for each indicator; and suggest evaluation measures, instruments, and tools. Resources developed from these workgroups are peer reviewed by a project staff, an evaluator, and a researcher with appropriate content expertise.

In addition, the State Strengthening evaluation team developed resources looking at several alternative methods for collecting evaluation data including focus groups, portfolio assessment, cost analysis, qualitative interviews, and using existing records in evaluation. Each document includes a brief literature review describing the method, what one can do and cannot do with the data, advantages and disadvantages, linkages to the tiers of the State Strengthening Evaluation Guide, and step by step "how to" guide. A future topic for this series includes "collecting evaluation information from low-literacy and non-English speaking populations".

Evaluation of CYF Networks, CYFERNet and Electronic Connectivity

NETCON focused its attention on the evaluation of the five Children, Youth and Family Networks (Child Care, Collaboration, Family Resiliency, Health, and Science & Technology), CYFERNet (the electronic internet-based children, youth and family information system), and the electronic connectivity component of the CSREES-USDA funded community based programs. In response to the needs expressed by the five CYF Networks, collaborating universities, and community programs, evaluation resources were developed to document CYF Network and Electronic Connectivity capacity to support community-based programs and to assist the CYF Networks and states conduct program evaluations.

Several evaluation studies were conducted to determine the needs as well as technology use and satisfaction of community and county-based personnel. These include surveys of community and county-based Youth-at-Risk and State Strengthening personnel, a web-based on-line survey, a telephone survey of key-users of the National Networks, and a focus group for non-users of the National Networks. Findings include community and county personnel's attitudes towards and needs regarding computer use as well as their use of electronic resources; and Benefits of using Children, Youth and Family National Networks and of having electronic connectivity for community and county personnel as well as barriers to their use. These findings can be accessed at: http://www.reeusda.gov/4h/cyfar/cyfar.html

Future evaluations focusing on the National Networks, CYFERNet, and electronic Connectivity include interviews with Extension administration to assess perceptions, attitudes and support for the national Networks, CYFERNet, Electronic Connectivity, and the CYFAR Initiative in general, and an electronic survey of State Strengthening Project Directors was conducted and is currently being analyzed.

Sustainability and Organizational Change

National Youth-At-Risk Project Sustainability Study

Demonstrating those factors that contribute to sustainability has become an important part of program development and implementation. There is little written about the topic of sustainability in comparison to many of the other criteria that make programs successful. In short, what is lacking is comprehensive empirical research to better understand what allows programs to become sustained (Lerner, 1995; Schorr, 1997). The CYFAR Evaluation Collaboration focuses its efforts on these dimensions of sustainability.

The National Youth-at-Risk (YAR) Project Sustainability Study examines the resiliency of community-based programs which were funded through grants from CSREES/USDA for a 5-year period beginning in 1991. For the duration of the YAR program, data have been collected that pertain to participation, collaboration, and program impact. This study conducts a broad-based and detailed analysis of these program reports providing closure on projects that are now off federal funding. In addition, this study also provides data that can be reflected back to ongoing projects for the purpose of fine-tuning those programs to increase their likelihood of sustaining themselves post-federal funding. The analysis of these programs and data began in Fall 1996.

Phase one was conducted in 1997 (Mancini & Marek, 1998). This phase of the study examined the meaning of sustainability of YAR projects, which are sustained and which are not, and the role that Extension has had in sustaining these projects. During this phase, first year proposals and close out reports for the sites that have been off funding for one year (n = 67) were examined and analyzed. Telephone interviews were conducted with each site's personnel and community collaborators; interviews were recorded, transcribed, and analyzed using computer qualitative research techniques (QSR NUDI*ST, 1997). These new data have been matched with those originating in reports. A summary report of these findings is available at: http://ext.vt.edu/vce/specialty/famhumdev/350-800.html

The second phase of the study began in 1998 by completing the analysis of those programs that closed-out in 1997 (n = 25) and revisiting those projects that were now off of USDA/YAR funding two years (n = 67). Another major focus during phase two is the development of a Program Sustainability Framework. The development of this framework will be based in the initial analysis of the closed Youth-at-Risk programs as well as requiring the collection of additional data and field testing. A report documenting the status of the Youth-at-Risk projects one and two years post-USDA/YAR funding will be available in spring 1999. Future evaluations will continue to track these Youth at Risk projects to contribute to our understanding of what facilitates and what detracts from sustaining community-based programs. The final phase of the study includes the development of a Community Program Sustainability Assessment Instrument which can be used by community projects in their planning and implementing of factors to assist in sustaining programs.

State Strengthening Evaluation Project

Additionally, to document statewide capacity to support community-based programs for children, youth and families, a 74-item Organizational Change Survey was developed by the State Strengthening Evaluation Project as part of their work with the Evaluation Collaboration. During late 1997 and early 1998 surveys of Extension professionals in 45 states and territories were conducted. Approximately 5,000 individuals responded to the national survey. In addition to individual state reports, results were reported in a national report (Betts, Marczak, Peterson, Sewell, & Lipinski, 1998) which can be accessed at http://ag.arizona.edu/fcr/fs/cyfar/cyfernetsite.htm. An Organizational Change Survey Implementation Kit is being developed to help states continue to document their statewide abilities to develop and sustain effective programs for children, youth and families.

Conclusion

The Evaluation Collaboration is integrating and focusing on improving statewide capacity to support the quality and quantity of comprehensive community based programs for children, youth, and families at risk. Within this philosophy, conducting high quality impact evaluations leading towards sustainability is the guiding force and centerpiece for its work. Far too many high quality programs are unable to continue their important work because they lack the capacity to do so; this initiative is aiming its efforts to assist and provide tools for assessing and building individual program and state's capacity to conduct and provide high quality impact evaluations and sustain themselves.

References

Betts, S. C., Marczak, M.S., Peterson, D. J., Sewell, M., & Lipinski, J. (1998). National results of the organizational change survey. Tucson: University of Arizona Institute for Children, Youth and Families.

Callor, S, Betts, S. C., Carter, R., & Marczak, M. S. (1997). State Strengthening Guide. Tucson: The University of Arizona Institute for Children, Youth & Families.

Children, Youth, and Families at Risk National Initiative Philosophy (1995). Families, 4-H, and Nutrition. Washington: Cooperative State Research, Education, and Extension Service, United States Department of Agriculture.

Children, Youth, and Families at Risk (1997). Annual report. Washington: U.S. Department of Agriculture, Cooperative State Research, Extension and Education Service.

Children, Youth and Families at Risk (1998). Annual report. Washington: U.S. Department of Agriculture, Cooperative State Research, Extension and Education Service.

Jacobs, F. H. (1988). The five-tiered approach to evaluation: Context and implementation. In H. Weiss & F. Jacobs (Eds.), Evaluating family programs (pp. 37-68). NY: Aldine de Gruyter.

Lerner, R. M. (1995). America's youth in crisis. Thousand Oaks, CA: Sage.

Mancini, J., & Marek, L. (1998). Patterns of project survival & organizational support: The national youth at risk program sustainability study. Blacksburg: Virginia Cooperative Extension, #350-800.

QSR NUDI*ST: Software for Qualitative Data Analysis. (1997). Qualitative Solutions and Research Pty Ltd.

Schorr, L. B. (1997). Common purpose: Strengthening families and neighborhoods to rebuild America. NY: Doubleday.

CYFAR Evaluation Collaboration Web Site

All resources developed as part of the CYFAR Evaluation Collaboration will be posted on the CYFAR web site for easy access and wide dissemination. They can be accessed through the CYFAR homepage: http://www.reeusda.gov/4h/cyfar/cyfar.html

For Further Information, contact:

Children, Youth, and Families at Risk Initiative
Sharon K. B. Wright, Chair (swright@reeusda.gov)

State Strengthening Evaluation Project
Sherry Betts or Mary Marczak (sbetts@ag.arizona.edu or marczak@ag.arizona.edu)

NetCon Evaluation
Richard Byrne (rb237@umail.umd.edu)

National Youth at Risk Project Sustainability Study
Jay A. Mancini or Lydia I. Marek (mancini@vt.edu or lmarek@vt.edu)