Evaluation Tool for Community Development Coalitions

Introduction

Extension professionals often work with grassroots groups of community members. In this article, we describe an evaluation tool that can assist in tracking the impact of these community coalitions over time.

In 1993, the Oregon legislature created the Oregon Commission on Children and Families (OCCF), a system of government, service providers, non-profits, and community partners to provide family support services. The OCCF created local commissions in each of Oregon's 36 counties and gave them state funding for a variety of projects to benefit children and families.

One local county commission organized grassroots community groups called Community Progress Teams (CPTs) in order to empower each community to identify and address locally prioritized issues. Each CPT signed memorandums of collaboration with the county and received $10,000 that they could allocate to fill local needs. By 1998, 13 CPTs had been established, representing all but one high school service area in the county and covering 1,200 square miles.

Each CPT was composed of community and neighborhood volunteers and representatives from local schools and public safety and human service agencies. CPTs sponsored a variety of activities, including after-school activities, mentoring programs, and family resource centers.

The local commission wanted to know if the CPT model was effective in making a difference in communities. Thus, it contracted with the Extension Service to create an evaluation process that reflected the participatory nature of CPTs. A design team of Extension and health department professionals, CPT volunteers, a university evaluator, and an expert in community development and group process was assembled.

Evaluation Strategy

The evaluation strategy treated each CPT as a case study, in an interactive, retrospective, facilitated evaluation. An evaluation strategy entitled My Community: Then and Now involved both CPT members and key informants (community members who were aware of the local community progress team). Members of CPTs that had been in existence two or more years were asked to respond individually to several statements that related to three goals. Eight of the 13 CPTs were surveyed.

Goal 1: The community supports the health and safety of all children, youth, and families.

1. All families can get any help they need when they need it, to be able to prevent problems, as well as deal with problems when they happen.

2. People are "good neighbors." Residents care about and support children, youth, and families through being neighborly, volunteering, etc.

3. People respect differences between cultures, races, beliefs, values, and lifestyles. All kinds of people are accepted in this community.

4. There are lots of good activities for children and youth in the community, including opportunities for service and leadership.

Goal 2: Community residents, schools, agencies, churches, businesses, and other organizations work together to support children, youth and families.

1. People, schools, organizations, and agencies have good relationships with one another.

2. People, schools, organizations, and agencies coordinate and work together.

3. By working together, people, schools, organizations, and agencies use their resources more effectively.

Goal 3: Community members work together to make plans and decisions about children, youth, and families.

1. People know what programs and services currently exist to help children, youth, and families.

2. People are aware of what additional programs and services are needed to help children, youth, and families.

3. Community goals have been set, and planning has been done to address the needs of children, youth, and families.

4. A wide variety of people and groups participate in community planning and decision making to address children, youth, and families.

A retrospective pretest was utilized (Hill & Betz, 2005; Pratt, McGuigan, & Katzev, 2000; Rockwell & Kohn, 1989). Using a 9-point Likert scale, CPT members assessed each statement (a, b, c, d) before the CPTs began and at the time of the current survey. Next, CPT members were instructed to place colored post-it flags on a wall-sized scale to reflect their rating of each statement. In a manner similar to nominal group technique, the facilitator then led a discussion in which each member shared their ratings and explained the placement of their flags.

The visual impact was dramatic. On some statements flags were clustered. On other statements, flags were spaced along the whole 9-point scale. The facilitated discussion asked members for evidence of the changes that had been observed. After the discussion, any individual could change his or her rating if the dialogue had changed their opinions.

In addition, the facilitators conducted telephone interviews with key informants. Five to eight members of the community who were not CPT members were interviewed. Examples of key informants included school principals, local leaders, police officers, and agency professionals. The same My Community Then and Now survey was utilized. Each informant rated the community relative to the three goals before and after the CPT initiative and identified the role of the CPT in community change.

Results and Conclusion

For data analysis, the final ratings of each CPT participant were utilized. Mean ratings on each goal before and after the CPTs and across CPT members and key informants were compared. CPT members and key informants agreed that community conditions had improved and that CPTs had contributed to these improvements.

Similar to the methodology of Lodl and Stevens (2002), this evaluation tool assesses the impact of a community coalition over time. It includes two unique components: interviews with key informants and a group process that enables coalition members to discuss their ratings. Learning more about sustaining community coalitions is an area of research important to Extension professionals and many private foundations that fund community action projects.

References

Hill, L. G., & Betz, D. L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26, 501-517.

Lodl, K., & Stevens, G. (2002). Coalition sustainability: Long-term successes & lessons learned. Journal of Extension [On-line], 40(1). Available at: http://www.joe.org/joe/2002february/a2.html

Pratt, C. C, McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.

Rockwell, S. K., & Kohn, H. (1989). Post-then-pre evaluation. Journal of Extension [On-line], 27(2). Available at: http://www.joe.org/joe/1989summer/a5.html