Evaluating a Domestic Violence Task Force: Methods to Strengthen a Community Collaboration

Introduction and Background

The Chittenden County Domestic Violence Task Force (DVTF) is a community collaboration comprised of 50 organizations and individuals, including victim advocacy agencies, law enforcement, government, courts, and refugee and immigrant service providers. The membership of the DVTF provides services to Chittenden County in Vermont, with a population of 146,571, encompassing 17 rural towns and one urban area (U.S. Census Bureau, 2000).

Even though the DVTF meets monthly to discuss current issues, grants, and projects, it is not without its challenges. Loss of grant funding in 2000 and turnover in individual participation led to centralized leadership and diminished meeting attendance and commitment to work on projects.

In response, the Center for Rural Studies (CRS) at the University of Vermont was contracted to evaluate the collaborative and make recommendations for strengthening membership, organizational structure, function, and cohesiveness. The evaluation was conducted as a case study, examining what characteristics hold the DVTF together, what can be strengthened to improve the collaborative, and how to directly link evaluation research to make real improvements to the community collaborative. The case study is presented in two parts:

1. How the evaluation was conducted and key findings, and

2. Recommendations made for improvement and the outcome of their implementation.

Collaboration Theory and Domestic Violence

The establishment of collaborations is one model to cope with complex problems as communities face issues that cross the disciplines of service providers, professionals, and government systems (Bradshaw, 2000). Further, people facing complex issues, such as domestic violence, often utilize multiple services to meet their needs. Through collaborations, organizations work together across multiple sectors to challenge complex issues facing their community and seek change (Borden & Perkins, 1999; Clark, Burt, Schulte, & Maguire, 1996; Fishman, Farrell, Allen, & Eiseman, 2000; Gray, 1989; Perkins, 2002; Shepard, 1999; Taylor-Powell & Rossing, 1998).

Domestic violence is a complex issue that has been increasingly responded to by agencies that collaborate in a coordinated community response (Shepard, 1999). A study of coordinated community responses in six communities throughout the United States reported that all had some level of collaboration or council to address domestic violence issues in a coordinated way (Clark et al., 1996).

Characteristics of a Strong Collaborative

Several researchers have identified characteristics of a strong collaborative with focus on member involvement, leadership, goal setting and decision-making, and accountability and impact (Allen & Hagen, 2003; Bradshaw, 2000; Fishman et al., 2000; Hogue, Perking, Clark, Bergstrum, & Slinski, 1995; Taylor-Powell & Rossing, 1998).

Member Involvement

Responding to current community issues requires a collaborative to have an active and diverse membership base across disciplines and sectors of society and active subcommittees that focus on different issues. Membership should include the criminal justice system, victim advocacy, and programs for perpetrators, as well as other agencies, such as health care facilities, child welfare, substance abuse treatment, and religious institutions (Clark et al., 1996; Shepard, 1999). Membership should also be flexible and continuously evolving. However, collaboratives may have difficulty diversifying and engaging members, because they are often faced with administrative challenges and spread their resources too thin to maintain active involvement (Bradshaw, 2000; Taylor-Powell & Rossing, 1998).

Several researchers identify characteristics of a successful collaborative to maximize member involvement and resources, including: shared vision and responsibility, diverse membership, voluntary participation, and point of views from multiple perspectives (Allen & Hagen, 2003; Fishman et al., 2000; Hogue et al., 1995; Taylor-Powell & Rossing, 1998). Fishman et al. (2000) state that a shared vision is needed to address mutually identified member needs. Members may have different agendas, but work to develop a common mission to address the needs of the collaborative (Allen & Hagen, 2003; Fishman et al., 2000).

Leadership

Allen and Hagen (2003) and Hogue et al. (1995) state that leadership is a key characteristic of successful collaboratives. Fishman et al. (2000) state that skilled leadership is important to provide direction and sustain the collaboration. Individuals who have taken on a leadership role within the DVTF work directly with domestic violence victims or offenders, and their work relies on the collaboration with other agencies through the DVTF. Leaders should take measures to ensure that the balance between member needs and expected outcomes of the collaborative is maintained. However, leadership can be difficult, because administrative time and coordination are often unnecessarily high (Bradshaw, 2000; Taylor-Powell & Rossing, 1998).

Goal Setting and Decision-Making

Bradshaw (2000) and Taylor-Powell and Rossing (1998) state that most collaboratives have difficulty in goal setting and decision-making, because they often have vague or broad goals surrounding their mission. A strong collaborative requires a process for shared decision-making needs that includes member needs while focusing on the agreed upon mission (Allen & Hagen, 2003). The agenda should be driven by the membership and continually assess whether or not the needs of its members are being addressed. This is crucial because members who feel they are vested in the collaboration are more likely to help the collaboration grow (Fishman et al., 2000).

Accountability and Improvement

Fishman et al. (2000) state that the collaboration should be held accountable by an evaluative process that monitors their progress towards reaching specified outcomes and recommends changes for improving and strengthening. Changes should be made in goals and programs and activities through sub-committees when necessary (Taylor-Powell & Rossing, 1998).

Part I: Evaluation of the Collaborative and Key Findings

This section reviews the evaluation methodology and key findings of the study. The purpose of the case study was to develop, evaluate, and implement recommendations to strengthen the DVTF membership, organization, function, and cohesiveness. The case study examined what characteristics strengthen the DVTF and what needs strengthening to improve the collaborative. Based on these findings, recommendations were made and implemented by the DVTF, as discussed in Part II of this article. Research questions included:

• To what extent and why is each organization involved?

• How does the DVTF benefit their organization?

• What are member needs and expectations of the DVTF?

• How does the DVTF rate using the Needs and Expectations Scale?

• How does the DVTF rate using the Characteristics of a Successful Community Collaborative Scale?

• What changes can be made to improve and strengthen membership, organizational structure, function, and cohesiveness?

• How are these changes implemented and do these changes prove to be a successful method for strengthening a community collaborative?

Methodology

Many researchers have developed tools to evaluate a collaborative. These include checklists, frameworks, self-assessments, and scales that measure how a collaborative compares to key characteristics (Allen & Hagen, 2003; Borden & Perkins, 1999; Butterfoss, 1998; Clark et al., 1996; Fishman et al., 2000; Shepard, 1999; Taylor-Powell & Rossing, 1998). These are process evaluation tools, to determine how the administration and organization of the collaborative, group dynamics, leadership, sub-committees, and activities are carried out to effectively bring about systems change (Allen & Hagen, 2003; Fishman et al., 2000; Perkins, 2002).

The study of the DVTF was funded under a grant awarded by the U.S. Department of Justice. CRS developed a survey instrument with assistance from key DVTF members, based on a previous DVTF evaluation, member interviews, and literature on evaluating coordinated community responses and collaborations (Allen & Hagen, 2003; Borden & Perkins, 1999; Butterfoss, 1998; Clark et al., 1996; Fishman et al., 2000; Shepard, 1999; Taylor-Powell & Rossing, 1998). Open-ended questions were asked, and the DVTF was rated on a Needs and Expectations Scale (Cronbach Alpha score is .90 for 21 statements) and Characteristics of a Successful Collaborative Scale (Cronbach Alpha score is .80 for 11 statements). Cronbach's Alpha scores were calculated for each scale, confirming the scale's reliability. An Alpha of .70 or greater is considered to be acceptable (Nunnally, 1978).

DVTF members were mailed a cover letter, the survey, and return envelope. The survey was also made available through an online form. Fifty surveys were mailed out to all individuals on the DVTF mailing list, with three follow-up communications through meetings and email. Because the survey was sent out to individuals, multiple individuals from a single agency could respond if they participated in the DVTF.

Nineteen completed surveys were received for a response rate of 38%. All "core members" of the DVTF who are most actively involved and present at all meetings responded (12), along with seven members who reported limited activity. Based on the results from the survey, CRS developed a revised meeting structure and sub-committee criteria and added a quarterly meeting to encourage member and public participation in the DVTF, given differing commitment levels. This meeting structure model was implemented by the DVTF, and positive feedback has been received about it as a method to strengthen collaboratives.

Results and Findings

Member Involvement

DVTF agencies are involved in the collaborative because they provide direct services in domestic violence and desire to be a part of a coordinated community response. Further, the DVTF provides a forum for networking and information and resource sharing, and assists them to improve their services. Individuals are involved because of their job affiliation, personal interest, and commitment, and to network and communicate with colleagues about domestic violence issues.

Sixty-three percent (n=12) reported regularly attending DVTF meetings, 26% (n=5) have minimal activity, and 11% (n=2) do not attend meetings. Those with minimal activity remain updated through meeting minutes and the listserv. The main reasons given for limited participation include: schedule conflicts, limited staff, and limited time to allocate for meetings. Several would attend meetings more often if the focus of the DVTF was more relevant to their work or if the meeting time were different.

Benefit of the DVTF to Member Organizations

The majority of member organizations valued the benefit of being a part of a network and having regular contact with other domestic and sexual violence service providers. Members also valued being kept informed of current issues related to domestic violence, involved in a forum to address concerns and communication, and a part of a coordinated community response to domestic violence. All of these benefits reflect member involvement and goal setting and decision-making characteristics of collaboration (Fishman et al., 2000; Hogue et al., 1995; Taylor-Powell & Rossing, 1998).

Needs and Expectations Scale

Using the Needs and Expectations Scale, members were asked to indicate which areas within the DVTF aligned with their needs and expectations, and to rank the DVTF in meeting these needs from "poor," "fair," "good," to "excellent." This scale was developed to determine if there were member needs and expectations that were not being met. Table 1 indicates the areas that are of a high, moderate, and low level of need to the DVTF (high level = 75% indication; moderate = 50% to 74%; low level = < 49%). Table 1 shows that the areas of highest need and expectation of the DVTF regard member involvement. These areas were ranked as "good." These include improving communication among member organizations and networking, which correspond to the main reasons why members are involved with the DVTF.

Areas listed as a moderate need and expectation of the DVTF and ranked as "good" also correspond to member involvement. These areas include: providing an umbrella organization and a forum for raising issues, meeting minutes, and mobilizing various community organizations for advocacy and systems change. However, areas listed as a moderate need and expectation and ranked as "fair" relate to goal setting and decision-making. These areas include: developing goals, improving community awareness and education, mobilizing community organizations, achieving systems change, project development, addressing organization specific agenda, and mobilizing community response to a current event.

Areas that were indicated to be of low need and expectation of the DVTF include: training, funding support, and enhancing services provided by organization. These areas were ranked from "poor" to "good." None of the areas of need and expectation received a modal rank of "excellent."

Characteristics of Successful Community Collaboration Scale

Using the Characteristics of Successful Community Collaboration Scale, based on the research of Fishman et al (2000), members ranked the DVTF in comparison to each characteristic, using the scale of "poor," "fair," "good," and "excellent" (Table 2). The boxes in gray indicate the majority rank. Areas that received high rankings, where 60% or more members ranked the area as "good" to "excellent," relate to member involvement and goal setting and decision-making. Areas that received moderate rankings, where 60% or more ranked the areas as "fair" to "good," relate to membership involvement, leadership, and accountability. Areas that received low rankings, where 60% or more ranked the areas as "poor" to "fair," were also related to member involvement and goal setting and decision-making. Low-ranked areas included development of an action plan, diversified membership, use of a membership driven agenda, and sub-committee activity.

Suggestions to Improve the DVTF Function and Membership

Members identified several areas to improve DVTF function and increase membership, including:

• Develop a shared vision and action plan;

• Provide clear incentives for participation through more outreach and educational events;

• Increase and diversify membership;

• Report data and outcomes to keep members informed of organization statistics;

• Improve sub-committee structure; and

• Focus on achieving outcomes through active sub-committees.

Part II. Recommendations to Strengthen the Collaborative

This section reviews the recommendations developed by CRS, based on the results of the survey conducted to strengthen the collaborative. This section also reviews the outcome of implementing the recommendations and the implications of the findings for Extension professionals and collaboratives.

Recommendations to improve the DVTF were determined based on specific areas that received a "fair" or "poor" rank on the Needs and Expectations Scale and the Characteristics of Successful Community Collaboration Scale. Further, recommendations were made based on reasons why members do not consistently attend meetings and specific suggestions members made to improve the DVTF function and membership. These areas include: increased information sharing, utilization of a membership driven agenda and goal setting, increase relevance of meetings to all members, shifting of meeting time, improvement of sub-committees, project development, diversified membership, community awareness and education, mobilization of community organizations, and community response to current events.

In response to these areas, CRS recommended a revised meeting structure and format, and parameters for establishing a sub-committee, as well as the addition of a quarterly meeting to focus on specific topics in the form of an event or outreach. The overall intent of these recommendations is to strengthen the DVTF collaborative, so that members feel more vested in the DVTF, increase member involvement, and enhance current efforts towards system change in the areas of domestic and sexual violence.

Monthly Meeting Changes

The main reasons cited for low meeting attendance included schedule conflicts, limited staff, and limited time to attend meetings, and the need for topics to be more relevant to participating agencies and individuals. Members also wanted DVTF meeting agendas to be more driven by member and agency needs.

Several changes were recommended to revise the monthly meeting schedule and format, based on these needs. The time of the monthly meeting was recommended to be moved back a half an hour to better accommodate schedules. Meeting agenda-setting and topics for discussion were recommended to be driven by specific member needs and recommendations, based on agency, sub-committee, or coordinator reports. For instance, several members identified a gap in services for underserved populations, such as non-English speaking populations. Thus, a meeting should be devoted to discussing this topic and the needs of this community.

The monthly meeting agenda structure was also revised. Each meeting should begin with a brief agency report and check-in. This addresses member needs of sharing information, reporting data and outcomes to keep members informed of organization statistics, networking, and identifying gaps in services. It also helps determine trends or challenges that providers face, which may result in DVTF action or the development of a sub-committee (see below). Sub-committee reporting and discussion of main meeting issues should follow agency reporting. Finally, any questions and discussion on topics brought up during the agency reports should be revisited at the end, to determine if the DVTF wants to further address this issue.

Sub-Committee Changes

Responses to the survey also identified that the DVTF sub-committee structure should be strengthened so that sub-committees may be better used as a vehicle to improve and diversify agency involvement, develop projects, and achieve outcomes that will affect the community for systems change. Thus, CRS made several revisions to their formation and structure. Under this new format, DVTF sub-committees will be in a better position to play a key role in the monthly meetings and agenda setting and carry out needed activities. To initiate the process, CRS recommended that the Executive Committee of the DVTF should broaden its membership, with assistance from the DVTF Coordinator, to include representatives from state agencies.

CRS also recommended that the other current and future sub-committees should take one of two formats: 1) ongoing to address continuous issues (i.e., Grants Committee) and 2) time limited to address issues in a limited time frame (i.e., Domestic Violence Awareness Month Sub-Committee). The following parameters should be met to form a sub-committee and ensure it has adequate participation and activity to address its issues so that it may play a role in agenda setting for DVTF meetings and project development for new initiatives.

• A Sub-Committee is needed to address an issue in depth;

• Members are available and interested to participate in the sub-committee;

• A Chairperson is identified to facilitate, provide leadership, and speak on its behalf;

• The Sub-committee meets on a consistent basis, with an identified date, time and location for each meeting; and

• A process is followed to address the selected issue, including:

  • Development of an objective;
  • Determination of how to achieve the objective and if the larger DVTF is to be involved;
  • Determination of how to receive input and feedback on the activities; and
  • Determination of how and when to share the outcome.

Addition of a Quarterly Meeting

As a strategy to increase member involvement, diversify membership, improve community education, and mobilize the community around a specific domestic violence issue to work towards systems change, CRS recommended that the DVTF institute a quarterly meeting on an annual basis, in place of that respective monthly meeting. The quarterly meetings should be action-oriented around a specific topic and held in the form of an event or forum for wider community participation. This meeting will serve to increase the involvement of current members, specifically those with limited time and low level of commitment, by offering them an important meeting to attend four times a year rather than 12.

Research shows that this type of meeting structure will increase meeting attendance by less active members (Cranwell & Schmidt, 2002). Furthermore, agencies not typically affiliated with the DVTF, including Congressional representatives and the media, should be invited to attend the event. This will broaden community involvement and increase awareness of specific domestic violence issues of direct relevance to the community, without requiring people to commit to attending regularly meetings.

Outcome of Recommendations

The meeting structure, format, and agenda-setting process were tested and implemented with success during the August and September 2003 meetings. CRS received positive feedback that members felt more involved and informed because of the agency reporting period and voted to institute this process as a formal part of the beginning of every meeting. Members also identified the desire to hold the first quarterly meeting in October 2003, in coordination with Domestic Violence Awareness Month. This meeting would address issues identified by member agencies, specifically gaps in services for underserved populations. A sub-committee was formed to organize the quarterly meeting, based on the new parameters.

The first quarterly meeting was successfully held in October 2003. A morning press conference was held, followed by a reception and meeting to further discuss the issues addressed. At the press conference, DVTF agencies highlighted barriers and gaps in services for various underserved populations, how these gaps will be met, and strategies to meet unmet needs. At the meeting, community members were invited to discuss how they and their agencies could assist in developing solutions to these areas of concerns.

Thirty agencies and individuals attended this quarterly meeting, of which 43% (13) were "core members" and 57% (17) were new attendees. The total attendance was high considering that attendance at monthly DVTF meetings in 2002 ranged from 10 to 15 people. In addition to new people being exposed to the DVTF, positive feedback was received from DVTF members and the larger community about the importance of educating, information sharing, and networking through this type of meeting to work toward systems change. The October 2003 meeting discussion was used to generate the 2004 DVTF Action Plan. Based on this outcome, the DVTF instituted that each October quarterly meeting would be used to develop the annual plan for the successive year.

Implications of Findings

Implications to the process and outcome findings of this case study evaluation include the following.

1. Extension and other professionals may find the survey instrument and scales useful for evaluating a community collaborative. The identification and ranking of member needs and expectations, and ranking the performance of the collaborative against successful characteristics can be adopted to other fields, in addition to domestic violence.

2. The outcome of the study is useful to Extension and other professionals working with a collaborative. Changes in the meeting structure and format, agenda-setting methods, and sub-committee parameters can assist a collaborative to hold meetings that are goal and agenda driven and meet member needs. It is crucial for the agenda to be driven by the membership, because members who feel vested in the collaboration are more likely to stay involved and help the collaboration grow (Fishman et al., 2000).

3. The quarterly meeting structure enables a collaborative to invite less active and outlying community members to meetings that address pertinent community issues. Quarterly meetings provide a forum for goal-setting and decision-making that is membership driven, broadens community involvement without requiring regular commitment, educates the larger community, and ultimately holds the collaborative and individual members accountable for their actions. These are characteristics of a successful collaborative (Bradshaw, 2000; Fishman et al., 2000; Hogue et al., 1995; Taylor-Powell & Rossing, 1998).

4. This case study links research with practice, demonstrating how evaluation findings can directly lead to real improvements in a community. Overall, this model of improving a collaborative through evaluation enables collaborative members to better serve their organizations, the collaborative, their target population, and the community at large.

Acknowledgements

This project was supported by Grant No. 96-WR-NX-004 awarded by the Violence Against Women Office, Office of Justice Programs, U.S. Department of Justice. Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the U.S. Department of Justice.

References

Allen, N. E., & Hagen, L. A. (2003). A practical guide to evaluating domestic violence coordinating councils. Harrisburg, PA: National Resource Center of Domestic Violence. Retrieved October 2003 from: http://www.vawnet.org/NRCDVPublications/TAPE/Papers/NRCDV_ EvalDVCC.pdf

Borden, L. M, & Perkins, D. F. (1999). Assessing your collaboration: a self-evaluation tool. Journal of Extension [On-line], 37(2). Available at: http://www.joe.org/joe/1999april/tt1.html

Bradshaw, T. K. (2000). Complex community development projects: collaboration, comprehensive programs, and community coalitions in complex society. Community Development Journal, 35, 143-145.

Butterfoss, F. D. (1998). Coalition effectiveness inventory (CEI): Self-assessment tool. Center for Pediatric Research: Center for Health Promotion. South Carolina.

Clark, S., Burt, R., Schulte, M., & Maguire, K. (1996). Coordinated community responses to domestic violence in six communities: Beyond the justice system. Washington, D.C.: The Urban Institute. Retrieved October 2003 from: http://aspe.hhs.gov/hsp/cyp/domvilnz.htm

Cranwell, M., & Schmidt, F. (2002). South Windsor County Even Start program evaluation report, 2001-2002. University of Vermont, Center for Rural Studies.

Fishman, M., Farrell, M., Allen, V., & Eiseman, E. (2000). Evaluating community collaborations: A research synthesis. Falls Church, VA: The Lewin Group, Inc.

Gray, B (1989). Collaborating. San Francisco: Josey-Bass.

Hogue, T., Perking, D., Clark, R., Bergstrum, A., Slinski, M, & Associates. (1995). Collaboration framework: Addressing community capacity. Columbus, OH: National Network for Collaboration.

Nunnally, J. C. (1978). Psychometric theory, 2nd edition. New York: McGraw-Hill.

Perkins, M. (2002). Comprehensive community initiatives (CCI): A comparison of community implementation plans. University of Nebraska Public Policy Center.

Shepard, M. (1999). Evaluating coordinated community responses to domestic violence. University of Minnesota at Duluth, Department of Social Work. Duluth, MN: Minnesota Center Against Violence and Abuse. Retrieved October 2003 from: http://www.vaw.umn.edu/documents/vawnet/ccr/ccr.html

Taylor-Powell, E. & Rossing, B. Evaluating collaborations: Challenges and methods. University of Wisconsin Extension Service. Retrieved October 2003 from: http://danr.ucop.edu/eee-aea/rossing.html

U.S. Census Bureau. (2000). Unites States Census 2000. Population and housing statistics. Retrieved October 2003 from: http://www.census.gov