A Formative Evaluation of the Children, Youth, and Families at Risk Coaching Model

The Children, Youth, and Families at Risk (CYFAR) initiative was created in 1991 by the U.S. Department of Agriculture Cooperative State Research, Education, and Extension Service. Since its inception, the primary purpose of CYFAR has been to fund and support community-based projects designed to promote positive developmental outcomes among vulnerable children, youth, and families. CYFAR has funded more than 600 projects based in every state and territory of the United States (Marek, Byrne, Marczak, Betts, & Mancini, 1999; U.S. Department of Agriculture, National Institute of Food and Agriculture, n.d.). A primary goal of the CYFAR initiative is to support funded projects through a combination of mentoring, problem solving, professional development, and other forms of technical assistance. In 2014, CYFAR shifted to a coaching model, in which seven CYFAR coaches provide ongoing support to local project sites. We developed the CYFAR coaching model, and in this article, we describe the results of a formative evaluation of it. We begin with a short description of strategies used by CYFAR coaches and then review the findings from a self-report survey designed to assess perceptions and attitudes of state-level CYFAR project principal investigators (PIs) toward the coaching activities. We conclude by discussing implications that these findings have not only for the CYFAR initiative but also for Extension professionals outside CYFAR whose work includes providing technical assistance to community-based stakeholders.

CYFAR Coaching Strategies

CYFAR coaching strategies are based on a peer-coaching approach that includes a combination of mentoring, technical assistance, guidance, reflection, problem solving, and team building (Allen, 2013; Denton & Hasbrouk, 2009; Olson, Hawkey, Smith, Perkins, & Borden, 2016). The key difference between peer coaching and other forms of mentoring is that peer coaching is characterized by an empowering relationship that emphasizes mutual respect, open communication, and trust. In the CYFAR initiative, coaches seek to build relationships that are power-neutral and encourage stakeholders to actively participate in knowledge and skill building. Although CYFAR coaches help facilitate the learning process, they do not manage or police specific behaviors (Bluckert, 2005; Denton & Hasbrouk, 2009; Kutilek & Earnest, 2001). The CYFAR coaches check in with project personnel at least once per month and tailor their support to meet the needs of individual projects.

We based the original design of the CYFAR coaching model on two existing initiatives. First, we drew from the Promoting School-Community-University Partnerships to Enhance Resilience (PROSPER) project. PROSPER is a community-based prevention planning system in which Extension educators serve as peer coaches who provide guidance, consultation, support, and general technical assistance to coalitions working on implementing evidence-based youth-focused prevention strategies (Chilenski et al., 2014; Perkins et al., 2011). Second, we drew strategies from the Military REACH project at the University of Minnesota, a project designed to enhance the capacity of local communities to develop, implement, and evaluate programs that serve children and youth from military families. Military REACH resources include coaching tools and mechanisms designed to promote high-quality program implementation and evaluation (Hawkey, 2015; "REACH: Supporting military families," n.d.).

With concepts from PROSPER and Military REACH forming our coaching model foundation, we based our specific coaching activities on the work of Denton and Hasbrouk (2009), who provide a general overview of types of coaching outlined by the American Institutes of Research (2005). They specifically describe five distinct but related types of coaching that have become central to the CYFAR coaching model. Those types of coaching are technical coaching, problem-solving coaching, reflective practice coaching, team-building coaching, and reform coaching:

• Technical coaching typically focuses on implementing school- or community-based interventions or curricula. Technical coaches often provide implementation support and help monitor and improve the fidelity with which interventions are implemented.
• Problem solving is a key component of coaching activities. In CYFAR, coaches often use a collaborative approach when working through problems with stakeholders (Stormont & Reinke, 2012; Wasylyshyn, 2003). Coaches seek to facilitate group processes and try to avoid acting like a leader, an expert, or a manager. They emphasize teamwork and tailor their feedback to the unique needs of individual CYFAR projects.
• In reflective practice coaching, coaches act as facilitators who work with stakeholders to empower them to think about their projects in new and innovative ways. Reflective practice coaching typically involves insightful questioning and encouragement and may include problem-solving exercises and the exploration of probing questions that encourage CYFAR grantees to think about their projects in innovative ways.
• Team building is an important part of most coaching models. As the name implies, team-building coaching is associated with building trust and mutual support among project stakeholders. CYFAR coaches seek to create a community of learning in which peers coach one another on best practices. This approach might involve making observations, attending professional development events, and participating in team-building exercises.
• While all types of coaching encourage some degree of change among stakeholders, reform coaching focuses specifically and deliberately on change. Commonly, this type of coaching is reserved for situations in which broad-scale organizational change is needed. When this is the case, coaches typically involve multiple levels of stakeholders, as buy-in is an essential part of creating a culture of reform and innovation.

In practice, CYFAR coaches use a combination of the five types of coaching, depending on the unique needs of a project relevant to the programming stage the project is in.

The purpose of the research project described in this article was to collect data as part of a formative evaluation designed to determine the degree to which CYFAR coaches engage in each of the above-mentioned types of coaching and to identify perceptions among PIs about the coaches' efforts. As such, we surveyed PIs who currently oversee state-level CYFAR-funded projects. We sought to answer six general research questions:

1. To what degree do PIs agree that CYFAR coaches engage in technical, problem-solving, reflective practice, team-building, and reform coaching?
2. Are any of these types of coaching associated with higher rates of perceived effectiveness of the CYFAR coaches?
3. How are coaches rated on the various components of the perceived effectiveness measure?
4. Are there types of coaching that PIs would like to see more or less use of by coaches?
5. Are there specific ways in which PIs have benefited from the work of coaches?
6. Are there specific changes or improvements to the coaching practices that PIs would like their coaches to make?

Methods

Participants

Data were gathered in 2015 from an online survey of PIs having existing CYFAR grants. The survey was developed by the CYFAR Professional Development and Technical Assistance leadership team. Individual items were designed to assess PIs' perceptions of the value of working with a CYFAR coach and their attitudes toward their coaches. The survey also included open-ended questions designed to assess specific benefits of working with a coach and to identify areas in which the coaching approach should be modified.

Of 43 PIs invited to participate in the survey, 31 responded, for a response rate of 72.1%. Given our interest in focusing this project as a formative evaluation rather than an assessment of the effectiveness of individual coaches, confidentiality of respondents was essential. To ensure that the identities of respondents remained anonymous, we partnered with the Office of Measurement Services at the University of Minnesota to remove any information that could potentially be used to identify individual participants.

Measures

Perceptions of Use of Types of Coaching

Participants were asked to indicate the degree to which they agreed that their coaches engaged in each type of coaching: technical, problem-solving, reflective practice, team-building, and reform. Each type was assessed with a single item. Response options for each item ranged from 1 (very unlike my CYFAR coach) to 4 (very much like my CYFAR coach).

Perceptions of Coaching Effectiveness

Perceptions of coaching effectiveness were assessed with an eight-item index. Examples of individual items include "I can benefit professionally from contact with my CYFAR coach" and "My relationship with my site's CYFAR coach has helped me to be more effective in the work that I do." Response options ranged from 1 (strongly disagree) to 5 (strongly agree). Responses from individual items were averaged to compute a composite index, with higher scores indicating perceptions of greater coaching effectiveness. Cronbach's alpha in the sample was 0.904.

Preferences for Types of Coaching

Participants were asked to indicate the degree to which they wanted more or less use of each of the five types of coaching. Response options for each item ranged from 1 (much less) to 5 (much more).

Perceptions of Benefits and Desire for Changes

Two open-ended questions were designed to gather subjective comments from study participants: "In your own words, please describe the ways in which your CYFAR coach has been of greatest benefit to you" and "In your own words, please describe any changes or improvements you would like your CYFAR coach to make."

Results

Perceptions of Use of Types of Coaching

To assess participants' perceptions of the coaches' use of the five common types of coaching, we ran simple descriptive statistics. Generally, respondents perceived that CYFAR coaches use all five types of coaching, although respondents perceived greater use of certain types of coaching. As shown in Table 1, the highest rates of agreement related to problem-solving coaching and team-building coaching, with more than 70% of respondents reporting that coaching styles represented by these types were like or very much like what they see from their coaches. In contrast, the perception was that coaches were least likely to engage in technical coaching or reform coaching, with only about 35% of respondents reporting like or very much like for these types of coaching. Reflective practice fell in the middle, with 60% of respondents indicating that this type of coaching characterized their coaches.

Table 1.
Reported Use of Types of Coaching (n = 31)

Response	Technical	Problem-solving	Reflective practice	Team-building	Reform
Very unlike coach	12.9%	6.7%	10.0%	3.2%	34.5%
Unlike coach	51.6%	13.3%	30.0%	25.8%	31.0%
Like coach	22.6%	56.7%	46.7%	64.5%	27.6%
Very much like coach	12.9%	23.3%	13.3%	6.5%	6.9%

Associations Between Type of Coaching and Perceived Coaching Effectiveness

To determine the degree to which each type of coaching is associated with perceptions of coaching effectiveness, we ran a series of t-tests. We began by forming dichotomous variables for the types of coaching, which served as independent variables: 0 for very unlike coach or unlike coach and 1 for like coach or very much like coach. Our measure of perceived coaching effectiveness was entered as the dependent variable. Results indicate that higher levels of technical, problem-solving, reflective practice, and team-building coaching were significantly associated with higher levels of perceived coaching effectiveness. We observed no significant effect for reform coaching (see Table 2).

Table 2.
Types of Coaching as Predictors of Perceptions of Coaching Effectiveness (n = 31)

	Coaching effectiveness mean score
Type of coaching	Unlike coach	Like coach	t	p
Technical	3.26	4.14	−3.47	.002
Problem-solving	2.98	3.77	−2.41	.023
Reflective practice	3.20	3.88	−2.61	.014
Team-building	3.13	3.75	−2.11	.043
Reform	3.51	3.93	−1.45	.158

Ratings of Perceived Coaching Effectiveness at the Item Level

To further understand PIs' ratings of coaching effectiveness, we examined responses to the individual items that comprise the composite index. Results suggest that responses for most items fall on the "agree" end of the spectrum, with the exception of responses related to the degree to which PIs contact their coach, which fell slightly on the "disagree" end of the distribution. All mean scores were near the middle of the possible range of scores, indicating only slight agreement or disagreement on each (see Table 3).

Table 3.
Ratings of Coaching Effectiveness (n = 31)

Perception	Mean	SD	Range
PI perceives professional benefit from contact with CYFAR coach	3.69	0.97	1–5
Relationship with CYFAR coach has improved effectiveness in CYFAR work	3.53	1.02	1–5
CYFAR coach offers assistance that is otherwise unavailable	3.25	1.14	1–5
PI understands coach's role and responsibilities	3.69	0.90	1–5
PI frequently contacts coach	2.84	1.22	1–5
Coach understands the details at the program site	3.78	0.91	1–5
PI perceives no way to benefit professionally from coach (reverse coded)	1.78	0.83	1–5
Coach is an important member of the project team	3.56	1.01	1–5

Preferred Types of Coaching

Results related to PIs' reports of the degree to which they wanted more or less use of each type of coaching indicate that respondents were most likely to want more reflective practice coaching and technical coaching, with more than 30% indicating a preference for more of these types of coaching. They seemed relatively satisfied with the current levels of the other types of coaching, with 73% to 81% of respondents indicating that they wanted their coaches to maintain the same levels of problem-solving, team-building, and reform coaching (see Table 4).

Table 4.
Self-Reported Preferred Types of Coaching (n = 31)

Preference	Technical	Problem-solving	Reflective practice	Team-building	Reform
Want much less	3.3%	0.0%	0.0%	0.0%	3.3%
Want less	0.0%	0.0%	0.0%	0.0%	0.0%
Want the same	66.7%	73.3%	67.7%	80.6%	80.0%
Want more	23.3%	26.7%	32.2%	19.4%	13.3%
Want much more	6.7%	0.0%	0.0%	0.0%	3.3%

Perceived Benefits of Coaching Activities

To identify PIs' perceptions of how they have benefited from coaching, we analyzed responses to the open-ended questions in the survey. Two members of the research team independently reviewed responses to these items and developed a coding scheme designed to capture the various dimensions of the responses. Each team member grouped responses into the categories identified in Table 5. For the question related to ways in which statewide CYFAR project PIs have benefited from their coaches, two main categories emerged, with a variety of subcategories nested in each. The interrater reliability rate across the two coders was .78. The most commonly cited benefits relate to the personal qualities of the coach. Within this broad category, 39% of respondents reported that their coaches were helpful/supportive, 23% reported that their coaches were available/responsive, and 19% reported that their coaches were nice/kind/positive. Respondents were less likely to comment on specific coaching behaviors, although about 10% reported that their coaches helped by searching for information and helped with evaluation or reporting requirements. (See Table 5.)

Table 5.
Benefits of CYFAR Coach (n = 31)

Categorized coded response	Frequency	Percentage
Qualities of the coach
Coach is helpful/supportive	12	39%
Coach is available/responsive	7	23%
Coach is nice/kind/positive	6	19%
Coach is a good listener/communicator	3	10%
Coach is honest/sincere	2	6%
Coach is knowledgeable	2	6%
Coach is well-organized	1	3%
Coach behaviors
Coach searched for information for the site	3	10%
Coach assisted with evaluation or reporting requirements	3	10%
Coach conducted site visit	2	6%
Coach assisted the site through project changes	2	6%
Coach acted as intermediary with the national team	2	6%
Coach provided general brainstorming	2	6%
Coach shared personal experiences	1	3%
Coach acted as advocate for the site	1	3%
Coach helped form connections across projects	1	3%
Coach helped with curriculum development	1	3%

Ways in Which Coaches Can Improve

To identify PIs' perceptions related to how CYFAR coaches could better meet their needs, we analyzed the responses to a second set of open-ended questions. We implemented the same approach for coding answers described in the preceding section, resulting in the categories identified in Table 6. Interrater reliability was .61. Suggestions for improvement were reported at lower rates than self-reported benefits. The most common responses called for "no changes," "none," or "nothing," as indicated by 29% of respondents. The only other category to receive more than two responses was related to requests for coaches to provide more assistance with reporting requirements and the grant renewal process (see Table 6).

Table 6.
Suggested Areas for Coach Improvement (n = 31)

Categorized coded response	Frequency	Percentage
Coach actions
Assist with reporting/renewal process	5	16%
Provide additional clarification about CYFAR resources	2	6%
Provide additional clarification about CYFAR guidelines	1	3%
Provide information about the CYFAR conference	1	3%
Put site in contact with other program PIs	1	3%
Facilitate group learning sessions	1	3%
Facilitate problem solving	1	3%
Meet grantees in-person	1	3%
Changes to coaching model
Reduce frequency of contact with coaches	2	6%
Provide coaches with additional training	1	3%
Provide earlier access to coaches	1	3%
Encourage less bureaucracy	1	3%
Clarify the role of coaches	1	3%
Suggestions for the CYFAR national team
Change annual reporting system	2	6%
Change CYFAR website	1	3%
Change CYFAR webinars	1	3%
No changes	9	29%

Discussion

The results of this formative evaluation project suggest that CYFAR coaches use a variety of types of coaching in their work and that PIs are generally satisfied with the coaches' methods and see value in the coaching model. Furthermore, four of the five types of coaching used in CYFAR are associated with increases in perceptions of coaching effectiveness. As revealed by our simple descriptive analysis of the various aspects of coaching effectiveness, PIs reported that their coaches seem to understand their projects and that there are professional benefits associated with working with a coach. In the open-ended questions, PIs reported that their coaches are helpful, kind, and responsive toward their needs. Together, these data suggest a number of positive aspects of the CYFAR coaching model.

In addition to identifying various strengths of the coaching model, the PIs noted several areas in which improvements could be made. According to the PIs, CYFAR coaches were less likely to engage in technical and reform coaching than they were to engage in the other types of coaching, and PIs specifically noted that they would like to see more technical and reflective practice coaching. The data also indicate that PIs are relatively unlikely to contact their coaches, meaning that for those who do not reach out to their coaches, contact happens only when it is initiated by the coach. Finally, PIs identified a number of areas in which they would like additional support, with the areas of reporting requirements and grant renewal process being mentioned by the largest numbers of respondents.

Limitations

As with any self-report study, several factors limit the generalizability of the study results. First, although our response rate of 72% would be considered more than acceptable by common self-report survey standards, it means that we did not receive data from 28% of the CYFAR PIs. We cannot rule out the possibility that systematic biases resulted from the missing data; thus, we should use caution before generalizing these findings to all CYFAR project sites.

Second, to maximize response rates, we purposefully designed self-report measures that were brief. As such, we included a variety of single-item measures. While these items assessed simple perceptions and provided us with valuable feedback on the CYFAR coaching model, it is likely that we would have received a fuller understanding of strengths and limitations of the coaching model had we used more comprehensive measures.

Finally, given the applied and formative nature of this evaluation project, we based this article on simple, mostly descriptive statistical analyses. Our intention is that future evaluation involve more comprehensive analyses through which we can begin to identify how various aspects of the CYFAR coaching model predict outcomes associated with CYFAR programming.

Implications for the CYFAR Coaching Model and Beyond

Despite the limitations of the study, the lessons learned can help inform future coaching activities. Ongoing sound implementation demands a variety of coaching types and techniques. Given the study findings, we have carried out several changes to our coaching activities, and we have plans to continue to refine the coaching model. To date, we have addressed or plan to address a variety of issues:

• Given our plan to develop a peer coaching model, we are concerned to learn that PIs are not likely to contact their coaches. To promote more bidirectional communication between sites and coaches, we have shared with the coaches professional development materials that describe common strategies associated with a peer coaching model. We also held a workshop session on communication strategies from which coaches can draw when working with their sites. Finally, we have employed several strategies designed to build relationships between coaches and site personnel. For example, we shared information about our coaching model with CYFAR grantees during an annual professional development event, and coaches spent time connecting with their sites during that event.
• PIs noted that they want more technical and reflective practice coaching and more help with the reporting and renewal processes. We are particularly interested in supporting increased efforts in the area of technical coaching as we found that this type of coaching was related to increased perceptions of coaching effectiveness but that CYFAR coaches have not used it as much as other types of coaching. To address these concerns, we have introduced a more intensive coaching approach for newly funded projects. We believe that coaches will be better able to assist these new sites through more intensive relationship building that includes more frequent contact, more focused problem solving, and a site visit early in the CYFAR grant cycle. By forming connections early in the process, we hope to immediately promote tighter connections between coaches and their stakeholders that will endure throughout the grant cycle. In addition, coaches are currently working with all sites to develop comprehensive calendars that outline grant requirements, and we plan to encourage coaches to become more involved in supporting sites that need help with the reporting process and technical support on other aspects of CYFAR funding requirements.
• In response to the many suggestions made through the open-ended questions, we planned several new initiatives. To improve cross-site communication, we developed a site directory that includes a brief summary of each project and contact information for site personnel. We also redesigned the CYFAR website to facilitate easier access to resources intended to support all aspects of CYFAR grant requirements. Finally, we are in the preliminary stages of developing technologies to enable CYFAR coaches to conduct virtual site visits in which sites record programming activities and then share the recordings with their coaches to get personalized feedback on the program implementation process.

On a more general level, the findings of our study have implications for Extension personnel from all program areas. Coaching is a versatile technical assistance strategy that has been used in a wide variety of settings, ranging from human resource trainings to assistance with the implementation of diverse programs, such as educational initiatives, prevention strategies, and community interventions (Allen, 2013; Becker, Bradshaw, Domitrovich, & Ialongo, 2013; Franz & Weeks, 2008; Kutilek & Earnest, 2001). The results of the study reported here underscore the importance of building positive relationships between coaches and their stakeholders. Participants in the study reported benefiting most when their coaches provided technical assistance, offered help with solving problems, and encouraged stakeholders to reflect on current practices. These general approaches to building coaching relationships can be tailored to meet the needs of stakeholders from all program areas in Extension.

Conclusion

The CYFAR coaching model is constantly evolving as we move from a more traditional system of technical assistance toward a true peer coaching approach. The changes we have outlined will help CYFAR coaches build more bidirectional relationships with CYFAR sites. Also, the lessons we learned from our first year of using the CYFAR coaching model have implications for Extension personnel from all program areas who may be interested in developing a similar approach to support their stakeholders. The results of our study underscore the importance of using diverse coaching strategies; being available, supportive, helpful, and responsive; and being prepared to assist with the technical details associated with a particular project or initiative. In light of recent trends toward increased accountability for project funding, we believe that providing technical support through a coaching model can help local projects run efficiently and maximize returns on funders' investments.

References

Allen, K. (2013). Coaching: A tool for Extension professionals. Journal of Extension [online], 51(5) Article 5IAW1. Available at: http://www.joe.org/joe/2013october/iw1.php

American Institutes for Research. (2005). Conceptual overview: Coaching in the professional development impact study. Author.

Becker, K. D., Bradshaw, C. P., Domitrovich, C., & Ialongo, N. S. (2013). Coaching teachers to improve implementation of the good behavior game. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 482–493.

Bluckert, P. (2005). Critical factors in executive coaching—The coaching relationship. Industrial and Commercial Training, 37(7), 336–340.

Chilenski, S., Perkins, D. F., Feinberg, M., Greenberg, M., Welsh, J., & Crowley, D. M. (2014, November). A university-community model of technical assistance: The PROSPER project. Paper presented at the annual meeting of the SCRA Biennial Meeting, Roosevelt University/Harold Washington Library, Chicago, IL.

Denton, C. A., & Hasbrouck, J. (2009). A description of instructional coaching and its relationship to consultation. Journal of Educational and Psychological Consultation, 19(2), 150–175.

Franz, N., & Weeks, R. (2008). Enhancing Extension employee coaching: Navigating the triangular relationships. Journal of Extension [online], 46(5) Article 5TOT1. Available at: http://www.joe.org/joe/2008october/tt1.php

Hawkey, K. (2015). Program quality improvement: A guide for CYFAR programs. Minneapolis, MN: University of Minnesota.

Kutilek, L. M., & Earnest, G. W. (2001). Supporting professional growth through mentoring and coaching. Journal of Extension [online], 39(4) Article 4RIB1. Available at: http://www.joe.org/joe/2001august/rb1.php

Marek, L. I., Byrne, R. A., Marczak, M. S., Betts, S. C., & Mancini, J. A. (1999). The Children, Youth, and Families at Risk (CYFAR) Evaluation Collaboration. Journal of Extension [online], 37(3) Article 3FEA4. Available at: http://www.joe.org/joe/1999june/a4.php

Olson, J. R., Hawkey, K., Smith, B., Perkins, D. F., & Borden, L. M. (2016). Applying coaching strategies to support youth- and family-focused Extension programming. Journal of Extension [online], 54(1) Article 1FEA4. Available at: http://www.joe.org/joe/2016february/a4.php

Perkins, D. F., Feinberg, M. E., Greenberg, M. T., Johnson, L. E., Chilenski, S. M., Mincemoyer, C. C., & Spoth, R. L. (2011). Team factors that predict to sustainability indicators for community-based prevention teams. Evaluation and Program Planning, 34(3), 283–291.

REACH: Supporting military families through research and outreach. (n.d.). Retrieved from https://reachmilitaryfamilies.umn.edu/

Stormont, M., & Reinke, W. M. (2012, Winter). Using coaching to support classroom-level adoption and use of interventions within school-wide positive behavioral interventions and support systems. Beyond Behavior, 11–19.

U.S. Department of Agriculture, National Institute of Food and Agriculture. (n.d.). Children, Youth and Families at Risk (CYFAR). Retrieved from https://nifa.usda.gov/program/children-youth-and-families-risk-cyfar

Wasylyshyn, K. M. (2003). Executive coaching: An outcome study. Consulting Psychology Journal: Practice and Research, 55(2), 94–106.