Program Development from Start-to-Finish: A Case Study of the Healthy Relationship and Marriage Education Training Project

Since 2006, federal funding has supported the launch and evaluation of several relationship and marriage education (RME) initiatives to promote family stability and child well-being (Bembry, 2011). RME involves teaching principles and skills (e.g., listening to understand, avoiding criticism, and defensiveness during conflict) that foster healthy couple and co-parenting relationships. (For a review of the core principles and skills needed to maintain healthy couple relationships, see Futris & Adler-Baeder, 2013.) Research has demonstrated that RME can positively impact relationship attitudes, behaviors, and quality across diverse and at-risk audiences (Cox & Shirer, 2009; Hawkins, Blanchard, Baldwin, & Fawcett, 2008; Hawkins & Fackrell, 2010). As well, when combined with parenting education, research reinforces the added benefits of RME (versus parenting education alone) on promoting positive parenting behaviors (e.g., Schulz, Pruett, Kerig, & Parke, 2010).

In 2009, the Administration on Children, Youth and Families Children's Bureau announced a 5-year funding opportunity (Grant 90CT0151) aimed to develop and pilot a training curriculum designed to equip child welfare professionals (CWPs) with information and strategies to strengthen couple and family relationships in order to meet the safety and permanency needs of children in the child welfare system. Historically, child welfare services have focused primarily on crisis intervention, and more recent efforts have emerged that aim at establishing comprehensive evidence-informed, community-based, prevention services (Mitchell et al., 2012). RME had been identified as one strategy to address the negative spill-over effects of couple and coparenting conflict on parenting and child well-being and promote healthy and stable families (Christensen, Antle, & Johnson, 2008).

In response to this call for proposals, a multi-state partnership of Cooperative Extension specialists from five land-grant universities was awarded this cooperative agreement to develop and pilot the Healthy Relationship and Marriage Education Training (HRMET). The partnering states included Arkansas (AR), Georgia (GA), Iowa (IA), Missouri (MO), and North Carolina (NC). In general, the curriculum development and implementation process was grounded by family life education best practices (Duncan & Goddard, 2011), conceptualized within a logic model framework in general (Taylor-Powell, Jones, & Henert, 2003) and specific to RME program design (Futris, Mallette, & Richardson, in press), and informed by formative and summative evaluation (Jacobs, 1988).

This article summarizes the process through which the training curriculum evolved and shares lessons learned that could inform future program development efforts. Our description of the process and lessons learned is organized below within the logic model framework, depicted in Figure 1, that guided the project's design and evaluation. We close with a summary highlighting a few of the main lessons we learned and recommendations.

Figure 1.
Healthy Relationship and Marriage Education Training (HRMET) Logic Model

Defining the Situation

The process of clarifying and becoming familiar with the needs and desired outcomes of any program, also referred to as "defining the situation," is a critical first step in program development. Although the federal funding agency had defined the need and purpose of the project, we discovered that child welfare services within each partnering state (our target audience) did not automatically share or prioritize this as a need. In fact, initial engagement of state child welfare administrators was often met with skepticism regarding the intent, value, and appropriateness of RME in child welfare.

Although we had made significant progress in developing the curriculum during the first year, we realized that we needed to take a step back and better understand our target audience. With state child welfare administrative support, a needs assessment was conducted in MO (May 2009), and later replicated in NC (June 2011), that reached nearly 5,300 CWPs caseworkers and administrators. Based on responses from 1,243 CWPs to an online survey, we found consistent results from the MO (n = 620) and NC (n = 623) samples. These findings, described in greater detail in Schramm, Futris, Galovan and Allen (2013), reinforced that most CWPs:

• Understood the connection between strong couple/marital relationships and successful parenting,
• Agreed that being trained to teach skills that strengthen couple relationships could help them meet client needs, and
• Were interested in receiving RME training.

This feedback reinforced prior research (Antle, Frey, Sar, Barbee, & Van Zyl, 2010; Sar, Antle, Bledsoe, Barbee, & Van Zyl, 2010) and our assumptions (logic model in Figure 1) about the value and interest in the training. Using these data, reports and presentations were prepared and shared with state administrators and other stakeholders when attempting to obtain "buy-in" and support for the training.

Respondents also shared several concerns and potential barriers related to attending RME training and teaching RME. This feedback, and how we used it to advance program development, is summarized in Table 1. Consistent with prior lessons learned implementing a similar training (Antle et al., 2010), we too learned that promoting the training by its formal title (the Healthy Relationship and Marriage Education Training) may have discouraged participation in the initial pilot trainings due to concerns shared in Table 1. As such, the training was promoted as "Improving the Lives of Children Through Healthy Couple Relationships and Stable Homes" to clearly emphasize the training's intent and connection to child welfare services.

In short, this initial assessment of our target audience provided critical insight that informed how we refined the training curriculum and supporting materials, what we communicated to our target audience when promoting and conducting the trainings, and potential external factors (Figure 1) that could impede training participation and impact.

Table 1.
Examples of Lessons Learned Based on Concerns CWPs Shared About RME

Concern	Sample Comment	Implication
I'm not a counselor.	I feel that we need to be careful that child welfare is not crossing the line by trying to be mental health workers or therapists with families or couples.	Clarify the difference between RME and therapy or counseling.
Is RME value-laden?	My knowledge is acquired from faith based marriage counseling/seminars I have attended in my own marriage. It would not feel appropriate to give my clients unsolicited marriage advice.	Establish the empirical foundation for RME.
Is RME a priority?	Although relationship issues may be a problem, so many families have so many other problems to deal with to insure safety of the children that relationship goes to the bottom of the list.	Recognize that other service needs may take priority, and explain how RME can offer additional benefits.
How does RME address child safety?	As a social worker I do not focus so much on marital status as with effective parenting... Sometimes child safety is all that you can focus on and unhappiness does not necessarily lead to abuse or neglect so these issues need to be tabled for other community partners to handle.	Demonstrate the influence of couple relationship behaviors on parenting practices, in general, and child safety, specifically.
My clients aren't married.	Most of the clients we work with are not married so I am not sure how much training would help. The majority of our clients are single moms.	Identify the broader audience that can benefit from RME.
But, I'm not married.	Sadly many child welfare workers have not had positive marriage experiences themselves. So, it would not be realistic for them to be giving training to others.	Clarify that unhealthy behaviors can be unlearned, and RME skills can be learned by everyone.
I am not sure I can do this.	I lack the training and skill to convey this information to parents in an educational/ coaching method that would create the outcome desired for improving their co-parenting status.	Equip CWPs with materials that clearly describe how to teach RME skills.
I don't have time!	To work on marital issues can take longer than we have...this may be possible, but there is already a lot for line staff to do.	Develop resources that are simple and can be easily adapted and integrated into current practices.

Program Inputs

The curriculum development process is also strongly influenced by the availability and effective utilization of various resources, or inputs. Inputs may include staff, facilities, curricula, resources to recruit and retain program participants, collaborators, and funding. The primary resource (in addition to funding) that moved this project forward was people.

• The entire project team met monthly by conference call during the first year, quarterly by conference call in years 2-5, and in-person once per year during years 2-5. These meetings were helpful in establishing clear and shared expectations, sharing engagement and training experiences that would inform module development and refinement, and processing evaluation results in order to improve subsequent training efficacy.
• Extension specialists involved in developing the National Extension Relationship and Marriage Education Model (NERMEM; Futris & Adler-Baeder, 2013), which served as the core foundation for the HRMET curriculum, also offered their time to review and share feedback on early drafts of the HRMET curriculum. Their involvement helped ensure the curriculum adhered to the original NERMEM framework.
• Other national experts familiar with RME and working with CWPs served in an advisory capacity and offered feedback on the curriculum. They were helpful in identifying gaps in the curriculum, pointing out content that needed clarification, and assuring that the information and resources related to the needs of our child welfare audience.
• State- and county-level partners also were engaged in program development (e.g., reviewing drafts of the training curriculum and supporting resources) and implementation (e.g., identifying and recruiting CWPs to attend the pilot trainings; securing training facilities) throughout the project. For example, CWPs in MO provided helpful input in focus groups, which have been show to be an effective and efficient component of program development (White, Arnold, & Lesmeister, 2008), during the first year that altered our approach from designing resources for traditional RME delivery to families (e.g., linear curriculum delivered in a workshop, retreat, etc.) to brief 1-page tools on various topics that CWPs could adapt and share with clientele based on their unique relationship needs.
• Regional/County Extension agents played various roles across states that included engaging local child welfare agencies/professionals, marketing, and coordinating arrangements for local trainings. Also, in Year 5, Family and Consumer Science agents in GA co-facilitated trainings (after being trained in Year 4); this facilitated sustainability of the program in GA post-funding.

Various partners within each state (Table 2) offered valuable insight, recommendations, and support that contributed to the program. While similarities in the implementation of the HRMET existed across states (e.g., curriculum content and format), there also were unique strategies employed to engage CWPs within each state. For example, the "buy-in" and promotion of the training initiated from state-level partners ("top-down") in AR, MO, and IA. In contrast, challenges engaging similar partners in GA and NC necessitated a more "bottom-up" approach to generate interest and support. Thus, we discovered that the approach for each state was different and needed to be tailored based on the reactions of state-level partners. In short, engaging our various partners from "start-to-finish" of the program development process helped ensure our curriculum and training implementation design was grounded in research and reflected best practices relevant to our target audiences.

Table 2.
Contributions Made by State Partners

State	Partner: Primary Contribution
Arkansas	Dept. of Social Work, University of Arkansas - Little Rock (UALR): shared advice on promoting and engaging social workers in Arkansas. National Association of Social Workers (NASW) Arkansas Chapter: promoted the training in their statewide newsletter. MidSOUTH, a community service unit at UALR: provided support with coordinating the event and recruiting participants.
Georgia	Georgia Association of Homes and Services for Children, a non-profit organization that provides training and support for private sector CWPs: promoted the training statewide and facilitated communications with state CWP administrators. Georgia Family Connection Partnership, a state-wide and state funded initiative that coordinates family strengthening services within each county in Georgia: promoted the training statewide through county partnerships. UGA Extension Family and Consumer Sciences County Agents: promoted, coordinated, and co-facilitated trainings within their county/region.
Iowa	Iowa Child Welfare Training Academy: promoted the training and managed registration. Iowa State University Extension and Outreach: county office staff hosted trainings (e.g., space, coordinated logistics for meals and audio visual needs).
Missouri	MO Children's Division Professional Development and Training Unit: Reviewed and provided feedback on early drafts of curriculum. Promoted the training on website and statewide listservs, and managed registration. MU School of Social Work: Reviewed modules and provided feedback on early draft of curriculum.
North Carolina	NC Division of Social Services: reviewed and provided feedback on early draft of curriculum. County Department of Social Services and NCSU Cooperative Extension: promoted the training and recruited participants.

Program Outputs

Outputs tend to reflect what is done and who is engaged during program implementation (McCawley, 2001). The project's implementation process involved a series of pilot trainings that occurred every spring and summer over the course of 3 years (2011 – 2013).

• Each training was approximately 6.5 contact hours and facilitated by a program team member.
• A combination of lecture, discussions, and group activities were used to engage participants and provide them with opportunities to process the information and explore ways to apply the tools with their clients.
• Participants received a training toolkit with PowerPoint handouts, research-based fact sheets that provided an overview of the content presented, and tip sheets and worksheets to share with clients and facilitate teaching of the concepts and skills.

Because grant funding was available, no registration fee was charged to cover training expenses, which included printed toolkits, participant supplies (e.g., pens, sticky notes), facility fees, facilitators' travel, and food/beverages (i.e., light breakfast, lunch, and snacks). However, during the Year 5 (2013) pilot in GA, a registration deposit of $30 was collected to assess whether attendance rates would increase. (Participants were informed at the time of registration that the $30 would be returned upon completion of the training; because this was grant funded, however, those who could not attend still received a refund.). This resulted in a higher attendance rate.

• Of the 298 CWPs who registered for one of eight trainings offered that year in GA, 94% attended.
• In comparison, the attendance rate in GA was only 57% in 2011 and 85% in 2012.
• In the other four states, attendance rates in 2013 ranged from 74% (MO) to 84% (IA).

Although other factors also influenced attendance rates (e.g., travel distance, competing job demands, organizational support, health), this pattern suggests some value in charging a fee to encourage registrants to attend the training. In fact, in 2014 (post-funding), UGA Extension conducted three trainings and charged an $80 (non-refundable) registration fee to cover all training expenses: out of 79 registrants, 74 (94%) attended or sent a substitute!

In total, 52 one-day trainings were conducted across the five states from 2011-2013. Of the 1,670 CWPs who registered, 1,375 (82%) completed the training. Below is a summary of the general profile of our participants, with more information available in our final report (Futris, Thurston, Lee, & Head, 2014):

• The majority of participants were female (92%) and Caucasian (64%).
• They ranged in age from 20 to 81years (M = 41.5; SD = 11.6).
• Most participants (75%) self-identified themselves as child welfare professionals. While both caseworkers (49%) and administrators (19%) attended the training, professionals in medical/mental health services (13%), family life education (6%), and the school (3%) and court (1%) system, as well as foster parents (3%), also attended.
• Experience ranged from less than one year to 40 years (M = 9.9; SD = 8.1).

Tracking the characteristics of our participants each year allowed us to regularly confirm we were reaching our target audience and identify unique client needs that could be addressed in future trainings. As summarized above, the majority of our participants represented the audience our funder intended for us to reach.

Outputs also included program evaluation efforts and response rates. Every CWP who registered for the training was invited to complete five surveys. Coordinated by Dr. Ted Futris (University of Georgia), below is a brief overview of the evaluation design:

• Following a tailored design method (Dillman, Smyth, & Christian, 2009), each participant received an initial email describing the survey and two reminder emails sent approximately 4-6 days apart.
• Emails were prepared by Dr. Futris' team and sent to the appropriate state team member to forward out to their training participants.
• To match data over time, while maintaining anonymity, each participant created their own study ID that they used each time they completed a survey. The ID consisted of 5-digits that we felt participants could easily recall: three-letter initials and two-digit day of month born.
• Because of state/county child welfare policies regarding gifts, no incentives could be provided to participate in the research.

Below is a summary of the five surveys, and the response rate for each one:

• One week prior to the training date, registered participants were asked to complete an online pre-test survey. Those who did not have an opportunity to complete the survey online were provided a paper-pencil survey the morning they arrived at the training. Of those who attended the training, 93% (n = 1,282) completed the pre-test survey.
• At the conclusion of the training, participants were asked to complete a brief paper-pencil post-test survey focused on their immediate reactions to the training. As recommended by prior research, these evaluation questions were tied to program outcomes (Radhakrishna & Relado, 2009). Again, most (96%, n = 1,322) participants responded.
• One week after the training, participants were asked to complete a survey focused on documenting changes in attitudes, knowledge, and skills acquired from the training. Of those who completed the initial pre-test (our baseline survey), 58% (n = 798) completed the one-week follow-up survey.
• Two and six months after the training, participants were asked to complete a survey focused on whether or not, and how, they had applied the information from the training. Nearly half of the participants (46%, n = 638) completed one or both surveys.

Not surprising, response rates declined over time. Still, 245 (18%) trainees completed all five surveys, and an additional 286 (21%) completed four of the five surveys. Overall, these are reasonably acceptable rates that are comparable to those reported in similar studies with child welfare professionals (e.g., Antle et al., 2008; Antle et al., 2010). In addition to non-responsiveness, the survey completion rate also was influenced, albeit minimally, by participants not providing an ID that could be matched with prior survey data (e.g., different initials, birth month or year was provided instead of birth day). Thus, careful consideration should be given to criteria used for self-constructing a memorable and distinguishable ID. Importantly, caution should be taken when interpreting the results (outcomes) and establishing conclusions since it is likely that behavioral change is more likely to be found among participants who are interested enough to return follow-up surveys (Groves, Presser, & Dipko, 2004).

Last, the outputs reflect the resources developed—the "fruits of our labor." Every fall (after pilot testing concluded), the project team met to process participants' survey feedback that was summarized in a report by the project evaluator. Also, the team processed responses they each shared on a brief facilitator survey focused on their experience teaching the material as well as their reflections on participants' responsiveness and concerns shared during the training. The training materials and delivery process were refined following each round of pilot trainings, resulting in the following resources (summarized in Table 3):

• An eight-module curriculum for in-person trainings that includes background information, scripted presentation notes, and PowerPoint slides to support Extension professionals in facilitating the HRMET.
• Eight online modules summarizing the concepts, principles, and skills addressed in the HRMET curriculum that could be used as refresher training or to share with co-workers/community partners. The online modules include PowerPoint slides with a professional voice-over narration, and some modules also feature videos, developed by Drs. Kim Allen and Andrew Behnke (North Carolina State University), to illustrate the application of the HRMET materials in a child welfare setting.
• Eleven research briefs, based on chapters written in Futris & Adler-Baeder (2013) that summarized the research related to each concept, were written for CWPs' own professional development.
• To support CWPs in sharing this information with clients, 53 tip sheets and worksheets were developed. Each 2-page worksheet focused on a specific principle/skill and included a 1-page overview (e.g., objective, target audience, instructions) for the CWP and a 1-page handout that could be shared with clients to complete and/or discuss.

Table 3.
Outputs: Product Deliverables

Module: Overall Objective	Approximate Time (in minutes)		Resources and Teaching Tools
	In-person	On-line	CWPs	Clients
Introduction: Understand the relevance of healthy couple relationships to child welfare and what healthy RME involves.	60	17	4	0
Care for Self: Engage clients in cultivating individual wellness and health in order to support the health of their couple relationship.	30	17	1	11
Choose: Support clients in making deliberate and conscientious decisions to be committed, intentional, proactive, and strengths-focused in their relationships.	45	15	1	7
Know: Help clients develop intimate knowledge of their partner's personal and relational needs, interests, feelings and expectations.	45	19	1	7
Care: Guide clients in expressing kindness, respect, and understanding to facilitate positivity and stability in their relationships.	45	13	1	6
Share: Demonstrate to clients the value of developing and maintaining couple time, a shared sense of couple identity, and a close friendship in healthy relationships.	60	20	1	8
Manage: Clarify to clients that conflict is normal in couple relationships, and share strategies to manage stress, listen to understand, accept differences, and ensure emotional and physical safety when conflict arises.	60	28	1	10
Connect: Help clients become better connected with their family, peers, and community as a source of support to them and their couple relationship.	30	20	1	4
Total	375	149	11	53

The video modules, research briefs, and tip sheets/worksheets are available for free online at www.hrmet.org. For more information about the facilitator's training curriculum, contact Drs. David Schramm (University of Missouri) or Ted Futris (University of Georgia).

Program Outcomes

As described above, program evaluation is an ongoing process (Jacobs, 1988). For example, to determine what was working and what required change, a formative evaluation of the inputs invested in the project was regularly conducted. Additionally, a summative evaluation was conducted to assess the worth of the training, also referred to as "program outcomes."

Using logic modeling as the basis for meaningful evaluation (Arnold, 2002), the design of the HRMET project evaluation was informed by past research on RME as well as research within the social work field focused on training evaluation (e.g., Louisville Child Welfare Training Evaluation Theoretical Model; Antle, Barbee, & van Zyl, 2008). As illustrated in Figure 1:

• Short-term outcomes focused on trainee satisfaction (i.e., affective reaction) as well as the core competencies required to deliver RME. This included helping trainees see the usefulness of RME to their work (i.e., utility reaction) and empowering them with the knowledge and efficacy to teach RME skills to their clients (i.e., immediate learning).
• Intermediate outcomes typically focus on behavior changes (i.e., learning transfer). Although direct observation of how trainees applied the skills in their work was not possible, both quantitative (e.g., how often they used the resources) and qualitative (e.g., how they used the resources) data were collected two- and six-months following the training.
• Long-term outcomes relate to the situation a program aims to improve. Based on prior research, evidence exists to support the benefits of RME in promoting healthy couple relationships, marital and family stability, and child safety and well-being. Unfortunately, limited resources (i.e., funding and time) made it challenging to document these outcomes. As well, the broader integration of RME into child welfare services was beyond the scope of our funding to assess.

Our results, described in the final report (Futris, Thurston, et al., 2014), showed that the training had a significant impact on the short-term and intermediate outcomes. The qualitative data reinforced the finding that most participants were applying the skills they learned (Table 4). Importantly, as reported in Futris, Schramm, Lee, Thurston, and Barton (2014) and consistent with our theory of change reflected in the logic model, the short-term outcomes influenced learning transfer.

Table 4.
Participants' Application of the Tools and Concepts in Their Work

Concept	Total (N=628)	2-mth (n=526)	6-mth (n=390)	Sample Comment
Choose	318 (49.8%)	247 (47.0%)	155 (39.7%)	With one family in particular, I used this topic to talk with the mom about how it was important that she focus on the strengths of the relationship instead of only the areas of weakness. I also talk with this mom about avoiding hurtful thoughts and behaviors. (IA)
Care for Self	357 (56.0%)	288 (54.8%)	175 (44.9%)	I have focused on this in individual training for a fostering couple. I have discussed the need for taking care of one's self in order to be a better partner and how this can affect overall parenting. (AR)
Know	295 (46.2%)	232 (44.1%)	126 (32.2%)	Helping clients understand that knowing their partner intimately can not only make it easier to understand their partner's perspective, but conversely makes them more self-aware. Knowing their partner creates an atmosphere of "I care" which can help our reactions to certain circumstances be less extreme. (MO)
Care	361 (56.6%)	282 (54.0%)	182 (46.7%)	We have talked with clients about showing care to their partners, even when they don't feel like being caring. Talked to them about how showing someone you care, regardless of how you feel at the moment, reminds the partner of one's commitment, respect, etc. (GA)
Share	313 (49.1%)	245 (46.6%)	154 (39.5%)	Participants are working on more effective communication which enables them to have more meaningful interactions with others. There are trust issues so this is not easy. [the power of 'we'] is stressed to them. Partners must commit to making time for each other that is full of fun and interest. Keep the romance alive. Be affectionate. Work toward common goals. (NC)
Manage	319 (50.0%)	245 (46.6%)	150 (38.5%)	This is a big area that I focus on with families. I talk with them about dealing with conflict effectively, managing anger, and most importantly, making sure that the children do not suffer when parents argue. (IA)
Connect	346 (54.2%)	271 (51.5%)	172 (44.1%)	I have expressed to my clients that having social networks, friends, family, outside environment, is valuable to the worth and well-being of their relationship. (MO)
Note: N = 638 participants responded to one or both follow-up surveys conducted two- and six-months post-training. N = 628 in the "Total" column represents the number of participants who responded to questions about their application of the tools received. The n's (%'s) reported for each construct represent the number (and percentage) of respondents who applied the respective tools.

As well, focus group interviews were conducted in MO and GA to solicit feedback from training participants about their experience during and after the training. In GA, for example, we learned that personal application, organizational support, and client need encouraged learning transfer and that certain organizational constraints and client characteristics posed challenges in using the training materials. These findings are reported in Scarrow, Futris, and Furhman (2014).

Last, sharing program evaluation results with stakeholders can help sustain and advance both public and financial support for future programming (Small et al., 2009). This was done through the development of annual reports, a final cumulative report, and individual state reports. These efforts reassured our partners in child welfare services that the training was of interest to CWPs and provided added value to their work and the services they offer to the families and children they serve. The support generated from these results was influential to the continued implementation of the HRMET in three of the five states (i.e., Georgia, Iowa, and Missouri) post-funding.

Conclusion

The 5-year project described here culminated with the development of a resource that shows promise in supporting child welfare efforts to strengthen couple and family relationships in order to meet the safety and permanency needs of children in the child welfare system. The experience yielded several lessons learned to inform future program development efforts. In addition to what was shared above, a few concluding points are worth reinforcing.

• Don't skip steps! Start the program development process by first defining or clarifying the situation, even when the aim and scope of the project are pre-defined by a federal or state funding agency. Disseminating a new program requires "buy-in" from state and local stakeholders.
• Know the audience. Engage the target audience to seek understanding about their needs, what they do, how the program fits with or can be integrated into what they do, and potential challenges and concerns related to participating in the program and applying what they learn into practice. This information should inform curriculum development, recruitment and marketing efforts, training and engagement strategies, and evaluation.
• Engage and educate partners. Expertise exists from various sources (e.g., academia, state/local agencies, non-profit organizations) that can inform program design and implementation. For a variety of reasons (e.g., competing demands, limited resources), not every potential partner will make similar investments (if any) in the program. To head-off partners' concerns related to a lack of interest in the program, it is critical to understand and clearly describe how the program fits with their organizations' mission and needs.
• Adapt and evolve. Curriculum development is a long process that requires patience as well as systematic methods (e.g., needs assessment, formative and summative evaluation) to make data-informed decisions. Grounding the curriculum in research, making adjustments in response to stakeholder and client feedback, and examining whether and how those modifications helped resulted in a better program.
• Document impact! Evaluation involves more than documenting outputs—it requires identifying and assessing measurable outcomes and objectives (e.g., McCann, Peterson, & Gold, 2009). Using an empirically based framework, we were able to document impact and frame our results in a meaningful way that connected with our various stakeholders.

In addition to the direct impact of a program on its intended target audience, impact is also reflected in whether the program is adopted by others. To date, the HRMET curriculum has been adapted into a national training curricula created by the National Resource Center for Healthy Marriage and Families (2012) as well as a new couple-focused program, "ELEVATE: Taking Your Relationship to the Next Level," that was developed in collaboration between Auburn University and the University of Georgia. (See http://www.nermen.org to learn more.)

Acknowledgements

In addition to the authors, project team members included Drs. Wally Goddard and James Marshall (University of Arkansas, AR), Dr. Anthony Santiago (Iowa State University, IA), Drs. Kim Allen and Andrew Behnke (North Carolina State University, NC), and Mrs. Kelly Warzinik (Project Coordinator, University of Missouri). We would also like to thank the project team, graduate assistants, and all those who helped develop the HRMET curriculum as well as our many state and local partners, and especially the county Extension agents in each project state for their support.

Funding for this project was provided by the United States Department of Health and Human Services, Administration for Children and Families, Grant: 90CT0151. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. Department of Health and Human Services, Administration for Children and Families.

References

Antle, B. F., Barbee, A. P., & van Zyl, M. A. (2008). A comprehensive model for child welfare training evaluation. Children and Youth Services Review, 30, 1063-1080.

Antle, B. F., Frey, S. E., Sar, B. K., Barbee, A. P., & Van Zyl, M. A. (2010). Training the child welfare workforce in healthy couple relationships: An examination of attitudes and outcomes. Children and Youth Services Review, 32, 223-230.

Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension [On-line], 40(3) Article 3FEA4. Available at: http://www.joe.org/joe/2002june/a4.php

Bembry, J. X. (2011) Strengthening fragile families through research and practice. Journal of Family Social Work, 14(1), 54-67.

Christensen, D., Antle, B., & Johnson, L. (2008). Integrating couple teamwork conversations into child welfare practice. Journal of Family Social Work, 11(3), 303-322.

Cox, R. B., & Shirer, K. A. (2009). Caring for my family: A pilot study of a relationship and marriage education program for low-income unmarried parents. Journal of Couple & Relationship Therapy, 8(4), 343-364. doi: 10.1080/15332690903246127

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Mail and Internet surveys: The Tailored Design Method, Third edition. New York: John Wiley and Sons.

Duncan, S. F. & Goddard, H. W. (2011) Family life education: Principles and practices for effective outreach (2^nd ed.). Thousand Oaks, CA: Sage.

Futris, T. G., & Adler-Baeder (Eds.). (2013) The National Extension Relationship and Marriage Education Model: Core teaching concepts for relationship and marriage enrichment programming. (Publication No. HDFS-E-157). Athens, GA: The University of Georgia. Retrieved from: http://www.nermen.org

Futris, T. G., Mallette, J. K, & Richardson, E. W. (in press). Program design and implementation. In J. J. Ponzetti (Ed.), Evidence-based approaches to relationship and marriage education. New York : Taylor & Francis/Psychology Press.

Futris, T. G., Thurston, D., Lee, T. K, & Head, L. (2014). The Healthy and Relationship Education Training: One-day training evaluation report (2011-2013). Athens, GA: University of Georgia Cooperative Extension. Retrieved from: http://www.hrmet.org

Futris, T. G., Schramm, D., Lee, T. K., Thurston, W. D., & Barton, A. W. (2014). Training child welfare professionals to support healthy couple relationships: Examining the link to training transfer. Journal of Public Child Welfare, 8(5), 560-583. doi: 10.1080/15548732.2014.953719.

Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31.

Hawkins, A. J., Blanchard, V. L., Baldwin, S. A., & Fawcett, E. B. (2008). Does marriage and relationship education work? A meta-analytic study. Journal of Consulting and Clinical Psychology, 76, 723–734.

Hawkins, A. J., & Fackrell, T. A. (2010). Does relationship and marriage education for lower-income couples work? A meta-analytic study of emerging research. Journal of Couple & Relationship Therapy, 9, 181–191. doi:10.1080/15332691003694927

Higginbotham, B., & Adler-Baeder, F. (2010). Enhancing knowledge and agreement among ethnically and economically diverse couples in stepfamilies with Smart Steps: Embrace the Journey program. Journal of Extension [On-line], 48(1). Available at: http://www.joe.org/joe/2010february/iw7.php

Jacobs, F. H. (1988). The five-tiered approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs, New York: Aldine DeGruyter.

McCann, A., Peterson, J., & Gold, A. J. (2009). Using planning and evaluation tools to target Extension outputs & outcomes: The New England Private Well Symposium example. Journal of Extension [On-line], 47(3) Article 3TOT4. Available at: http://www.joe.org/joe/2009june/tt4.php

McCawley, P. F. (2001). The logic model for program planning and evaluation. Moscow, ID: University of Idaho Extension. Retrieved from: http://www.uidaho.edu/extension/LogicModel.pdf

Mitchell, L., Walters, R., Thomas, M. L., Denniston, J., McIntosh, H., & Brodowski, M. (2012). The Children's Bureau vision for the future of child welfare. Journal of Public Child Welfare, 6, 550–567.

National Resource Center for Healthy Marriage and Families (2012). Strong Relationships, Strong Ffamilies: Integrating healthy relationship education skills into social services. (Product No. 050). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Family Assistance.

Radhakrishna, R. B., & Relado, R. Z. (2009). A framework to link evaluation questions to program outcomes. Journal of Extension [On-line], 47(3) Article 3TOT2. Available at: http://www.joe.org/joe/2009june/tt2.php

Sar, B. K., Antle, B. F., Bledsoe, L. K., Barbee, A. P., & Van Zyl, M. A. (2010). The importance of expanding home visitation services to include strengthening family relationships for the benefit of children. Children and Youth Services Review, 32, 198-205. doi:10.1016/j.childyouth.2009.08.016

Scarrow, A., Futris, T. G., & Fuhrman, N. E. (2014). The factors associated with child welfare professionals' application of relationship education. Children and Youth Services Review, 46, 265-275. doi: http://dx.doi.org/10.1016/j.childyouth.2014.08.023

Schramm, D., Futris, T. G., Galovan, A. M., & Allen, K. (2013). Is relationship and marriage education relevant and appropriate to child welfare? Children and Youth Services Review, 35(3), 429-438. doi: 10.1016/j.childyouth.2012.12.013

Schulz, M. S., Pruett, M. K., Kerig, P. K., & Parke, R. D. (Eds.). (2010). Strengthening couple relationships for optimal child development: Lessons from research and intervention. Washington, DC: American Psychological Association.

Small, S. A., Cooney, S. M., & O'Connor, C. (2009). Evidence-informed program improvement: Using principles of effectiveness to enhance the quality and impact of family-based prevention programs. Family Relations, 58, 1–13. doi: 10.1111/j.1741-3729.2008.00530.x

Taylor-Powell, E., Jones, L., & Henert, E. (2003). Enhancing program performance with logic models. Madison, WI: University of Wisconsin-Extension. Retrieved from: http://www.uwex.edu/ces/lmcourse/

White, D. J., Arnold, M. E., & Lesmeister, M. (2008). Using focus groups to evaluate youth development program direction. Journal of Extension [On-line], 46(6) Article 6RIB3. Available at: http://www.joe.org/joe/2008december/rb3.php