Process Monitoring Evaluation of an Online Program for Parents

Introduction

The eXtension Just in Time Parenting (JITP) program is an electronically delivered age-paced parenting newsletter developed by a national network of Extension Family Life Specialists (see JITP.extension.org). JITP was designed to reach parents with research-based information about pregnancy, parenting, child development, health, safety, nutrition, school readiness, family stress management, and couple relationships. The newsletters are age-paced, such that parents receive issues based on their infant or child's birth date. Traditionally, printed parenting newsletters were mailed or hand delivered, yet the electronic version of the JITP newsletters were designed to reach more parents and meet the needs of contemporary parents (Radey & Randolph, 2009; Rothbaum, Marland, & Jannsen, 2008; Walker, Dworkin, & Connell, 2011).

Past evaluations of the effectiveness of traditional parenting newsletters have shown that age-paced newsletters improve parents' knowledge of child development, parenting self-confidence, ability to be nurturing, and other positive parenting behaviors (Cudaback, et al., 1985; Riley, Meinhardt, Nelson, Salisbury, & Winnett, 1991). The effectiveness of the electronic delivery mode, however, has yet to be evaluated and is the focus of this article, although the impact of JITP content is currently being assessed. The purpose of the study reported here was to employ a process monitoring evaluation in which we explored the effectiveness of the JITP by analyzing data surrounding parent reach and engagement.

JITP Subscription Process

There are two ways in which the JITP audience can access the newsletter series; they go directly to the website, or they can subscribe to the newsletters. To subscribe, parents first submit their email address and either their due date or their child's birth date. Then, they must respond to a confirmation email to complete the subscription process (two-step process). Once subscribed, parents receive an email with a link to the appropriate newsletter, programmed according to their due date or their child's age. JITP currently has three prenatal newsletters delivered quarterly, issues from newborn to 12 month delivered monthly, and issues from 13 to 60 months delivered bi-monthly. The idea is that when the emails are sent to JITP subscribers, they will open the email and click on the link to read the newsletters. However, parents who subscribe and receive the newsletters do not always open the email and click on the link to read the newsletters.

Process Monitoring

Extension educators have long recognized the importance of evaluating program impact (McCann, Peterson, & Gold, 2009; Rennekamp & Arnold, 2009). There is, however, a limited understanding of the importance of evaluating program implementation (Hughes, Bowers, Mitchell, Curtiss, & Ebata, 2012), even though it is critical to know whether a program is being implemented as originally planned. There are many perceived benefits of online program delivery (e.g., cost, time, convenience), yet a major limitation is the difficulty in tracking participants' responses and reactions during the program implementation phase. Therefore, it is important to employ process evaluations to understand the effectiveness and efficiency of the electronic delivery system (Patton, 1994) and to document how this is functioning (Rossi, Lipsey, & Freeman, 2004).

Purpose of the Study

The purpose of the research reported here was to conduct a process monitoring evaluation of the JITP electronic delivery system to further our understanding of reaching and engaging contemporary parents. To explore the effectiveness of the delivery system, we asked the following questions:

1. Where do JITP visitors access the site from?
2. How do JITP visitors access the site?
3. What types of content are JITP visitors accessing?
4. How do subscribers engage with the content?

Method

We collected data from Google Analytics, a visitor-tracking website program, and Bronto (http://www.bronto.com/), a commercial email marketing platform that manages subscriptions. We used Google Analytics to analyze website visit trends and track number of website visits and pageviews visitors made. We used Bronto to gather information about subscribers: subscription rate, open email's rate, and click link rate. In this article, we focus on the data collected in 2012 and present the results using descriptive statistics and paired-sample Wilcoxon Singed Rank test.

Results

Access

Google Analytics revealed that the JITP website received 111,057 visits from 75,969 individuals from 126 different countries or territories. They came from 126 countries, but most were from the United States followed by India, the United Kingdom, Philippines, and Canada. Visitors viewed the website in 76 different languages (using translation tool). The states with the largest numbers of viewers in the United States were California, Iowa, New York, Texas, and Illinois.

Among those visitors, about half (49%) accessed JITP using search engines such as Google, Yahoo, and Bing. Participants who came to the website via search engines frequently used common search words such as Just in Time Parenting, newborn baby care, MyPlate, friends, parenting tips, and children in school. Approximately 27% got to the site directly from email reminders (JITP subscribers), 24% were referred from other websites (i.e., university or Cooperative Extension websites), and .05% from Facebook. Approximately 15% of visits (around two times higher than 2011) were from mobile devices. In 2012 social media platforms were used less than other referral sources. Facebook directed 679 visits and 3,014 pageviews to JITP's website, and Twitter directed only 36 visits and 36 pageviews to the JITP website.

Pageviews

Data from Google Analytics also revealed that among 335,984 pages viewed, the eNewsletter page was viewed 52,127 times, making it the most viewed webpage, and the homepage was viewed 18,778 times. The eNewsletter page is different from the homepage—the homepage is the site root (the top-most directory of a website hierarchy; extension.org/parenting), and the eNewsletter page is nested under the site root and includes a list of all monthly JITP newsletters, in English and Spanish. Among the JITP newsletters, issues for the newborn, 1 month, 4 month, and 8 months old were the most frequently viewed. We found that more visitors accessed and downloaded newsletters for infants than prenatal or older children (Figure 1).

Figure 1.
Newsletter Issue Viewed by Age Groups

Subscriptions

Unlike Google Analytics, the Bronto delivery platform allowed us to examine data for those who subscribed to the newsletter. During 2012, 2,119 new people initially signed up to receive JITP newsletters (Table 1). Among them, more than half did confirm subscriptions. The states with the largest numbers of subscribers were: New Hampshire (155), Illinois (102), Iowa (84), Indiana (74), and Delaware (59).

Table 1.
From Registration to Active Subscription in 2012

Status
Initial Sign-up	2,119
Confirmed Subscribers	1,168
Unconfirmed	872
Bounced	73
Unsubscribed	6

JITP Newsletter Subscription and Open/Click Rates

Data from Bronto revealed that expectant parents were less likely to subscribe to and receive JITP newsletters than parents of infants, toddlers, and preschoolers (H(2) = 7.30, p < .05), and among those who subscribed and received JITP emails, on average 51% opened JITP emails. As can be seen in Figure 2, there was a proportionally higher open rate among expectant parents and parents of infants than parents of toddlers or preschoolers, H(2) = 8.80, p < .05. Figure 3 provides information about the numbers who clicked on the link to the newsletter. Among those who opened JITP emails, there was an average click rate of 49% actually to read the JITP newsletter. There was a higher click rate among parents of toddlers and preschoolers than parents of infants and expectant parents, H(2) = 7.56, p < .05 (Figure 3).

Figure 2.
Percent of subscribers Who Opened Emails

Figure 3.
Percent of Subscribers Who Clicked on the Link to Read Newsletters

Discussion and Implications

This evaluation of a parenting newsletter indicates a need to carefully reconsider how parents and caregivers use electronic delivery systems, so that program developers or program recruiters have realistic expectations for technology applications. There are a variety of types of online programs and a variety of ways to evaluate online programs at different stages of their development (Hughes et al., 2012). Because this process monitoring evaluation was specific to JITP newsletters, caution is warranted when generalizing this information to all online programs. At the same time, we believe that the results of the study provide several valuable implications for Extension educators who have developed online programs, or are considering areas for online program improvement.

• Considering that there were unexpected visitors from other countries, and there were more visitors than subscribers, it is worthwhile to promote any educational websites on various subjects for learning on search engines and directories such as Google, Yahoo, or Bing, if possible. This in combination with the ways that articles or posts are tagged will help individuals to quickly and easily find research-based information on any subject.
• Given that a growing number of parents accessed the JITP emails and newsletters through mobile phones, the JITP site has since been re-designed with a responsive template (e.g., mobile, desktop, laptop, iPad). Extension professionals who run other educational websites might consider using responsive templates, so Web pages are easy to read and interact with from a variety of devices.
• Referral data from Google Analytics revealed that only a small number of referrals to the JITP site came from social media platforms, yet the literature on parents' social media usage (Brenner & Smith, 2013) facilitated the addition of social media functions with the redesign of the website. It is possible that more parents would have come to the JITP site via social media if the social media functions would have been employed prior to the study. We have since revised the JITP site to include social media share functions and would recommend that other Extension professionals continue to consider this as a way to reach and engage contemporary parents, youth, and families.
• Results also revealed differential usage based on the age of the child. Google Analytics indicated that the newborn and 1 month issues were the most frequently accessed, indicating that expectant parents and parents of younger infants are seeking information on the Internet that leads them to JITP. However, the Bronto data indicated that fewer expectant parents or parents of younger infants subscribed to JITP newsletters. This discrepancy across data sources suggests that they should be targeted in new subscription recruitment efforts. Training and publicity about the availability of JITP, especially to educators, medical providers, and home visitors who work with those parents are in progress. Using email marketing campaigns or content management systems could help other Extension professionals who seek to expand the reach of their websites or evaluate sub-groups of their online audience.
• Given that only about half of parents who initially subscribed to the JITP newsletters actually confirmed their subscription indicates that better understanding of the two-step subscription process was needed. As a result, the JITP team enhanced communication with Extension educators and parents this process. Further, the JITP team is considering simplifying their registration process to one step in which parents would just have to enter in their email address and their child's birth date without confirming that they would like to subscribe. Simple registration process can increase participants of any online resources.
• More information is needed on the factors that influence open and click rates among parents and caregivers. With an analysis of the Bronto data, the JITP team realized that parents were more likely to click on the link to read the full newsletters when the message in the email was shorter (parents of toddlers and preschoolers received the shorter email). This encouraged the JITP team to shorten the emails that are sent to parents. Other Extension professionals who use email marketing campaigns should consider keeping email messages to their audiences brief and highlighting only the key points, encouraging them to click on the link to view the full content.

The process monitoring data helped JITP to prioritize program improvement efforts, although randomized controlled trials of program impact are also important and should be used in future research with electronic, as well as face to face, program delivery. Electronic delivery does not automatically solve the challenges of reaching more parents/caregivers. Further, online delivery systems can be costly because they require ongoing maintenance and technical expertise in addition to the time and money spent on curriculum development. Extension specialists and educators need to work together to develop strategies to most effectively recruit and engage parents, and ongoing process monitoring evaluations will be critical in this process.

References

Brenner, J., & Smith, A. (2013). 72% of online adults are social networking site users. Retrieved from: http://www.pewinternet.org/Reports/2013/social-networking-sites.asp

Cudaback, D., Darden, C., Nelson, P., O'Brien, S., Pinsky, D., & Wiggins, E. (1985). Becoming successful parents: Can age paced newsletters help? Family Relations, 34, 271-275.

Hughes, R. H., Bowers, J. R., Mitchell, E. T., Curtiss, S. L., & Ebata, A. T. (2012). Developing online family life prevention and education programs. Family Relations, 61, 711-727.

McCann, A., Peterson, J., & Gold, A. (2009). Using planning and evaluation tools to target Extension outputs & outcomes: The New England private well symposium example. Journal of Extension [On-line], 47(3) Article 3TOT4. Available at: http://www.joe.org/joe/2009june/tt4.php

Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15, 311 – 319. doi: 10.1016/0886-1633(84)90026-4

Radey, M., & Randolph, K. A. (2009). Parenting sources: How do parents differ in their efforts to learn about parenting? Family Relations, 58, 536-548.

Rennekamp, R. A., & Arnold, M. E. (2009). What progress, program evaluation? Reflections on a quarter-century of Extension evaluation practice. Journal of Extension [On-line], 47(3) 3COM1. Available at: http://www.joe.org/joe/2009june/comm1.php

Riley, D., Meinhardt, G., Nelson, C., Salisbury, M. J., & Winnett, T. (1991). How effective are age-paced newsletters for new parents? A replication and extension of earlier studies. Family Relations, 40, 247-253.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation; A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

Rothbaum, F., Martland, N., & Jannsen, J. B. (2008). Parents' reliance on the Web to find information about children and families: Socio-economic differences in use, skills and satisfaction. Journal of Applied Developmental Psychology, 29, 118-128.

Walker, S. K., Dworkin, J., & Connell, J. H. (2011). Variation in parent use of information and communications technology: Does quantity matter? Family & Consumer Sciences Research Journal, 40, 106-119.