Using an Engaged Scholarship Symposium to Change Perceptions: Evaluation Results

Introduction

Although engaged scholarship (ES) is a growing initiative in many institutions nationwide, it is not new to land-grant colleges and universities, whose missions are traditionally compatible with community engagement (Kellogg Commission on the Future of State and Land-Grant Universities [Kellogg Commission], 1999). Proper ES entails a symbiotic relationship between the community and the university. ES involves instructors and students in a reciprocal partnership with their communities through the three major activities faculty typically perform: teaching, research, and service. These relationships are beneficial for all involved parties: Students gain hands-on experience, communities engage in expert collaboration, and faculty observe examples of the synergy that arises between theory and practice (Paynter, 2014). However, despite these obvious benefits, the institutionalization of ES has been challenging for many public colleges and universities (Jameson, Clayton, Jaeger, & Bringle, 2012; Moore & Ward, 2010).

Faculty often are hesitant to get involved in campus–community partnerships because of a lack of associated institutional support and academic rewards (Buys & Bursnall, 2007; Jameson et al., 2012). Calleson, Jordan, and Seifer (2005) suggested that the nature and scope of scholarly productivity and dissemination of scholarly products should be broadened to include community-engaged research. This expansion would allow faculty working on community engagement to generate scholarly products that address community priorities while also meeting university tenure and promotion requirements. Calleson et al. (2005) highlighted three primary types of community-engaged scholarship products: peer-reviewed articles, applied products, and community dissemination products. These products are used to disseminate knowledge through mediums that benefit academics, practitioners, and community members alike (Calleson et al., 2005). Extension professionals and representatives in communities can benefit from these collaborations and the associated dissemination of knowledge.

In spring 2015 East Carolina University (ECU) implemented the Engaged Scholarship Symposium to address institutional issues and faculty and administrator perceptions related to ES. A planning committee comprising faculty experienced in community-engaged research planned this 2-day event that targeted ECU faculty and administrators. The goals of the symposium were to increase awareness of ES as a platform for integrating research, teaching, and service among faculty and administrators and to potentially change unfavorable perceptions about ES through education and testimonials. With guidance from the planning committee, a group of graduate students enrolled in a program evaluation course assessed the event. That evaluation, which we took part in and report on here, explored the effect of the symposium on faculty and administrator perceptions of ES, intent to work on ES projects, and institutional commitment to ES.

Literature Review

In 1995, the American Association for Higher Education (AAHE) challenged colleges and universities to become "engaged campuses" (Hodge, Lewis, & Kramer, 2001). The AAHE detailed a need for higher education institutions to promote engagement and communal involvement through learning opportunities with students and faculty members (Hodge et al., 2001). Similarly, in 1999 the Kellogg Commission urged leaders at state universities and land-grant colleges to move their institutions beyond traditional "outreach and service" and toward a commitment to "sharing and reciprocity" in their respective communities. More recently, Reed, Swanson, and Schlutt (2015) highlighted the importance of infusing Extension and engagement throughout the entire university, stating that "engagement should be the overarching goal of the land-grant system" ("Introduction," para. 2) and that Extension should be at the forefront of making this happen. Despite these calls to action, the institutionalization of community-engaged scholarship has met challenges in many academic institutions, including land-grant and public research universities (Jameson et al., 2012).

One major challenge is inconsistent views among faculty and administrators about the scholarship of ES, which can hinder faculty members' perceptions of ES and their willingness to participate in it. For example, Sobrero and Jayaratne (2014) examined department heads' perceptions of the role of community engagement scholarship in the faculty promotion and tenure process in relation to the five realms of scholarship that their institutions use in the standards for promotion and tenure. Those realms include teaching and mentoring of undergraduate and graduate students, discovery of knowledge through discipline-guided inquiry, creative artistry and literature, technological and managerial innovation, Extension and engagement with constituencies outside the university, and service in professional societies, including service and engagement within the university (Sobrero & Jayaratne, 2014). They found that the realm of Extension and engagement with constituencies outside the university was rated the third most important; however, a large standard deviation for responses indicated wide variation among department heads regarding faculty effort in Extension and engagement (Sobrero & Jayaratne, 2014). The authors concluded that a lack of institutional support might thwart the intent of individual faculty members to participate in ES (Sobrero & Jayaratne, 2014). There is no doubt that if faculty members continue to be unrewarded for their efforts in ES, community approaches to scholarship may not be strengthened (Bloodworth et al., 2014; Saltmarsh, Giles, Ward, & Buglione, 2009). For this reason, the Engaged Scholarship Symposium planning committee made a concerted effort to engage both faculty and administrators in the symposium.

Indeed, to create a supportive academic climate for ES, it is necessary to revise unit codes that guide faculty reappointment, promotion, and tenure; however, more effort is needed to make a significant shift in academic culture at college and departmental levels, where these decisions reside (Doberneck, Glass, & Schweitzer, 2011). Evidence of a culture change is reflected in administrator and faculty perceptions of what constitutes ES and how it is valued in the tenure and promotion process. ECU's Engaged Scholarship Symposium was implemented and evaluated to document this culture change.

Methods

Participants and Procedures

To assess changes in faculty and administrator perceptions resulting from the symposium, we administered pretests and posttests to symposium participants and took steps to control for potential selection bias. Because individuals who attended the symposium may have had greater knowledge of or more interest in ES, using the evaluation instruments with only symposium attendees could have introduced a selection bias problem. If a sample is selected from individuals who have high pretest scores, their posttest scores are more likely to be closer to the mean, due to the regression to the mean, which would make estimating changes difficult. Selection bias also makes generalizing from a sample more difficult. Consequently, to control for selection bias, we also administered the pretest to a randomly selected control group of 150 university faculty who chose not to attend the symposium, indicating potentially less interest in the ES event. The control group received only the pretest. We selected symposium participants using the registration rosters and sign-up logs, and they completed both the pretest and the posttest. We received institutional review board approval prior to data collection.

The pretest was administered online via Qualtrics (a web-based, online survey software) and was available to participants via email. To improve response rates, participants in all groups were incentivized with the chance to win $50 worth of professional developmental funds. Winners were chosen through the use of a random number generator that selected unique numbers associated with participants' email addresses. To maintain participant anonymity, email addresses were not associated with individual responses.

As shown in Table 1, 76 registrants received emails with a link to the pretest. Forty-four individuals completed the pretest, resulting in a response rate of 57.9%. After the symposium, emails containing a link to the pretest were sent to the 150 members of the control group. Twenty-nine of those individuals completed the pretest, resulting in a response rate of 19.3%. This response rate is consistent with other web-based unsolicited surveys (Sauermann & Roach, 2013) and may indicate a low interest in ES among the faculty in the control group. This view is supported by estimates from the university indicating that out of a total of 1,516 full- and part-time faculty, about 100 are active in ES.

We sent posttests, via Qualtrics, to all individuals who registered for the ES event 2 days after it concluded. The posttest contained the same questions as the pretest; however, additional questions related to specific symposium events were included for evaluation purposes. As shown in Table 1, 92 emails containing a link to the posttest were sent. Forty-one attendees completed the posttest, resulting in a response rate of 44.6%. Twenty-seven participants completed both the pretest and posttest. We used pretest data to establish an ES knowledge baseline at the university and posttest data to establish a postsymposium baseline and to determine differences in participants' attitudes and intentions from before to after the symposium.

Table 1.
Pretest and Posttest Distribution and Response Rates

Group	Test sent (#)	Replied (#)	Response rate
Pretest	76	44	57.9 %
Posttest	92	41	44.6%
Control	150	29	19.3%

Instruments

Pretest and posttest questions were designed to evaluate participant knowledge of ES concepts, ES experience, and examples of ES as well as effects of the symposium on the participants' perceptions of ES. Questions also addressed whether ES was recognized as part of the tenure and promotion review in participants' departments and whether participants thought ES should be given more weight in the faculty tenure and promotion process. As noted earlier, the posttest also included symposium-specific questions. These additional questions were designed for evaluation purposes and were intended to measure whether participants forged professional relationships and were informed of funding opportunities while attending the symposium.

The planning committee oversaw the development of the pretest and posttest, which were created by the graduate students. As part of the development of the pretest and posttest, members of the planning committee (N = 5) completed an online open-ended survey, providing words and phrases they felt best summarized concepts of ES and depicted the most common misconceptions about ES. The survey data were collected and tallied and then cross-referenced with ES literature for the purpose of creating a final list of words and phrases representing positive components of and misconceptions about ES. This process generated 17 items (nine positive components and eight misconceptions) that were included in both the pretest and the posttest. Appendix A shows the questions for both the pretest and the posttest, including the list of ES positive components and misconceptions.

Results

Tables 2–5 outline descriptive statistics for select pretest and posttest items (we can provide full results upon request), and Appendix B contains descriptive statistics for the pretest and control groups.

Table 2.
Descriptive Statistics for Pretest and Posttest Groups—Question 1

Construct	Group	No. of respondents	Min.	Max.	M	SD
Index of positive components of ES (the sum of all components selected correctly)	pretest	44	3	9	5.98	1.77
	posttest	41	1	9	6.85	2.21
Index of misconceptions about ES (the sum of all misconceptions selected)	pretest	44	0	7	2.64	1.71
	posttest	41	0	7	2.83	1.84
Correct positive components picked up from pretest to posttest	posttest	27	−6	3	0.74	2.10
Misconceptions picked up from pretest to posttest	posttest	27	−2	3	0.26	1.13
Note. ES = engaged scholarship.

Table 3.
Descriptive Statistics for Pretest and Posttest Groups—Question 4

Question	Group	No. of respondents	Yes	Do not know	No	M	SD
Is ES recognized as part of tenure and promotion review in your department?	pretest	42	15	15	12	2.07	0.81
	posttest	41	20	11	10	2.24	0.83
Note. ES = engaged scholarship.

Table 4.
Descriptive Statistics for Pretest and Posttest Groups—Question 8

Question	Group	No. of respondents	Yes, I have been for sometime	No, but intend to in the near future	No, and I do not intend to	M	SD
Do you currently work on ES project(s)?	pretest	43	24	15	4	2.47	0.67
	posttest	39	31	6	2	2.75	0.55
Note. ES = engaged scholarship.

Table 5.
Descriptive Statistics for Pretest and Posttest Groups—Questions 9 and 10

Question	Group	No. of respondents	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree	M	SD
I feel that the university should put more emphasis on ES.	pretest	43	1	2	8	21	11	3.91	0.92
	posttest	41	0	2	7	14	18	4.17	0.89
I feel that ES can enhance the academic environment at the university.	pretest	43	0	1	6	21	15	4.16	0.75
	posttest	41	0	0	6	16	19	4.32	0.72
I feel that ES should be given more weight in the faculty tenure and promotion process.	pretest	43	1	1	14	16	11	3.81	0.93
	posttest	41	0	3	5	14	19	4.20	0.93
Note. ES = engaged scholarship.

To identify the intensity of the selection bias for symposium participants, we conducted the Mann-Whitney U-test. This test allowed us to compare between-group mean ranks as the distributions of the two groups were not similarly shaped; Table 6 shows these ranks, Mann-Whitney U scores, and z scores. The Mann-Whitney U-test is analogous to the independent-samples t-test; it is appropriate when the variables are not normally distributed and are at least of ordinal scale. Overall, there were no statistically significant differences between the mean ranks of the control and pretest groups regarding their selection of misconceptions or recognition of ES in the tenure and promotion process. However, there were statistically significant differences between the mean ranks of the two groups regarding their selection of positive ES components, their recognition that they were working on ES projects at the time, and their perceptions of the need for more emphasis on ES, ES's potential for enhancing the academic environment at the university, and the need for more weight for ES. This circumstance may be due to the selection bias of the symposium attendees; these individuals seemed to view ES as an important component to overall university success. The pretest group's higher scores made estimating the differences between the pretest results and the posttest results more difficult due to the regression to the mean. Because the initial level of understanding of what constitutes ES was high, there may have been less room for a change in knowledge to occur as a result of attending the event.

Table 6.
Mann-Whitney U-Test for Differences Between Pretest and Control Groups

Test question	Group	No. of respondents	Mean rank	Mann-Whitney U z
Index of positive components of ES (total number of components selected)	control	26	27.21	356.500**
	pretest	44	40.40	−2.645
Index of misconceptions about ES (total number of misconceptions selected)	control	26	34.42	544.000
	pretest	44	36.14	−0.345
Is ES recognized as part of tenure and promotion review in your department?	control	27	36.00	513.000
	pretest	42	34.36	−0.709
Do you currently work on ES project(s)?	control	28	29.59	422.500*
	pretest	43	40.17	−2.295
I feel that the university should put more emphasis on ES.	control	28	25.38	304.500**
	pretest	43	42.92	−3.664
I feel that ES can enhance the academic environment at the university.	control	26	27.44	362.500**
	pretest	43	39.57	−2.630
I feel that ES should be given more weight in the faculty tenure and promotion process.	control	27	25.07	299.000**
	pretest	43	42.05	−3.561
Note. ES = engaged scholarship. *p < .05. **p < .01.

We conducted the Wilcoxon signed-rank test, a nonparametric test designed for related-samples analysis, to compare the pretest and posttest results. This test is equivalent to the dependent-samples t-test, without the normal distribution assumption. The Wilcoxon signed-rank test transforms data into ranks. Table 7 lists negative, positive, and tie ranks between pretest and posttest groups for selected questions as well as z scores. The negative ranks show the number of times someone selected a lower ranking on the posttest compared to the pretest (e.g., four individuals selected fewer positive components of ES on the posttest than on the pretest). The positive ranks show the number of times someone selected a higher ranking on the posttest compared to the pretest (e.g., 14 individuals selected more positive components of ES on the posttest than on the pretest). The differences between the mean ranks on pretests and posttests for symposium participants are not statistically significant at the 95% confidence level for most of the questions. Despite high scores on the pretest, the posttest showed that after the symposium participants were more likely to suggest that the university should put more weight on ES (11 positive ranks). The posttest also showed that after the symposium several respondents realized they already were doing ES projects (four positive ranks).

Table 7.
Wilcoxon Signed-Rank Test for Differences Between Pretest and Posttest

Pair #	Paired pretest/posttest concept	Ranks	f	Mean rank	z
Pair 1	Index of positive components of ES (total number of components selected)	Negative	4	10.25	−1.965*
		Positive	14	9.29
		Ties	9
Pair 2	Index of misconceptions about ES (total number of misconceptions selected)	Negative	5	4.60	−1.283
		Positive	7	7.86
		Ties	15
Pair 3	Is ES recognized as part of tenure and promotion review in your department?	Negative	6	6.25	−0.411
		Positive	5	5.70
		Ties	15
Pair 4	Do you currently work on ES project(s)?	Negative	0	0.00	−2.000*
		Positive	4	2.50
		Ties	21
Pair 5	I feel that the university should put more emphasis on ES.	Negative	2	7.00	−1.459
		Positive	8	5.13
		Ties	17
Pair 6	I feel that ES can enhance the academic environment at the university.	Negative	2	6.75	−1.155
		Positive	7	4.50
		Ties	17
Pair 7	I feel that ES should be given more weight in the faculty tenure and promotion process	Negative	2	8.75	−2.066*
		Positive	11	6.68
		Ties	14
Note. As indicated by Dimitrov and Rumrill (2003), the raw scores and the net gain scores give very similar results in our case. For Pair 4, 3 = yes; 2 = no, but intend to; and 1 = no and do not intend to. ES = engaged scholarship. *p < .05.

The symposium seems to have fulfilled one of its goals: increasing participants' awareness regarding ES. We were concerned that selection bias and the regression to the mean would have hindered our ability to assess accomplishment of this goal. Figure 1 shows the differences (which are not statistically significant) among control, pretest, and posttest results regarding whether ES was recognized as part of the tenure and promotion process in a participant's department. Forty-eight percent of the control group chose "I don't know" as their answer, compared to 36% of the pretest group and 19% of the posttest group.

Figure 1.
Recognition of Engaged Scholarship as Part of the Tenure and Promotion Process: Responses from All Groups

About 32% of the pretest group neither agreed nor disagreed that the university should place more weight on ES, whereas only 7.4% of respondents selected this answer after the symposium. This shift toward desiring that more weight be placed on ES is statistically significant and is detailed in Figure 2.

Figure 2.
Pretest and Posttest Group Members' Responses Regarding Whether Engaged Scholarship Should Be Given More Weight in the Tenure and Promotion Process

Finally, the symposium seems to have enhanced participants' recognition of their existing involvement in ES projects and their intentions to be involved in ES in the future (Figure 3). After the symposium, individuals' recognition of whether they were participating in ES changed, with more people indicating that they were active in ES projects.

Figure 3.
Pretest and Posttest Group Members' Responses Regarding Existing Involvement and Potential Future Involvement in Engaged Scholarship Projects

We conducted comparisons of those who replied to the pretest only and those who replied to both the pretest and the posttest. These comparisons were used to determine whether there were systematic differences between those who did not reply to the posttest and those who did. No statistically significant differences were found. Both sets of respondents were relatively positive in their attitudes toward ES. Findings show that the symposium helped increase ES recognition.

Discussion and Conclusions

The objective of our study was to evaluate the short-term outcomes of the Engaged Scholarship Symposium. Specifically, we wanted to know whether the symposium achieved its goals, which were to increase awareness of ES as a platform for integrating research, teaching, and service and to change unfavorable perceptions of ES among faculty and administrators. Despite selection bias, we found some statistically significant differences between pretest and posttest scores, indicating that the symposium was successful with regard to some aspects of its goals. For example, the posttest respondents tended to agree and strongly agree to a higher extent than the pretest group that the university should put more weight on ES in the tenure and promotion process. Furthermore, symposium participants were more likely to suggest that more weight be given to ES in the tenure and promotion process as compared to the control group.

Although tenure and promotion expectations are beginning to change in favor of ES on some campuses, there is more work to be done before ES activities are fully supported and rewarded in the faculty promotion process. ES recognition in the tenure and promotion process may be especially challenging for Extension programs that have not yet changed to more current community–university engagement models, which emphasize reciprocal partnerships between universities and communities rather than the traditional outreach and service approach (Bruns & Franz, 2015). Historically, Extension has followed the program development model, which is being challenged by other engagement models. According to Bruns and Franz (2015), "For Extension to leverage an important place in community–university engagement, it must fully align with the standards for assessing successful university engagement" (p. 158).

In our study, we also found that participants gained a better understanding of what constitutes ES, as evidenced by their ability to recognize that some of their existing work included ES projects. We believe that the observed shift from participants' intention to participate in ES to the recognition of current projects as ES suggests that the symposium achieved its goal of increasing awareness about ES as scholarly work, at least for a small number of participants. Once someone knows he or she is doing an ES scholarly project and not a service project, that person may find new funding venues and new opportunities for collaborations, both of which may help with the recognition of ES work in the tenure and promotion process. Overall, these findings are encouraging for faculty and administrators looking for effective ways to educate and promote ES as scholarly work on their campuses.

There are limitations to our study. The way we measured the number of misconceptions about ES may have contributed to the lack of differences between control and pretest groups. Further development of ES measurements may fine-tune and enhance our understanding of perceptions about components that make up ES and common misconceptions about ES. Although in the short term the symposium may increase attendee intent to participate in ES projects, further evaluations are needed to determine long-term changes. Furthermore, even though we used a quasi-experimental design with a control group and pretest-posttest group, the symposium participants were not randomly selected. Our results indicate a snapshot of one event at one university, and more research is needed, both across time and at different institutions. Finally, as we noted above, results indicate a predicted selection bias in the people who attended the symposium, as they appeared to view ES as an important component to overall university success.

A symposium designed specifically to encourage open dialogue between faculty and administrators can lead to increased awareness and changes in attitudes on topics such as ES, as shown in the study reported here. In this case, the event provided an opportunity to give recognition to ES on campus and deliver additional educational support (e.g., networking and funding support). Essentially, our findings provided evidence that an ES symposium can be a synergistic platform for promoting community engagement and scholarship on campus and that it may help "[erode] the (false) boundaries between knowledge production and knowledge use" (McCormack, 2011, p. 112) that exist in academia. That is, the traditional model, wherein academic institutions are producers of knowledge and practitioners (society) are users of knowledge with the two rarely intersecting, continues to dominate research practice in most institutions today. Engaged scholarship can offer a framework that helps bridge research and practice through sustained community partnership, which is at the heart of Extension work. Furthermore, according to our findings, an ES symposium might be exactly what institutions need to begin and/or continue the ES conversation and, perhaps, serve as a starting point for reexamining how the academy measures the impacts of research outputs (i.e., traditional peer-reviewed publications versus, for example, direct changes in the quality of life of community members). Land-grant institutions and Extension professionals would benefit from implementing ES events, such as a symposium, to further support their mission of developing mutually beneficial partnerships and relationships in their respective communities. Ultimately, we believe there is potential for ES symposiums to draw more people to ES and extend the reach of ES overall.

Practical suggestions for future ES symposiums may include the recommendation to conduct such events regularly, thereby allowing a wider range of participants to get involved. A university planning to position itself to serve the surrounding community may need to pay closer attention to encouraging and rewarding ES projects (LaBelle, Anderson-Wilk, & Emanuel, 2011; Saunders & Reese, 2011), and conducting symposiums on ES may encourage institutions to put more weight on ES, help individuals recognize more of their existing work as ES, and promote individuals' intent to participate in such projects. Combined with clear rules for tenure and promotion (e.g., a type of road map as outlined by Saunders & Reese, 2011), symposiums may help promote ES and incentivize both faculty and administrators in their ES efforts. Indeed, ES is an important part of the work environment for both community-engaged faculty and Extension professionals, and events such as the ECU symposium can serve as important bridges between those groups.

Acknowledgments

We would like to thank Lindsay Gold, Kelvin Stallings, Robert Villanueva Jr., John Wynne, and Taylor Quinn Yancey for their work on the program evaluation. This article would not have been possible without their help. We also would like to thank our anonymous reviewers; their thoughtful feedback helped increase the quality of our manuscript. All remaining errors and omissions are ours.

References

Bloodworth, L., Haines, S., Kearney, K., Lipowski, E., Sorensen, T., Thompson, D., & Lau, Y. S. (2014). Considerations for embracing and expanding community engaged scholarship in academic pharmacy: Report of the 2013-2014 Research and Graduate Affairs Committee. American Journal of Pharmaceutical Education, 78(8), 1–9.

Bruns, K., & Franz, N. (2015). Cooperative Extension program development and the community-university engagement movement: Perspectives from two lifelong Extension professionals. Journal of Human Sciences and Extension, 3(2), 156–169.

Buys, N., & Bursnall, S. (2007). Establishing university-community partnerships: Processes and benefits. Journal of Higher Education Policy and Management, 29(1), 73–86.

Calleson, D., Jordan, C., & Seifer, S. (2005). Community-engaged scholarship: Is faculty work in communities a true academic enterprise? Academic Medicine, 80(4), 317–321.

Dimitrov, D., & Rumrill, P. (2003). Pretest-posttest designs and measurement of change. Work, 20(2), 159–165.

Doberneck, D., Glass, C., & Schweitzer, J. (2011). Beyond activity, place, and partner: How publicly engaged scholarship varies by intensity of activity and degree of involvement. Journal of Community Engagement and Scholarship, 4(2), 18–28.

Hodge, G., Lewis, T., & Kramer, K. (2001). Collaboration for excellence: Engaged scholarship at Collin County Community College. Community College Journal, 25(9), 675–690.

Jameson, J. K., Clayton, P. H., Jaeger, A. J., & Bringle, R. G. (2012). Investigating faculty learning in the context of community-engaged scholarship. Michigan Journal of Community Service Learning, 18(2), 40–55.

Kellogg Commission on the Future of State and Land-Grant Universities. (1999). Returning to our roots: The engaged institution (Report 3). Washington, DC: National Association of State Universities and Land-Grant Colleges.

LaBelle, C., Anderson-Wilk, M., & Emanuel, R. (2011). Leveraging new media in the scholarship of engagement: Opportunities and incentives. Journal of Extension, 49(6), Article 6FEA3. Available at: https://www.joe.org/joe/2011december/a3.php

McCormack, B. (2011). Engaged scholarship and research impact: Integrating the doing and using of research in practice. Journal of Research in Nursing, 16(2), 111–127.

Moore, T. L., & Ward, K. (2010). Institutionalizing faculty engagement through research, teaching, and service at research universities. Michigan Journal of Community Service Learning, 17(1), 44–58.

Paynter, S. (2014). Tackling wicked problems through engaged scholarship. Journal of Community Engagement and Scholarship, 7(1), 48–59.

Reed, A. S., Swanson, L., & Schlutt, F. (2015). Timberline manifesto: Seven concepts linking Extension and engagement. Journal of Extension, 53(4), Article 4COM1. Available at: https://www.joe.org/joe/2015august/comm1.php

Saltmarsh, J., Giles, D. E., Ward, E., & Buglione, S. M. (2009). Rewarding community‐engaged scholarship. New Directions for Higher Education, 2009(147), 25–35.

Sauermann, H., & Roach, M. (2013). Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Research Policy, 42(1), 273–286.

Saunders, K. S., & Reese, D. (2011). Developing a roadmap for excellence in Extension. Journal of Extension, 49(3), Article 3TOT2. Available at: https://www.joe.org/joe/2011june/tt2.php

Sobrero, P., & Jayaratne, K. S. (2014). Scholarship perceptions of academic department heads. Journal of Higher Education Outreach and Engagement, 18(1), 123–150.

Appendix A

Content of the Pretest and Posttest Instruments

Pretest	Posttest
Ia) Are you a faculty/administrator/other?a (check one box) Please select individual components of ES that you include in your definition: (check all that apply) Mutually beneficial partnership Combines three academic pillars: research, teaching, and service Collaborative partnership Shared power Rigorous research Publications accessible to many Expanding knowledge base Society benefits Multidisciplinary Added misconceptions: service-learning civic engagement volunteerism time-consuming Difficult to produce scholarly products. community outreach service activity How many people in your department are involved in ES? Please estimate using numbers __________ or percentages _________ Please, provide examples of ES in your unit: (essay-type question) Is ES recognized as part of tenure and promotion review in your department? Yes/No/I do not know ES is reported in which section of [personnel dossier] in your department: (check all that apply?) In research section In service section In teaching section Not included Is ES defined in your unit code? Yes/No/I do not know. Currently, ES is important for tenure and promotion decisions in my department: (Agree 5-Disagree 1 scale). Do you currently work on engaged scholarship project(s)(ES)? YES, I have been for sometime. NO, but I intend to in the near future.* NO, and I do not intend to in the near future. *skip pattern to add question: I would like to be contacted by [Contact Person] from the [Dedicated Office] for more information about engaged scholarship, enter email [field] I feel that the university should put more emphasis on ES (Agree 5-Disagree1 scale) I feel that engaged scholarship can enhance the academic environment at the university (Agree 5-Disagree 1 scale) I feel that ES should be given more weight in the faculty tenure and promotion process (Agree 5-Disagree 1 scale) n/a n/a	Ia) Are you a faculty/administrator/other? (check one box) Please select individual components of ES that you include in your definition: (check all that apply) Mutually beneficial partnership Combines three academic pillars: research, teaching, and service Collaborative partnership Shared power Rigorous research Publications accessible to many Expanding knowledge base Society benefits Multidisciplinary Added misconceptions: service-learning civic engagement engagement of the audience volunteerism time-consuming Difficult to produce scholarly products. community outreach service activity How many people in your department are involved in ES? Please estimate using numbers __________ or percentages _________ Please, provide examples of ES in your unit: (essay-type question) Is ES recognized as part of tenure and proportion review in your department? Yes/No/I do not know n/a n/a n/a Do you currently work on ES project(s)(ES)? YES, I have been for sometime. NO, but I intend to in the near future.* NO, and I do not intend to in the near future. *skip pattern to add question: I would like to be contacted by [Contact Person] from the [Dedicated Office] for more information about ES, enter email [field] I feel that the university should put more emphasis on engaged scholarship (Agree 5-Disagree 1 scale) I feel that ES can enhance the academic environment at the university (Agree 5-Disagree 1 scale) I feel that ES should be given more weight in the faculty tenure and promotion process (Agree 5-Disagree 1 scale) Through the symposium, I have developed professional and/or community relationships that will be beneficial for future ES activities Through the symposium, I have become aware of funding opportunities available at the university for ES projects.
Note. ES stands for engaged scholarship and was not abbreviated in the original surveys; n/a stands for not applicable. a"Other" was added to the pretest for the control group and to the posttest at the request of a respondent who took the pretest.

Appendix B

Descriptive Statistics for the Pretest and Control Groups

In the tables that follow, ES stands for the term engaged scholarship, which was not abbreviated in the original surveys.

Question	Group	# respondents	Range	Min.	Max.	M	SD
Index of positive components of ES (the sum of all selected components correctly)	pretest	44	6	3	9	5.98	1.77
	control	26	8	1	9	4.54	2.32
Index of misconceptions about ES (the sum of all misconceptions selected)	pretest	44	7	0	7	2.64	1.71
	control	26	7	0	7	2.50	2.12

Question	Group	# respondents	Yes	Do not know	No	M	SD
Is ES recognized as part of tenure and promotion review in your department?	pretest	42	15	15	12	2.07	0.81
	control	27	9	13	5	2.15	0.72

Question	Group	# respondents	Yes, I have been for sometime	No, but intend to in the near future	No, and I do not intend to	M	SD
Do you currently work on ES project(s)?	pretest	43	24	15	4	2.47	0.67
	control	28	11	5	12	1.96	0.92

Question	Group	# respondents	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree	M	SD
I feel that ES should be given more weight in the faculty tenure and promotion process.	pretest	43	1	1	14	16	11	3.81	0.93
	control	27	4	4	11	8	0	2.85	1.03
I feel that the university should put more emphasis on ES.	pretest	43	1	2	8	21	11	3.91	0.92
	control	28	3	4	13	6	2	3.00	1.05
I feel that ES can enhance the academic environment at the university.	pretest	43	0	1	6	21	15	4.16	0.75
	control	26	2	1	7	13	3	3.54	1.03