Overcoming Language and Literacy Barriers: Using Student Response System Technology to Collect Quality Program Evaluation Data from Immigrant Participants

Identifying a Tool

This article presents a novel technique for ensuring representation of immigrant adult learners in program evaluation. It adopted survey instrument language translation and literacy level adaptations with "clicker" technology for a fun and efficient group data collection method.

Minnesota's Early Childhood Family Education (ECFE) program offers families of children up to age 5 weekly group educational sessions facilitated by licensed parent educators and early childhood educators (Early Childhood Family Education Coordinated Outreach, 2013). Due to wide differences in resources for evaluation among the 300 programs in the state, a method was developed for individual districts to efficiently assess parent perceptions of program impact on their own learning and on the development of their young children (Walker, 2014). The evaluation employs a retrospective pretest survey design to measure perceived change in 12 parenting skill and knowledge items and 5 child development items (Ham & Walker, 2014; Pratt, McGuigan, & Katzev, 2000). Additional items gather demographic and program satisfaction data.

A large urban school district adopted the evaluation method, yet the diversity of its outreach to over 4,500 parents in 11 program locations (Walker, Sysyn, Hennen-Clements, & Brown, 2012) required the survey to be translated into multiple languages, including accommodation for Karen families. Karen immigrants originate from Burma and Tham Hin, a refugee camp in Thailand (Administration of Children and Families, Office of Refugee Resettlement, 2009). The Karen people employ oral and written forms of language (Karen Buddhist Dhamma Dhutta Foundation, 2011). According to program staff, a number of ECFE parents are not literate in the Karen language, thus requiring survey administration in oral and written forms. A creative solution was also needed to collect responses from Karen families that would ensure high participation rates and an accurate and efficient data recording.

Student response system (SRS) technology was employed, suggested for use in Extension by Bird and McClelland (2010). SRS is either a software technology or a combination of hardware and software technologies that enable classroom instructors to create interactive polls. Teachers use software that integrates with Microsoft PowerPoint™ to create and post prepared questions. Students use response device hardware (i.e., "clickers") with numbers on one side and the device identification number on the other to respond. Aggregated poll results can immediately be viewed by respondents, offering felt engagement in interactive learning and data collection (Cain, Black, & Rohr, 2009; Heaslip, Donovan, & Cullen, 2014; Mula & Kavanagh, 2009). All responses received are recorded in the software program and are efficiently exported into a data management and analysis tool (e.g., Microsoft Excel). All data are deidentified upon collection—using only the clicker identification numbers. In addition to the traditional use of clickers for classroom instruction and engagement, recent efforts have employed clickers for research data collection in Extension (Parmer, Parmer, & Struempler, 2012). Although no published studies indicate the use of clickers with limited-English, limited-literacy adult audiences, it was reasoned that this method, in combination with oral translation, would permit secure data collection in an efficient manner that respects the collectivistic value of learning as part of group and individual comprehension needs.

Procedure

Survey administration occurred during regular ECFE class times. Prior to the class, the parent educator shared information about the evaluation purpose, survey, and data use with participants. The survey was translated into Karen (Figure 1). The SRS program was set up with individual survey items copied into separate slides (Figure 2). Although written forms of the survey were not used for data collection, participants with reading ability could refer to the written questions. Survey forms also were used to record clicker numbers.

Figure 1.
ECFE Evaluation Survey in Karen

Figure 2.
Slide with Single Question Response for SRS employment

After introductions by the teacher, the evaluator and the interpreter distributed informed-consent forms and introduced the clicker technology; also, clickers were distributed to participants. A sample slide was shown to give participants a chance to practice using the technology. A question common to all was asked: "Which of these colors do you like the most? A. Red, B. Blue, C. Yellow, D. Green, and E. Purple." Five response choices were selected to encourage familiarity with the number of response options in the Likert-scale survey questions. Participants' responses were recorded in the SRS program and, for this question only, displayed in a bar graph to demonstrate how the technology works. When participants expressed comfort with the technology, the survey questions were asked one at a time. The question appeared on the slide in English and Karen and was read in English by the evaluator. The translator then spoke the question in Karen, along with the response choices. Questions by the parents about the wording or meaning of an item were resolved through the translator and the evaluator. A monitor reviewed the data collection to ensure that all individuals with devices recorded their responses. When all responses were in, the next question was asked. The process continued until all survey questions were asked and answered. At the end of data collection, respondents were thanked for their time and participation. Collection of responses to 25 items from approximately 20 parents was completed in approximately 45 minutes. Data collection was repeated for three Karen classes.

Result

The method proved to be very successful as evidenced by the high response rate without missing data (100% to all questions) and popularity with participants. Comments indicated parent appreciation of slower pacing due to translation and the option of listening or reading questions. Parents also noted that the activity felt "game-like." Follow up discussion with staff indicated that parents understood the questions. Based on the integrity of the data and pattern of responses that paralleled data collected from other groups in the school district program and elsewhere in the state, there is no reason to expect that the data from the Karen families were different due to the data collection method. Finally, parent educators and English-language-learner teachers expressed appreciation for a method that respected literacy abilities and encouraged reading and English competency. They also remarked that data collection as a group and during class time encouraged Karen families, as representing a collective culture and as busy parents of young children, to fully participate.

References

Administration of Children and Families, Office of Refugee Resettlement (2009). Burma history and immigration to the United States: Burma geography, people, and history. Retrieved from Centers for Disease Control and Prevention website: http://www.cdc.gov/tb/publications/guidestoolkits/ethnographicguides/burma/chapters/chapter1.pdf

Bird, C., & McClelland, J. (2010). Have you used clickers in programming? Journal of Extension [Online], 48(5) Article 5TOT9. Available at: http://www.joe.org/joe/2010october/tt9.php

Cain, J., Black, E. P., & Rohr, J. (2009). An audience response system strategy to improve student motivation, attention, and feedback. American Journal of Pharmaceutical Education, 73(2), 1–7. doi:10.5688/aj730221

Early Childhood Family Education Coordinated Outreach (2013). Early childhood family education resources for parents and families. Retrieved from http://ecfe.info/ecfe-home/

Ham, Y. H., & Walker, S. (2014). Toward a measurement of parenting program effectiveness: A psychometric approach. National Council on Family Relations conference, Baltimore, MD. November, 2014.

Heaslip, G., Donovan, P., & Cullen, J. G. (2014). Student response systems and learner engagement in large classes. Active Learning in Higher Education, 15(1), 11– 24. doi:10.1177/1469787413514648

Karen Buddhist Dhamma Dhutta Foundation (2011). The Karen people: culture, faith and history. Retrieved from http://www.karen.org.au/docs/Karen_people_booklet.pdf

Mula, J. M., & Kavanagh, M. (2009). Click go the students, click-click-click: The efficacy of a student response system for engaging students to improve feedback and performance. e-Journal of Business Education and Scholarship of Teaching, 3(1), 1–17. Retrieved from http://www.ejbest.org/upload/Mula,_Joseph.pdf

Parmer, S., Parmer, G., and Struempler, M. (2012). Testing a new generation: Implementing clickers as an Extension data collection tool. Journal of Extension [Online], 50(5) Article 5TOT. Available at: http://www.joe.org/joe/2012october/tt5.php

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341–349. doi:10.1177/109821400002100305

Walker, S. (2014). Never doubt the power: Stakeholder impacts from a collaborative evaluation of a primary prevention community-based early childhood and education program. Poster sessions presented at Society for Research on Child Development special topics conference, Alexandria, VA.

Walker, S., Sysyn, D., Hennen-Clements, C., & Brown, M. (2012). Saint Paul school district early childhood family education program: An evaluation of benefits to parenting and children's learning and development from program participation. Retrieved from Saint Paul Public School website: https://ecfe.spps.org/uploads/st_paul_ecfe_report_updated_1-31_final_3.pdf