Robots, GPS/GIS, and Programming Technologies: The Power of "Digital Manipulatives" in Youth Extension Experiences

Introduction

Educational robotics and other technologies, such as graphing calculators, computer based laboratory devices, and Global Positioning System devices, are being used to try to make the instruction of the science, engineering, and technology (SET) disciplines more relevant, interesting, and effective (Ball & Stacey, 2005; Chambers & Carbonaro, 2003; Heid, 2005; Reece et al., 2005; Sinclair & Crespo, 2006; Barker & Ansorge, 2006). Technologies like robotics and GPS devices that allow youth to build and experiment with physical objects are sometimes referred to as "digital manipulatives" (Resnick, 1998).

Unlike traditional manipulatives found in any elementary school classroom (blocks, LEGOs), digital manipulatives embed computation capabilities, often using a technology-based context. These computational capabilities permit youth to collect and interact with various forms of data. Fundamentally, the general use of manipulatives is based on constructivist learning theories where youth actively construct knowledge from experience (Piaget, 1972).

Digital manipulatives can essentially be seen as technology "catalysts" that help youth to undertake a more systematic approach to solving problems using the technology as a tool to aid in their thinking. In contrast to traditional technology applications, where youth learn more directly from the technology (e.g., drill and practice exercises, a Google search, or watching a YouTube video), digital manipulatives allow youth to learn with the technology and to more efficiently construct their understanding while actively engaged in the learning process. The study reported here investigated the effectiveness and the potential to scale-up a Nebraska 4-H SET program that employed digital manipulatives in the form of robotics, handheld global positioning system (GPS) devices, and global information systems (GIS) concepts.

The Nebraska 4-H Robotics and GPS/GIS program has used three digital technologies; the LEGO NXT Robot, GPS handheld receivers, and GIS mapping. The purpose of the program is to prepare youth for the 21st century workplace by providing them opportunities to learn SET concepts and examine associated SET careers and to foster positive attitudes about SET concepts and opportunities. The Nebraska 4-H Robotics and GPS/GIS project team seeks to understand how digital manipulatives can increase SET content knowledge.

The Nebraska 4-H Robotics and GPS/GIS Program

Beginning in 2006, Nebraska 4-H received a grant from the National Science Foundation's (NSF) Information Technology Experiences for Students and Teachers (ITEST) competition to develop a comprehensive youth outreach program utilizing robotics and GPS/GIS technologies. The result of the funding was the Robotics and GPS/GIS in 4-H: Workforce Skills for the 21st Century project. Due to the initial success of the project, in 2008 it received further NSF ITEST grant funding as a Scale Up project, with the goal of expanding the program across the nation in the next 5 years.

Currently, youth ages 10-15 participate in the statewide program for 2 years, beginning with an intensive 40-hour summer camp. They then receive 80-hours of hands-on instruction during the school year in their afterschool programs or 4-H clubs. In year two, youth attend an advanced summer camp and receive 80 additional hours of hands-on instruction during the school year. In total, youth receive at least 240 hours of focused hands-on instruction over 2 years. The current program serves approximately 150 youth in Nebraska and surrounding states. The future goal of the program is to reach more than 4,800 students nationwide.

Reviewing the Research

As a funded NSF-related endeavor, the project has been careful to establish its activities on a sound research foundation. In review of the literature, only a few quantitative studies were found that directly involved the use of robotics to teach SET topics. Nourbakhsh, Hamner, Crowley, and Wilkinson, (2004) reported that a 7-week robotics course for high-school students led to significant increases in learning based on self-evaluations. However, possible gains reported in the study relied upon student self-report, and the results might not represent actual learning, because students may have not correctly estimated their performance during the intervention.

Fagin and Merkle (2003) examined the effect of robotics on teaching an introductory computer science course at the college level. The researchers compared computer programming laboratory midterm and final exam scores between groups that used the robotics and groups that did not. Their results indicated that laboratory sections using robots had significantly lower scores in the course than students who had not been taught with robots. However, the robotics sections were only allowed to practice problems in the laboratory, while students in the control sections had no such restriction. Given that the participant groups were treated differently, the results of the study may not accurately reflect the impact robotics can have on learning. In addition, it is unclear how well the robotics experiences aligned with the content of the midterm and final exam instruments.

In 2005, Barker and Ansorge (2006, 2007) examined the effectiveness of using the previous generation of LEGO Robotics, called the "RCX," within a nonformal education setting. The first study examined SET learning in an afterschool program in rural Nebraska using a pre- and post-content exam. In addition, for comparison purposes, the study used a control group that was in an afterschool program but not involved with robotics. The results of this early study showed a significant increase from the pretest (M=7.93, SD=3.71) to the posttest (M=17.00, SD.88) t(14) = -8.95, P <.001. The control group did not show an overall increase from pretest to posttest. Moreover, the content instrument, which involved multiple-choice SET-related questions, was shown to be reliable with a Cronbach's alpha reliability coefficient of 0.86 (Barker & Ansorge, 2007).

Encouraged by these initial research results, a follow-up study was then conducted in 2006 with 121 youth in nine different schools running LEGO RCX robotics programs compared with 36 youth from three separate schools acting as the control group (Barker, Nugent, Grandgenett, & Hampton, 2007). Again the results of the study indicated a significant increase in scores on SET concepts from pretest (M=9.49. SD= 3.66) to posttest (M=11.07, SD=3.82) t (119) = -5.06, P < .001. In contrast, the control group did not display a significant increase from pre to posttest scores.

A third study was then conducted in 2007, with two pilot camps that were held in Nebraska using the LEGO NXT robotic kits and GPS/GIS technologies. Unlike the previous two studies, this third study was conducted during a 4-H camping experience, with the additional integration of handheld global positioning system (GPS) devices and global information systems (GIS) concepts. The testing instrument was modified to include additional questions related to GIS and GPS concepts, as well as mathematics, to address a more general STEM understanding (science, technology, engineering, and mathematics). The results of the study were consistent with previous studies, where a significant increase from pre (M=12.63. SD= 3.67) to posttest (M=16.50, SD=4.09) t(38) = -7.380, P < .001 (Table 1). Together, the results of these studies are supportive that digital manipulatives (such as robotics and GPS/GIS technologies), when used in afterschool programs and in summer camp settings, have the potential to promote solid learning of SET concepts.

Table 1.
Past 4-H Robotic Study Results

Study	Environment	N	Pretest Mean	Posttest Mean	t-score	p	Effect Size (r)*
1. Barker and Ansorge (2007)	Afterschool	14	7.93	17.00	-8.95	<.001	.94
2. Barker, Nugent, Grandgenett, Hampton (2007)	Afterschool	121	9.48	11.09	-5.06	<.001	.58
3. Barker, Nugent, Grandgenett (2008)	Camp	38	12.63	16.50	-7.38	<.001	.66
* Using Pearson Correlation

Purpose and Methodology

Moving to Scale with Digital Manipulatives

The encouraging results of the first three studies, where robotics and GPS/GIS technologies functioned as digital manipulatives within the context of non-formal educational settings, encouraged us to scale the study into more diverse camp settings. The purpose of this larger fourth study was to determine the effectiveness of the 4-H Robotics and GPS/GIS intervention to increase youths understanding of SET concepts on a larger scale. Thus, the study looked at the feasibility of scaling-up the project from 38 participants in two camps during the summer of 2007 to 147 participants in six camps during the summer of 2008.

The instrument used for the study was an enhanced version of the earlier content instrument, representing a 37-item, paper-and-pencil, multiple-choice assessment, covering topics in computer programming, mathematics, geospatial concepts, engineering, and robotics. The same assessment instrument was used as the pre- and post-test. The pretest was administered on the first day of the camp prior to introductory activities, and the posttest was administered on the last day of camp. A Cronbach's alpha reliability coefficient of .80 was reported for the administration of the posttest.

Student Participants

A total of 147 students in six different 4-H facilitated camps participated in the summer program. Overall, 112 males and 35 females attended the camps. In addition, 75% of participants were identified as Caucasian, 12% were African American, 12% were Hispanic, and 1% Asian. The overall mean age for the camps was 12.28 years with a median age of 12.00 years (Table 2).

Table 2.
Camp Participant Demographics

Nebraska Location	N (Total)	Male	Female	Age (Mean)	% Minority	Overnight
1. Omaha North	18	9	9	11.39	100	No
2. Omaha South	16	9	7	11.12	56	No
3. Lincoln	67	55	12	12.52	12	Yes
4. Ord	10	9	1	12.40	0	No
5. Chadron	16	13	3	12.69	0	Yes
6. Grand Island	20	17	3	12.80	5	No

Instructional Facilitators

The instructional activities for the 2007 pilot camps were facilitated by two faculty members and two staff members from the University. During the 2008 camps, two faculty and five staff members facilitated the camp activities. All faculty members and staff were moderately experienced in the use of the robotics, GPS, and GIS technologies, but none of these facilitators would be considered to be "experts" in these technologies.

Results

Overall, there was a significant increase from the pretest (M = 15.63, SD = 4.52) to the posttest scores (M = 20.12, SD = 5.60, t(136)=-13.71), p < .001 for the six combined groups. However, not all sites had a significant increase in scores. The North Omaha site experienced a pretest score lower than other sites (M = 10.80, SD = 2.93) and had a decrease in the posttest score (M = 10.53, SD = 3.20) t(14) = .32, p < .NS. Results by camp are shown in Table 3 and Figure 1.

Table 3.
Content Questionnaire Paired Samples Test

Location	Paired Differences
	Pre Mean	Post Mean	Mean Difference	Std. Deviation	t	df	Sig. (2-tailed)
Omaha North	10.80	10.53	.267	3.22	.32	14	.753
Omaha South	11.58	15.50	-3.92	3.55	-3.82	11	.003
Lincoln	16.87	20.87	-4.00	3.01	-10.90	66	.001
Ord	17.60	23.80	-6.20	4.21	-4.66	9	.001
Chadron	16.53	23.20	-6.67	2.90	-8.93	14	.001
Grand Island	15.89	23.78	-7.89	3.18	-10.53	17	.001

Figure 1.
Pretest and Posttest Mean Scores by Location

A one-way ANOVA was used to test for posttest score differences among the six camp locations. Due to a violation of the necessary homogeneity of variance condition, the Brown-Forsythe statistic is reported. Posttest scores differed significantly across the six camp locations, F (5, 59.56) = 32.26, p < .001. A Dunnett C post-hoc comparison of the six locations indicated that the Omaha North (M = 10.53) and the Omaha South Camp (M=15.50) were significantly different from the other locations. Comparisons between the other four locations were not statistically significant at p < .05.

Summary and Recommendations

Results from the study reported here indicate that the digital manipulatives used by the 4-H Robotics and GPS/GIS program can indeed increase STEM content knowledge as represented by the pre- to post-test scores. Moreover, the project was successfully scaled-up from two camps to six camps reaching 147 participants. When compared to previous camps and afterschool programs, the 2008 camps showed similar positive results on student learning. Evidence from the four studies collectively now support the continued scaling of participant numbers (Table 4).

Table 4.
Results of Nebraska 4-H Robotics Programs

Environment	N	Pretest Mean	Posttest Mean	t-score	P	Effect size (r)*
Afterschool (1 school)	14	7.93	17.00	-8.95	<.001	.94
Afterschool (9 schools)	121	9.48	11.09	-5.06	<.001	.58
Summer Camps 2007 (2 camps)	38	12.63	16.50	-7.38	<.001	.66
Summer Camps 2008 (6 camps)	147	15.63	20.12	-13.71	<.001	.73
* Pearson Correlation

The results of the fourth study do, however, suggest a potential problem in the urban camps. In particular, the Omaha North camp did not have an increase in scores from pre- to post-test. Of particular concern is that the North Omaha camp involved the greatest percentage of minority students of any location, with 100% minority participation. The Omaha South location (with a significant growth, but experiencing less growth then the other four remaining settings) had the second highest minority participation at 56%. Post-hoc comparison results indicate that the Omaha North and South camps scores were significantly lower than the other camps.

One plausible explanation for the lower posttest scores of these two groups is the lack of prior knowledge of the student attending the camps. These camps were held in two of the most impoverished areas of the city, with a 52% free and reduced lunch rate. Results indicate that prior knowledge, as represented by the pretest score, accounted for 53% (R²) of the variability of the posttest scores for the overall group representing all six locations, using a regression analysis strategy where the predicted posttest score is equal to the slope (.907) multiplied by the pretest score (10.8) plus the Beta coefficient score of 5.94. Using this equation, the expected posttest score of the Omaha North group should have been 15.73, which is still lower than the other sites, but would have represented some growth. Using the same formula for the Omaha South group. with a pretest score of 11.58, the predicted posttest score is 16.44 compared to the observed score of 15.50.

Even with lower pretest scores we might anticipate an overall increase from pre- to post-test scores. One additional explanation is perhaps the influence of the timing of the posttest. The posttest was giving midday at the first two camps and in later in the afternoon for subsequent camps. Youth in the first two camps may have potentially rushed through the questions to return to camp activities, thereby increasing the errors on the posttest and lowering the resultant scores.

While the results are encouraging overall, more research is needed. In particular, further research should be undertaken in diverse settings involving significant percentages of minority students. To extend the program beyond a regional scope, additional staff and volunteers will no doubt be needed to facilitate the experiences for youth in their local community settings. Currently, trained staff and faculty to facilitate the camps have been available and have demonstrated that they can successfully and logistically scale-up their efforts across various camp and afterschool settings.

However it is also clear that a more systematic professional development program is now needed to train others to facilitate camp experiences within a wider geographic area and across more diverse settings involving both rural and urban youth. Moreover, the current program model with five staff and two faculty members to deliver the camps uses a considerable amount of instructional resources drawn from core program coordination. The professional development program will permit the migration of control from the core project staff at the state level to more distributed county level staff and volunteers, thereby reducing the centralized resources needed to run robotics and GPS/GIS camps. A future area of study needed within the program is an examination of the return on investment to determine the net benefits compared to the costs of the program.

It also seems that further research would benefit from a more direct focus on the roles that the digital manipulatives (Robotics and GPS/GIS) are playing in the student learning experiences and how student thinking in this context is being influenced. Researchers may wish to use assessment strategies such as student interviews, think-out-loud protocols, or journaling to try to better understand how students are interacting with these tools. The tools themselves may also be a significant variable. In other words, could similar results be achieved with different digital manipulatives? Control groups may provide some of the needed evidence, but it would be interesting to run simultaneous camps, with randomly assigned students, that might use alternative digital manipulatives (pursuing the same objectives) to better understand the relative roles and effects of the digital manipulatives in student learning.

In entering our fourth year of such Extension programs, we are also beginning to consider the long-term impacts. How long will the effects of such interventions last? Do students retain the information and continue to use the problem solving strategies they develop? Do they have positive remembrances of such camp experiences and include them in their decision making process as they look toward taking upper level courses, and in even considering STEM careers? We hope so, but as yet, we have not examined such long-term impacts. Follow-up assessments and interviews may provide important clues. Eventually, it will be important to know the actual academic choices that students have made, whether students reflected upon their non-formal learning experiences when they made these formal academic choices, and ultimately if they choose SET careers.

Although many research questions remain, it is clear to us that the digital manipulatives of educational robotics, GIS, and GPS devices show significant promise as student thinking tools for nonformal educational settings where STEM-related goals are undertaken. Extension in Nebraska and the rest of the United States has a proud and rich history of serving our youth, and digital manipulatives may well play an important role as we strive to continue that proud tradition into the future.

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant ESI-0624591 and DRL-0833403.

References

Ball, L., & Stacey, K. (2005). Teaching strategies for developing judicious technology use. In Masalski, W.J. and Elliott, P.C. (Eds.) The sixty-seventh yearbook of the National Council of Teachers of Mathematics. Reston, VA: National Council of Teachers of Mathematics.

Barker, B., & Ansorge, J, (2006). Using robotics as an educational tool in 4-H. Journal of Extension [On-line], 45(5) Article 5IAW6. Available at: http://www.joe.org/joe/2006october/iw6.php

Barker, B., & Ansorge, J. (2007). Robotics as means to increase achievement scores in an informal learning environment. Journal of Research on Technology Education 39(3), 229-243.

Barker, B., Nugent, G., & Grandgenett, N.(2008). Examining 4-H robotics and geospatial technologies in the learning of science, technology, engineering, and mathematics topics. Journal of Extension [On-line] 46(3). Available at: http://www.joe.org/joe/2008june/rb7.php

Barker, B., Nugent, G., Grandgenett, N., & Hampton, A. (2008). Examining 4-H Robotics in the learning of science, engineering and technology topics and the related student attitudes. Journal of Youth Development. Volume 2(3). Retrieved July 20, 2009, from: http://www.nae4ha.org/directory/jyd/jyd_article.aspx?id=f5a34e58-1cd3-4994-981d-b81fa406cd74

Chambers, J., & Carbonaro, M. (2003). Designing, developing, and implementing a course on LEGO robotics for technology teacher education. Journal of Technology and Teacher Education, 11(2), 209-241.

Fagin, B., & Merkle, L. (2003) Measuring the effectiveness of robots in teaching computer science. In proceedings of the 34th SIGCSE technical symposium on computer science education, February 19-23, 2003.

Heid, M. K. (2005). Technology in mathematics education: Tapping into visions of the future. In Masalski, W.J. and Elliott, P.C. (Eds.) The sixty-seventh yearbook of the National Council of Teachers of Mathematics. Reston, VA: National Council of Teachers of Mathematics.

Nourbakhsh, I., Hamner, E., Crowley, K., & Wilkinson, K. (2004). Formal measures of learning in a secondary school mobile robotics course. Paper presented at the ICRA, New Orleans, LA.

Piaget, J. (1972). The psychology of the child. New York: Basic Books.

Reece, G. C., Dick, J., Dildine, J. P., Smith, K., Storaasli, M., Travers, K. J., et al. (2005). Engaging students in authentic mathematics activities through calculators and small robots. In Masalski, W.J. and Elliott, P.C. (Eds.) The sixty-seventh yearbook of the National Council of Teachers of Mathematics. Reston, VA: National Council of Teachers of Mathematics.

Resnick, M. (1998). Technologies for lifelong kindergarten. Educational Technology Research and Development, 46(4), 43-55.

Sinclair, N., & Crespo, S. (2006). Learning mathematics in dynamic computer environments. Teaching Children Mathematics, 12(9), 437-444.