Difference between revisions of "Rummel Scripted Collaborative Problem Solving"
DejanaDiziol (talk  contribs) (→Annotated bibliography) 
(→Dependent variables) 

Line 84:  Line 84:  
Students took several post tests during the test phase that took place two days after instruction:  Students took several post tests during the test phase that took place two days after instruction:  
−  * ''  +  * ''[[Normal posttest]], assessment during instruction'': Student progress on training problems during the learning phase was analyzed. Their improvement in solving system of equations, i.e. their ability to generalize from one problem to an isomorphic problem with different content, can be assessed by looking at the learning curves. 
−  * ''  +  * ''[[Normal posttest]], assessment immediately following instruction'': Two posttests measured immediate learning (near transfer). As during the learning phase, students solved system of equations problems on the Cognitive Tutor. The problems were “isomorphic” to those in the instruction, but had different content. One test was solved collaboratively to measure both students’ algebra skills and their collaboration skills, the other test was solved individually to assess if the students were able to generalize the gained knowledge to isomorphic problems when working on their own. 
−  * ''  +  * ''[[Transfer]]'': This paper and pencil test assessed students' understanding of the main mathematical concepts from the learning phase: slope, yintercept and intersection point. The [[conceptual tasks]] students had to solve tapped the same knowledge components as the problems in instruction; however, the problems where nonisomorphic to those in the instruction, thus demanded students to flexibly apply their knowledge to problems with a new format. The test consisted of two problem sets with two different question formats: problem set 1 asked for the basic concepts yintercept and slope, problem set 2 asked for the novel system’s concept of intersection point. Both sets were composed of questions with a distinct answer format and open format questions that asked students to explain their problemsolving. 
* ''[[Accelerated future learning]]'': To measure accelerated future learning, students learned how to solve problems in a new area of algebra – inequality problems. This test was administered on the Cognitive Tutor, thus students could ask the Tutor for hints and received immediate error feedback. According to their condition during the experimental learning phase, they either worked individually or collaboratively on this post test. However, in contrast to the learning phase, they did not receive additional instructions to support their collaboration.  * ''[[Accelerated future learning]]'': To measure accelerated future learning, students learned how to solve problems in a new area of algebra – inequality problems. This test was administered on the Cognitive Tutor, thus students could ask the Tutor for hints and received immediate error feedback. According to their condition during the experimental learning phase, they either worked individually or collaboratively on this post test. However, in contrast to the learning phase, they did not receive additional instructions to support their collaboration.  
Revision as of 09:33, 18 April 2007
Contents
Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving
Nikol Rummel, Dejana Diziol, Bruce M. McLaren, and Hans Spada
Summary Tables
PIs  Bruce McLaren, Nikol Rummel & Hans Spada 
Other Contributers 

Pre Study
Study Start Date  03/13/06 
Study End Date  03/21/06 
LearnLab Site  CWCTC 
LearnLab Course  Algebra 
Number of Students  N = 30 
Average # of hours per participant  3 hrs. 
Full Study
Study Start Date  04/03/06 
Study End Date  04/07/06 
LearnLab Site  CWCTC 
LearnLab Course  Algebra 
Number of Students  N = 139 
Average # of hours per participant  3 hrs. 
Abstract
In this project, the Algebra I Cognitive Tutor is extended to a collaborative learning environment: students learn to solve system of equations problems while working in dyads. As research has shown, collaborative problem solving and learning has the potential to increase elaboration on the learning content. However, students are not always able to effectively meet the challenges of a collaborative setting. To ensure that students capitalize on collaborative problem solving with the Tutor, a collaboration script was developed that guides their interaction and prompts fruitful collaboration. During the scripted problem solving, students alternate between individual and collaborative phases. Furthermore, they reflect on the quality of their interaction during a recapitulation phase that follows each problem they solve. During this phase, students self evaluate their interaction and plan how to improve it, thus they engage in metacognitive activities.
To assess the effectiveness of the script, we conducted two classroom studies. In an initial, small scale study (pre study) that served to establish basic effects and to test the procedure in a classroom setting, we compared scripted collaboration with an unscripted collaboration condition in which students collaborated without support. In the full study, we furthermore compared these collaborative conditions to an individual condition to assess the effect of the collaborative Tutor extension to regular Tutor use. The experimental learning phase took place on two days of instruction. On the third day, during the test phase, students completed several post tests that assessed robust learning. Analyses of the collected measures are in progress. Results from one test of the pre study revealed significant differences between conditions. This test assessed students' transfer of main mathematical concepts of the learning content to tasks of a different format. These differences where particularly clear for the open format questions that asked students to explain their problemsolving. Further results will be presented soon.
Background and Significance
In our project, we combined two different instructional methods both of which have been shown to improve students’ learning in mathematics: Learning with intelligent tutoring systems (Koedinger, Anderson, Hadley, & Mark, 1997) and collaborative problem solving (Berg, 1993). The Cognitive Tutor Algebra that was used in our study is a tutor for mathematics instruction at the high school level. Students learn with the Tutor during part of the regular classroom sessions. The Tutor's main features are immediate error feedback, the possibility to ask for a hint when encountering impasses, and knowledge tracing, i.e. the Tutor creates and updates a model of the student’s knowledge and selects new problems tailored to the student’s knowledge level. Although several studies have proven its effectiveness, students do not always benefit from learning with the Tutor. First, because the Tutor places emphasis on learning procedural problem solving skills, yet a deep understanding of underlying mathematical concepts is not necessarily achieved (Anderson, Corbett, Koedinger, & Pelletier, 1995). Second, students do not always make good use of the learning opportunities provided by the Cognitive Tutor (e.g. help abuse, see Aleven, McLaren, Roll, & Koedinger, 2004; gaming the system, Baker, Corbett, & Koedinger, 2004). So far, the Cognitive Tutor has been used in an individual learning setting only. However, as research on collaborative learning has shown, collaboration can yield elaboration of learning content (Teasley, 1995), thus this could be a promising approach to reduce the Tutor’s shortcomings. On the other hand, students are not always able to effectively meet the challenges of a collaborative setting (Rummel & Spada, 2005). Collaboration scripts have proven effective in helping people meet the challenges encountered when learning or working collaboratively (Kollar, Fischer, & Hesse, in press). To ensure that students would profit from a collaboratively enhanced Tutor environment, we thus developed a collaboration script that prompts fruitful interaction on the Tutor while students learned to solve “systems of equations”, a content novel to the participating students. The script served two goals: First, to improve students’ robust learning of knowledge components relevant for the systems of equation units (i.e. both problem solving skills and understanding of the concepts). Second, students should increase their explicit knowledge of what constitutes a fruitful collaboration and improve their collaboration skills in subsequent interactions without further script support. This is particularly important due to the risk of overscripting collaboration that has been discussed in conjunction with scripting for longer periods of time (Rummel & Spada, 2005).
Glossary
See Scripted Collaborative Problem Solving Glossary
Research question
Does collaboration – and in particular scripted collaboration – improve students’ robust learning in the domain of algebra?
Does the script approach improve students’ collaboration, and does this result in more robust learning of the algebra content?
Independent variables
The independent variable was the mode of instruction. Three conditions were implemented in the learning phase that took place on two days of instruction:
 Individual condition: Students solved problems with the Algebra I Cognitive Tutor in the regular fashion (i.e. computer program as additional agent).
 Unscripted collaborative condition: Students solved problems on the Cognitive Tutor while working in dyads; thus, a second learning resource was added to the regular Tutor environment (i.e., both a peer and a computer program as additional agents).
 Scripted collaboration condition: As in the unscripted collaborative condition, students solved problems on the Tutor while working in dyads; however, their dyadic problem solving was guided by the Collaborative ProblemSolving Script (CPS) (i.e., both a peer and a computer program as additional agents). Three script components supported their interaction in order to reach a more fruitful collaboration and improved learning:
 A fixed script component structured students’ collaboration in an individual and a collaborative phase. During the individual phase, each student solved a single equation problem; during the collaborative phase, these problems were combined to a more complex system of equations problem. This division of expertise aimed at increasing the individual's responsibility for the common problemsolving process and thus at yielding a more fruitful interaction. In addition, the fixed script component structured the collaborative phase in several sub steps and prompted students to engage in elaborative interaction.
 An adaptive script component provided prompts to improve the collaborative learning process when the dyad encountered difficulties in their problem solving. E.g., to prevent help abuse in the tutoring system and to stop students from proceeding in the hint sequence until reaching the bottom out hint, the dyad is prompted to elaborate on the Tutor hints when reaching the penultimate hint of the hint sequence. This aims at enabling students to find the answer by themselves on basis of the help received and at students gaining knowledge to solve similar steps later on their own.
 A metacognitive script component prompted students to engage in reflection of the group process to improve their collaboration over subsequent interactions. Following each collaborative phase, students rated their interaction on 8 collaboration skills and set goals for the following collaboration. This phase was referred to as the "recapitulation phase"
In all conditions, the Cognitive Tutor supported students’ problem solving by flagging their errors and by providing ondemand hints.
Communication between the collaborative partners in the collaborative conditions was face to face: they joined together at a single computer to solve the problems. In the scripted collaboration condition, the script structured the allocation of subtasks between the two students.
In the pre study that aimed at establishing basic effects and at testing the procedure in a classroom setting, we only compared unscripted and scripted collaborative problem solving.
Hypothesis
The hypothesis can be stated in two parts: knowledge gains in the domain of algebra and improvement of the dyads’ collaboration skills:
 We expect the collaborative conditions – in particular the scripted collaboration condition – to outperform the individual conditions in measures of robust learning of the algebra content.
 We expect that problemsolving with the Collaborative ProblemSolving Script will lead to a more effective interaction than unscripted collaboration in the learning phase. In addition, we expect that the newly gained collaborative skills will sustain during subsequent collaborations in the test phase when script support is no longer available.
Dependent variables
Students took several post tests during the test phase that took place two days after instruction:
 Normal posttest, assessment during instruction: Student progress on training problems during the learning phase was analyzed. Their improvement in solving system of equations, i.e. their ability to generalize from one problem to an isomorphic problem with different content, can be assessed by looking at the learning curves.
 Normal posttest, assessment immediately following instruction: Two posttests measured immediate learning (near transfer). As during the learning phase, students solved system of equations problems on the Cognitive Tutor. The problems were “isomorphic” to those in the instruction, but had different content. One test was solved collaboratively to measure both students’ algebra skills and their collaboration skills, the other test was solved individually to assess if the students were able to generalize the gained knowledge to isomorphic problems when working on their own.
 Transfer: This paper and pencil test assessed students' understanding of the main mathematical concepts from the learning phase: slope, yintercept and intersection point. The conceptual tasks students had to solve tapped the same knowledge components as the problems in instruction; however, the problems where nonisomorphic to those in the instruction, thus demanded students to flexibly apply their knowledge to problems with a new format. The test consisted of two problem sets with two different question formats: problem set 1 asked for the basic concepts yintercept and slope, problem set 2 asked for the novel system’s concept of intersection point. Both sets were composed of questions with a distinct answer format and open format questions that asked students to explain their problemsolving.
 Accelerated future learning: To measure accelerated future learning, students learned how to solve problems in a new area of algebra – inequality problems. This test was administered on the Cognitive Tutor, thus students could ask the Tutor for hints and received immediate error feedback. According to their condition during the experimental learning phase, they either worked individually or collaboratively on this post test. However, in contrast to the learning phase, they did not receive additional instructions to support their collaboration.
Several variables from these post tests are compared to test the two hypotheses.
 For the post tests conducted on the Cognitive Tutor (retention and accelerated future learning), variables such as number of errors per problem, time per problem, decrease of error rates over the course of several problems etc are extracted from the log files to compare conditions.
For the paper and pencil post test (far transfer), we developed a scoring systems. For distinct format questions, students received 0 points for incorrect/missing answers and 1 point for correct answers; for open format questions , students received 0 points for incorrect, irrelevant or missing answers, 1 point for partly elaborated answers, and 2 points for fully elaborated answers. We summed points separately for the two question formats and the two problem sets of this posttest, resulting in four variables.  To compare collaboration skills of students, we recorded students’ dialogues both during learning phase and during test phase. Based on a rating scheme from one of our former projects (Meier, Spada, & Rummel, 2007), we develop a rating scheme to assess students’ interaction on skill dimensions such as “giving elaborated explanations to the partner”.
Furthermore, the paper and pencil post test entailed an open format question asking students to explain what is important for a fruitful collaboration. The answers to this question are also coded and compared between conditions
Method
The study procedure took place during 3 classroom periods over the course of a week. During day 1 and day 2 (the learning phase), the experimental conditions (i.e. the independent variables) were implemented. On day 3, students had to solve several posttests that assessed different aspects of robust learning.
Two in vivo classroom studies were conducted: a pre study with 3 classrooms, and a full study with 8 classrooms. The pre study aimed at establishing basic effects of the script and testing the procedure in a classroom setting, thus we only compared scripted to unscripted collaboration. In the full study, we compared the scripted collaboration condition to both an individual condition and an unscripted collaboration to evaluate the merits of both enhancing the Tutor with collaboration in general and scripted collaboration in particular. Due to the disruptiveness of students in the same class using different interventions, we used a betweenclass design.
Participants pre study:
The unscripted condition consisted of two classes (12 and 4 students), and the scripted condition consisted of one class (13 students). All classes were taught by the same teacher.
Participants full study:
 Individual condition: 1 class (21 students)
 Unscripted condition: 3 classes (18 + 16 + 16 dyads = 25 dyads)
 Scripted condition: 4 classes (16 + 16 + 18 + 18 dyads = 34 dyads)
Findings
Findings pre study:
First results of the paper and pencil test (far transfer) revealed substantial differences between conditions. The analysis was restricted to students who always worked collaboratively when present, as we are interested in the script’s effect on collaborative learning in particular. Due to student absenteeism, only 9 students in the unscripted and 10 students in the scripted condition were included in our analysis.
Significant differences were found both in the MANOVA (PillaiSpur, F(4, 14) = 7.35, p < .05) and in subsequent ANOVAs. Multiple choice answers on basic concepts did not show a significant difference, F(1,17) = 2.26, ns. However, the scripted condition outperformed the unscripted condition on the discrete answer questions about the system’s concept, F(1,17) = 22.16, p < .01. A significant difference between conditions was also found for the open format questions of both problem sets with F(1,17) = 5.85, p < .05 for the basic concepts and F(1,17) = 17.01, p < .01 for the system’s concept. Particularly the results of the open format questions demonstrate the script’s effect on conceptual knowledge acquisition: after scripted interaction during the learning phase, students were better at articulating their mathematical thinking compared to their unscripted counterparts. However, it should be noted that in general students in both conditions had difficulties providing explanations, thus only reached low scores in the open format questions. The amount of wrong explanations and the number of students who did not even try to articulate their thinking was very high.
Results on tests that were administered on the Cognitive Tutor (hypothesis 1: near transfer, immediate; near transfer, retention; acceleration of future learning) as well as results of the analysis of the collaboration process (hypothesis 2) are coming soon.
Findings full study:
Coming soon.
Explanation
This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. According to the IC cluster’s hypothesis, instruction that yields robust learning should be designed to have the right target paths. Furthermore, it should improve the students’ path choice in the learning event space, i.e., increasing the probability that students take correct paths and decreasing the probability that students take alternative paths.
By enhancing the Cognitive Tutor with collaboration, the study tried to reach these two goals in two steps:
 Individual condition: The existing Cognitive Tutor already offers correct learning paths. However, individual learning on the Cognitive Tutor has several shortcomings. For example, students don’t have the opportunity to reflect on the underlying mathematical concepts in natural language interaction in order to gain a deeper understanding. In addition, students do not always use the offered learning paths (as the opportunity to reflect on ondemand hints and error feedback) effectively, thus often do not engage in sense making processes
 Unscripted collaboration condition (1st step): By enhancing the Algebra I Cognitive Tutor to be a collaborative learning environment, we added a second learning resource – the learning partner. This adds further correct learning paths to the learning event space, for instance, learning by giving explanations, the possibility to request help, and learning by knowledge coconstruction. However, similarly to the learning paths in an individual setting, students do not always capitalize on these learning opportunities.
 Scripted collaboration condition (2nd step): To increase the probability that students take correct learning paths, students’ interaction in the scripted collaboration condition is guided by the Collaborative Problem Solving Script. By prompting collaboration skills that have shown to improve learning and by guiding students’ interaction when meeting impasses, we expect students to engage in a more fruitful collaboration that yields robust learning.
Annotated bibliography
 Rummel, N., Diziol, D., Spada, H., & McLaren, B. M. (accepted). Scripting Collaborative Problem Solving with the Cognitive Tutor Algebra: A Way to Promote Learning in Mathematics. Submitted to the European Association for Research on Learning and Instrution (EARLI07). Budapest, August 28  September 1, 2007.
 Rummel, N., Spada, H., Diziol, D. (accepted). Evaluating Collavborative Extensions to the Cognitive Tutor Algebra in an In Vivo Experiment. Lessons Learned. Submitted to the European Association for Research on Learning and Instrution (EARLI07). Budapest, August 28  September 1, 2007.
 Diziol, D., Rummel, N., Spada, H., & McLaren, B. M. (accepted). Promoting Learning in Mathematics: Script Support for Collaborative Problem Solving with the Cognitive Tutor Algebra. Submitted to the Conference on Computer Supported Collaborative Learning (CSCL07). Rutgers University, July 1621, 2007. [pdf]
 Rummel, N., Diziol, D., Spada, H., McLaren, B., Walker, E. & Koedinger, K. (2006) Flexible support for collaborative learning in the context of the Algebra I Cognitive Tutor. Workshop paper presented at the Seventh International Conference of the Learning Sciences (ICLS 2006). Bloomington, IN, USA.
 Diziol, D. (2006). Development of a collaboration script to improve students` algebra learning when solving problems with the Algebra I, Cognitive Tutor. Diploma Thesis. AlbertLudwigsUniversität Freiburg, Germany: Institute of Psychology, June 2006.
 McLaren, B., Rummel, N. and others (2005). Improving algebra learning and collaboration through collaborative extensions to the Algebra Cognitive Tutor. Poster presented at the Conference on Computer Supported Collaborative Learning (CSCL05), May 2005, Taipei, Taiwan.
References
 Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modelling to metacognitive skills. In J. C. Lester, R. M. Vicari & F. Paraguaçu (Eds.), Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004 (pp. 227239). Berlin: Springer.
 Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167207.
 Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. Paper presented at the Proceedings of the 7th International Conference on Intelligent Tutoring Systems.
 Berg, K. F. (1993). Structured cooperative learning and achievement in a high school mathematics class. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA.
 Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 3043.
 Kollar, I., Fischer, F., & Hesse, F. W. (in press). Collaboration scripts  a conceptual analysis. Educational Psychology Review.
 Meier, A., Spada, H. & Rummel, N. (2007). A rating scheme for assessing the quality of computersupported collaboration processes. International Journal of ComputerSupported Collaborative Learning. published online first at DOI http://dx.doi.org/10.1007/s114120069005x, February 2007
 Rummel, N., & Spada, H. (2005). Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in ComputerMediated Settings. Journal of the Learning Sciences, 14(2), 201241.
 Teasley, S. D. (1995). The role of talk in children's peer collaborations. Developmental Psychology, 31(2), 207220.
Connections
As this study, Walker A Peer Tutoring Addition adds scripted collaborative problem solving to the Cognitive Tutor Algebra. The studies differ in the way collaboration is integrated in the Tutor. First, in this study, both students first prepare one subtasks of a problem to mutually solve the complex story problem later on. Thus, although the students are experts for different parts of the problem, they have a comparable knowledge level during collaboration. In contrast, in Walker et al.'s study, one student prepares to teach his partner. Then, they change roles. Thus, during collaboration, their knowledge level differs. Second, in this study, collaboration was face to face, whereas the Walker et al. study used a chat tool for interaction.
Similar to the adaptive script component of the Collaborative ProblemSolving Script, the Help Tutor project aims at improving students' helpseeking behavior and at reducing students' tendency to game the system.
Furthermore, both studies contain instructions to teach metacognition. The metacognitive component in our study instructs students to monitor their interaction in order to improve it in subsequent collaborations; the Help Tutor project asks students to evaluate their need for help in order to improve their helpseeking behavior when learning on the Tutor.
Both in this study and in the Reflective Dialogue study from Katz, students are asked to engage in reflection following each problemsolving. In this study, the reflection concentrates on the collaborative skills, while in Katz' study, the reflection concentrates on students' domain knowledge of the main principles applied in the problem.
Furthermore, both our study and the Help Lite (Aleven, Roll) aim at improving conceptual knowledge.
Future plans
Our future plans for June 2007  December 2007:
 Analysis of data from Tutor logfiles
 Indepth analysis of collaboration during learning phase (i.e. collaborative learning process) and during test phase
 Conference contributions in symposiums at AERA and EARLI (robust learning and in vivo experimentation) and at CSCL and EARLI (scripted collaboration)
 Paper publication