Chapter 14. Scaling up ambitious learning practices
This chapter reports on findings from an effort to train teachers to adopt classroom facilitation practices referred to as Accountable Talk (AT) while jointly training students to engage in articulation of reasoning and transactive exchange in the midst of teacher-led classroom discussions as well as in computer-supported collaborative learning (CSCL) activities with one another in small groups. A key enabler in this work was AI-enabled scaffolding for collaborative discussions in the form of intelligent conversational computer agents that acted as discussion facilitators using the same AT practices that were the target of the teacher professional development. A key finding was that support for student engagement in transactive exchange prior to teacher-led discussion facilitated teacher uptake of AT practices.
Introduction
A danger of the growing emphasis on scale in education is that ambitious learning practices, such as many forms of discussion-based and group learning, may be eliminated in favour of efficiency-based individual learning practices. An extreme emphasis on efficiency that might crowd out opportunities to develop communication and collaboration skills threatens the ability of K-12 school systems to meet the demands of the 21st century working world. This chapter argues from evidence both that ambitious learning practices involving collaborative and discussion-based learning are feasible and beneficial to students in K-12 education even in challenging urban environments, and that the elimination of ambitious learning practices is not necessary for achieving effective learning at scale.
In particular, in this chapter we espouse a concept of discussion-based learning that integrates a set of classroom facilitation practices referred to as Accountable Talk (AT) (Michaels, O’Connor and Resnick, 2008[1]; Resnick, Michaels and O’Connor, 2010[2]; Resnick, Asterhan and Clarke, 2015[3]) paired with a key property of collaborative discourse practices referred to as Transactivity (Azmitia and Montgomery, 1993[4]; Berkowitz and Gibbs, 1983[5]; De Lisi and Golbeck, 1999[6]; Gweon et al., 2013[7]; Teasley, 1997[8]). At the core of both of these practices is the aim to keep student reasoning at centre stage. Though these frameworks grew up in separate research communities, specifically AT in the classroom discourse community and Transactivity in the collaborative learning community, they dovetail perfectly in that the key facilitation moves of AT appear to be designed to elicit transactive contributions from students (Adamson et al., 2014[9]). Specifically, a transactive contribution is one that makes reasoning explicit and connects it with an articulation of reasoning from earlier in the discussion, which may be the student’s own expressed reasoning, but more often is the reasoning of another student.
Demonstrations of the positive impact of AT and similar practices have been in evidence worldwide (Resnick, Asterhan and Clarke, 2015[3]). In particular, we know that in large-scale evaluations of AT, what has been reported are steep changes in student achievement (Bill et al., 1992[10]; Chapin and O’Connor, 2004[11]) retention for up to 3 years (Adey and Shayer, 1993[12]; Topping and Trickey, 2007[13]) transfer across domains for up to 3 years (Bill et al., 1992[10]; Adey and Shayer, 1993[12]; Chapin and O’Connor, 2004[11]). There are reports of students performing better on non-verbal reasoning tests e.g. Ravens (Wegerif, Mercer and Dawes, 1999[14]) while reasoning itself also improves (Kuhn et al., 2013[15]). Similarly, we know that Transactivity is a property of discourse where students are working on reasoning together, where students make their reasoning explicit and integrate or connect it with the expressed reasoning of other students. The finding is that the concentration of Transactivity correlates with learning gains in many studies of collaborative learning (Azmitia and Montgomery, 1993[4]).
Nevertheless, teachers may feel daunted by the demands of facilitating collaborative learning groups in the classroom or assessing student writing or the products of collaborative groups, automated support both for facilitation and assessment have recently been achieved (Rosé and Ferschke, 2016[16]). The purpose of this chapter is to illustrate how these technological advances can achieve positive impact in the classroom in a challenging, urban school environment through a large-scale professional development effort. Success in this environment serves as a proof-of-concept that ambitious learning practices can feasibly be implemented with typical teachers in challenging urban settings with positive impact in K-12.
In the remainder of the chapter, we first offer a historical overview of the work. We then detail the theoretical underpinnings of the work and describe first an investigation we conducted in the context of a school-district-wide professional development effort. Next, we describe how similar principles were used to motivate the design of a novel protocol for team formation in team-based Massive Open Online Courses (MOOCs), which demonstrates the generality of applicability of the theoretical framing as well as its potential in facilitating scale without sacrificing the employment of ambitious learning practices like team-based project learning (MOOCs). We conclude with implications for policy and practice.
Historical perspective
Our work builds on an international compendium of insight related to AT and similar teaching practices (Resnick, Asterhan and Clarke, 2015[3]). Despite growing evidence that AT produces measurable advances in student learning, most teachers – especially those teaching students from low socio-economic backgrounds and students of colour – do not employ these strategies. Several studies have documented that it is rare to find this kind of instruction in “high need” learning environments, and that teachers struggle in shifting the ways in which they use talk in classroom learning. Much of the evidence of success of these approaches come from research that has been conducted in elite schools, with master teachers.
In contrast, our project, in collaboration with the Learning Research and Development Center (LRDC)’s Institute for Learning (IFL), aimed to extend the success of AT to an urban school district with more typical teachers. The approach integrated the expert coaches of the IFL with technology for dynamic support of collaborative learning developed at Carnegie Mellon University (CMU) (Adamson et al., 2014[9]; Kumar et al., 2007[17]; Kumar and Rosé, 2011[18]).
The goal of our work was to develop protocols for achieving scale without sacrificing ambitious learning practices. We did this by building on the foundation of a key insight that we consider the DNA of well-functioning discussion-based and collaborative learning. That DNA is the phenomenon of students positioned as active reasoners who take responsibility for their reasoning, understanding and learning together (Howley, Mayfield and Rosé, 2013[19]; Sionti et al., 2011[20]). We began an investigation into this approach in the midst of a professional development effort that positioned both the teacher and the students as having agency to bring about change, each with a role to play in order to bring success (Clarke et al., 2016[21]). Note that students play a key, active role in this configuration, and technology support for collaboration acts as a catalyst, aiding in their effective role-taking within the configuration. In particular, the professional development effort involved first, a district-wide effort where human coaches worked directly with teachers and second, a computer-supported collaborative learning (CSCL) intervention in classrooms where technology support enables reaching out to an arbitrary number of small groups simultaneously such that students are better prepared to take their important role within the changing classroom culture (Adamson et al., 2014[9]; Clarke et al., 2003[22]; Dyke et al., 2013[23]). The implication is that with appropriate technology support, the push towards scale need not push out desirable, ambitious learning practices. In particular, the teacher professional development interventions focused on developing teachers’ capacity to promote active engagement between students using AT. In addition, we deployed artificially intelligent interventions, termed AT agents, in online collaborative learning experiences to support students’ capacity of engaging in this kind of learning dialogue.
During a five-year period of time, this district-wide collaborative effort aimed to impart AT discussion facilitation practices focused on ninth grade biology classrooms. The study was structured at two levels. First, there was a macro-study involving the professional development programme in which teachers across the school district were engaged in intensive workshops on AT facilitation practices. A subset of the teachers within the school district, who opted to participate in the research study, were then audio-recorded leading whole-class discussions in which they demonstrated the extent to which they had appropriated AT practices in their teaching in each unit of ninth grade biology. Observations occurred at the beginning and end of each unit. In the midst of this longitudinal professional development programme, we also introduced in vivo micro-studies, in other words, controlled experiments run within classrooms, where we evaluated interventions for supporting small group discussion activities meant to prepare the students for the whole group discussions. The interventions in the small group were aimed to intensify transactive exchange between students using conversational computer agents employing automated AT facilitation practices (Adamson et al., 2014[9]). By preparing the students for active engagement in the discussions, the hope was to better enable the teachers to put their training into practice in the classroom, and the evidence confirmed this effect (Clarke et al., 2003[22]).
We evaluated the success of the professional development programme using a design-based research approach employing a synergy of qualitative and quantitative approaches. At the same time, we evaluated the effects of the in vivo interventions as short-term studies quantitatively while also observing the effect of the interventions within longitudinal analyses on the nature of whole group teacher-led discussions in the classrooms that housed the in vivo studies using hierarchical time series models.
Overall, we observed positive growth over time in teacher uptake of AT practices, with significant intensifications immediately following interventions involving small group automated coaching of AT practices. The significant intensifications surrounded three specific studies of automated AT facilitations. Using LightSIDE (Mayfield and Rosé, 2013[24]) we are able to code transcripts of whole-class discussions and small group discussions in order to observe where students and teachers were engaging in important facilitation behaviours. Multilevel growth models offer a path towards accomplishing this measurement. Using growth-modelling techniques, we were able to measure growth over time both at the teacher level and at the student level in terms of frequency of appropriation during class discussions. The teacher and student trajectories, that are constructed using latent variable techniques within this framework, allow us to control for dependencies between successive opportunities, as well as the complexities of the contextual influence of groups and classrooms. With this as a tool, we were able to systematically study the factors that affect teacher learning of discussion facilitation behaviours. An analysis of discussion transcripts from the first two years of the professional development programme demonstrates that there is a significant local effect on teacher uptake of facilitation behaviours from the training resulting from students participating in online collaboration activities prior to the teacher-led discussion, with an effect size of 1.7 standard deviations (Clarke et al., 2003[22]).
Technology support for collaboration
Our research in CSCL demonstrates that students benefit from interactions in learning groups when automated support is provided, especially interactive, context-sensitive support administered by intelligent conversational agents (Adamson et al., 2014[9]; Kumar et al., 2007[17]). This dynamic form of collaboration support “listens” to student conversations in search of important events that present opportunities for discouraging dysfunctional behaviour or encouraging positive behaviour using automated analysis of collaborative learning processes.
Note: Screen shot of collaborative interface. On the right-hand side is the chat panel where students interact with one another, supported by the computer facilitator whose turns are labelled Sage (Tutor).
Enabled by technologies including Bazaar (Adamson et al., 2014[9]), TuTalk (Jordan et al., 2007[25]; Rosé et al., 2001[26]; Mayfield and Rosé, 2013[24]), and LightSIDE (Mayfield and Rosé, 2013[24]), we have built interventions in which conversational computer agents employ AT practices, like re-voicing moves and agree-disagree moves, in order to test causal connections between AT moves, self-efficacy and learning in lab studies using automated analysis and conversational agent technology both in ninth grade biology and in college level Chemistry (Adamson et al., 2014[9]).
Lab experiments, such as the cell model lab illustrated in Figure 14.1, are a valued component of most secondary school science instruction. But in practice, classroom experiments and demonstrations are difficult to manage and may produce less academic content learning than educators hope for. It is hard to insure reliability of experiment outcomes, given variability in materials, measuring instruments and the like. Furthermore, when teachers ask students to discuss the outcomes of their class experiments (a frequent suggestion in the biology curriculum in the urban district in which we did our work, textbook and many other widely used textbooks), few students have the tools necessary to conduct these discussions. They are unfamiliar with techniques for summarising and interpreting data, and they do not know what their group discussions should attend to or how to build upon or challenge each other’s interpretations. As a result, while classroom experiments are engaging for students, it is often the case that little substantive learning takes place. Under pressure to produce measurable learning results within limited time spans, teachers often tell students what they “should” be seeing in their experiments, rather than building students’ capacities to design and interpret experiments themselves. Our goal has been to give students more time to reflect on what is happening in lab exercises, supported by conversational agents triggered in a context-sensitive way using automatic analysis of the collaborative discussion behaviour and to prepare them to take these insights back to the classroom for further whole-class interaction there, and in that way, the engagement in small groups enhances the efforts of the teachers to foster dynamic group discussions with student reasoning at centre stage.
The interventions we designed acted as scaffolding for small group discussion of classroom labs. We tested the effectiveness of these techniques in increasing students’ ability to interpret experimental results and improving learning of the core biology concepts that are involved. In collaboration with the teachers participating in our study, we selected classroom experiments from the curriculum. We designed structured activities in which students specified hypotheses, recorded data and interpreted their data. Students worked through the activities in collaborative groups. Pre- and Post-tests on the biology content of the experiments and students’ skill in interpretation of data were administered and all worksheets collected, permitting comparison of learning in the collaborative and the individual condition.
Our classroom evaluations of AT conversational agents as support for collaborative learning demonstrate their significant positive effect on student learning. First, in a January 2011 in vivo study, students in the experimental condition where they did a small group activity with the support of a conversational agent, were significantly more active in the immediately following whole group discussion than the students in the control condition, who did not have the support of a conversational agent during their small group activity. In the January 2012 in vivo study, students in the re-voicing agent condition learned significantly more on the immediate post-test than students in the control condition that did not have the support of the AT agents during collaboration. In the March 2012 in vivo study, students in the condition with re-voicing agents learned marginally more from the whole group discussion that immediately followed the intervention than students in the control condition who did not have the support of re-voicing agents. The series of studies were discussed in a review of early studies of AT agent support for collaborative learning (Adamson et al., 2014[9]).
Into the future: Fostering collaboration on Massive Open Online Courses (MOOCs)
Above, we have argued that Transactivity is a property of effective collaborative discourse that can be thought of as the DNA of a well-functioning group. On the one hand, Transactivity is frequently thought of in connection with its cognitive underpinnings, in that it signifies that students are openly sharing their reasoning and integrating their reasoning. This behaviour creates opportunities for students to experience cognitive conflict, which explains the correlation we see between the prevalence of transactive exchanges and learning. But, Transactivity also has social underpinnings (Gweon et al., 2013[7]). It signifies the experience of power balance, intimacy and a desire to build common ground. In recent work, we have leveraged the social underpinnings of Transactivity, using a measure of the exchange of transactive contributions in one setting as an indicator that a pair of students would collaborate well in a different setting. We developed a paradigm for team formation based on this idea, validated it in a controlled experiment in a lab setting and then deployed it in a successful team-based MOOC. Here we report briefly on this work, which is more fully described in separate publications (Wen et al., 2017[27]; Wen et al., 2018[28]).
In typical team-based MOOCs, team formation occurs immediately upon starting a course, and usually it is done through self-selection. Using this protocol for team formation, many of the teams fail. In our enhanced protocol, students first complete some individual work upon starting a course. During this time, many students who are not committed to the course quit. Those who are committed enough to complete the work are then required to post the work to the public discussion forum, and students are then asked to give feedback to a few students publicly within the forum. An automated analysis of these exchanges is then performed using a Transactivity detection model. Then, for each pair of students, we count the number of transactive contributions that were exchanged. After that, an approximate constraint satisfaction algorithm is used to assign students to groups of four in such a way that students are more likely to end up in the same team with others they have had transactive exchanges with during the public feedback discussion. The automatically assigned teams then do their teamwork together.
In order to validate the team formation protocol, we tested it in the Amazon Mechanical Turk environment. We ran the protocol several times, each time either using the full team formation protocol or using the same paradigm but with random assignment of teams rather than basing the assignment on the measure of Transactivity as in the full protocol. We then compared the extent of knowledge integration in the group proposal constructed during the collaborative phase. In this experiment, we found a significant advantage for the Transactivity based matching over random assignment, with an effect size of three standard deviations.
Finally, partnering with the Smithsonian Institute, we developed a three-week team-based MOOC in which we adapted the team formation protocol. Though there was no experimental manipulation, we measured the correlation between observed Transactivity during the feedback stage of the groups that were formed and their ultimate task success and found positive correlations that were consistent with our expectations based on the experimental study.
Conclusions and current directions
In this chapter, we have briefly described first, a theoretical construct referred to as Transactivity, which operationalises a key quality of well-functioning collaborative discourse, and which has both cognitive and social underpinnings. We have identified AT practices, as well as other similar discussion facilitation techniques (Resnick, Asterhan and Clarke, 2015[3]), which can be viewed as tools that stimulate higher concentrations of transactive exchange in student discussions. We first recounted a district-wide professional development effort that offered evidence that enhancing transactive exchange through automated AT facilitation in small groups leads to enhanced learning, and that experience of these collaborative learning encounters also facilitates greater uptake of AT facilitation practices among teachers. In a second investigation, we leveraged the same concept of Transactivity, but this time we used it as an indicator used in automated team formation. The new paradigm was validated in a crowdsourcing environment and then tested in a real MOOC deployment, which was successful.
Policy Implications
The implications of this work for policy synergise with those associated with Intelligent Tutoring technology discussed separately in this volume (Chapter 13, by Koedinger). Specifically, scaffolding for learning enabled by technology and designed to support the cognitive processes underlying learning and problem solving holds great promise for enhancing K-12 education at scale. In this chapter, specifically, we focus on learning through argumentation, developing learners’ identities as reasoners and problem solvers (Chapter 3, by Meltzoff and Cvencek). We specifically address learning through social interaction, which engages learners not just as cognitive systems, but also as social beings who must develop the ability to function in a 21st Century world, which demands the ability to communicate and work in teams (Chapter 19, by Law and Ming Cheng). What we uniquely offer is a demonstration of specific technology that can act as a catalyst in facilitating widespread use of ambitious learning practices such as team-based project learning and collaborative learning more broadly to achieve these ends. While a push for efficiency and scale has sometimes resulted in minimising inclusion of classroom discussion, collaborative learning and engagement in writing with peer review both in teaching and in assessment, we offer evidence that with the support of technology, teachers can learn to enact collaborative and other discussion-based learning and social learning practices in their classrooms, and students benefit when they do, even in challenging urban environments. Resources have been made publicly available to facilitate widespread uptake of these practices.1
References
[9] Adamson, D. et al. (2014), “Towards an agile approach to adapting dynamic collaboration support to student needs”, International Journal of Artificial Intelligence in Education, http://www.cs.cmu.edu/~dadamson/pubs/IJAIED2013_Adamson_Agents.pdf.
[12] Adey, P. and M. Shayer (1993), “An exploration of long-term far-transfer effects following an extended intervention program in the high school science curriculum”, Cognition and Instruction, Vol. 11/1, pp. 1-29, https://doi.org/10.1207/s1532690xci1101_1.
[4] Azmitia, M. and R. Montgomery (1993), “Friendship, transactive dialogues, and the development of scientific reasoning”, Social Development, Vol. 2/3, pp. 202-221, https://doi.org/10.1111/j.1467-9507.1993.tb00014.x.
[5] Berkowitz, M. and J. Gibbs (1983), “Measuring the developmental features of moral discussion”, Merrill-Palmer Quarterly, Vol. 29, pp. 399-410, https://doi.org/10.2307/23086309.
[10] Bill, V. et al. (1992), “From cupcakes to equations: The structure of discourse in a primary mathematics classroom”, Verbum, Vol. 1/2, pp. 63-85.
[11] Chapin, S. and C. O’Connor (2004), “Project challenge: Identifying and developing talent in mathematics within low-income urban schools”, Boston University School of Education Research Report, Vol. 1, pp. 1-6.
[22] Clarke, S. et al. (2003), “The impact of CSCL beyond the online environment”, To See the World and a Grain of Sand: Learning Across Levels of Space, Time and Scale: CSCL 2013 Conference Proceedings, Vol. 1, pp. 105-112, http://hub.hku.hk/handle/10722/194444.
[21] Clarke, S. et al. (2016), “Student agency to participate in dialogic science discussions”, Learning, Culture and Social Interaction, Vol. 10, pp. 27-39, https://doi.org/10.1016/J.LCSI.2016.01.002.
[6] De Lisi, R. and S. Golbeck (1999), “Implications of the piagetian theory for peer learning”, The Rutgers Invitational Symposium On Education Series. Cognitive Perspectives on Peer Learning, pp. 3-37.
[23] Dyke, G. et al. (2013), “Enhancing scientific reasoning and discussion with conversational agents”, IEEE Transactions on Learning Technologies, Vol. 6/3, pp. 240-247, https://doi.org/10.1109/TLT.2013.25.
[7] Gweon, G. et al. (2013), “Measuring prevalence of other-oriented transactive contributions using an automated measure of speech style accommodation”, International Journal of Computer-Supported Collaborative Learning, Vol. 8/2, pp. 245-265, https://doi.org/10.1007/s11412-013-9172-5.
[19] Howley, I., E. Mayfield and C. Rosé (2013), “Linguistic analysis methods for studying small groups”, in International Handbook of Collaborative Learning, Taylor and Francis Inc., https://learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf (accessed on 13 November 2018).
[25] Jordan, P. et al. (2007), “Tools for authoring a dialogue agent that participates in learning studies”, Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, pp. 43-50.
[15] Kuhn, D. et al. (2013), “Developing norms of argumentation: Metacognitive, epistemological, and social dimensions of developing argumentive competence”, Cognition and Instruction, Vol. 31/4, pp. 456-496, https://doi.org/10.1080/07370008.2013.830618.
[18] Kumar, R. and C. Rosé (2011), “Architecture for building conversational agents that support collaborative learning”, IEEE Transactions on Learning Technologies, Vol. 4/1, pp. 21-34, https://doi.org/10.1109/TLT.2010.41.
[17] Kumar, R. et al. (2007), Tutorial Dialogue as Adaptive Collaborative Learning Support, http://www.cs.cmu.edu/~cprose/TagHelper.html.
[24] Mayfield, E. and C. Rosé (2013), “LightSIDE: Open source machine learning for text accessible to non-experts”, in Handbook of Automated Essay Grading, Routledge Academic Press.
[1] Michaels, S., C. O’Connor and L. Resnick (2008), “Deliberative discourse idealized and realized: Accountable Talk in the classroom and in civic life”, Studies in Philosophy and Education, Vol. 27/4, pp. 283-297, https://doi.org/10.1007/s11217-007-9071-1.
[3] Resnick, L., C. Asterhan and S. Clarke (2015), Socializing Intelligence Through Academic Talk and Dialogue, American Educational Research Association, Washington, DC, https://www.jstor.org/stable/j.ctt1s474m1.
[2] Resnick, L., S. Michaels and C. O’Connor (2010), “How (well structured) talk builds the mind”, in Innovations in Educational Psychology: Perspectives on Learning, Teaching and Human Development.
[16] Rosé, C. and O. Ferschke (2016), “Technology support for discussion based learning: From computer supported collaborative learning to the future of Massive Open Online Courses”, International Journal of Artificial Intelligence in Education, Vol. 26/2, pp. 660-678, https://doi.org/10.1007/s40593-016-0107-y.
[26] Rosé, C. et al. (2001), “Interactive conceptual tutoring in Atlas-Andes”, in Proceedings of the 10th International Conference on AI in Education.
[20] Sionti, M. et al. (2011), “A framework for analyzing development of argumentation through classroom discussions”, in N. Pinkward and B. McClaren (eds.), Educational Technologies for Teaching Argumentation Skills, Bentham Science, http://www.cs.cmu.edu/~huaai/pub/Sionti.pdf.
[8] Teasley, S. (1997), “Talking about reasoning: How important is the peer in peer collaboration?”, in Discourse, Tools and Reasoning, Springer Berlin Heidelberg, Berlin, Heidelberg, https://doi.org/10.1007/978-3-662-03362-3_16.
[13] Topping, K. and S. Trickey (2007), “Collaborative philosophical inquiry for schoolchildren: Cognitive gains at 2-year follow-up”, British Journal of Educational Psychology, Vol. 77/4, pp. 787-796, https://doi.org/10.1348/000709907X193032.
[14] Wegerif, R., N. Mercer and L. Dawes (1999), “From social interaction to individual reasoning: An empirical investigation of a possible socio-cultural model of cognitive development”, Learning and Instruction, Vol. 9/6, pp. 493-516, https://doi.org/10.1016/S0959-4752(99)00013-4.
[27] Wen, M. et al. (2017), “Supporting virtual team formation through community-wide deliberation”, in Proceedings of the ACM on Human-Computer Interaction, ACM, https://doi.org/10.1145/3134744.
[28] Wen, M. et al. (2018), “Transactivity as a predictor of future collaborative knowledge integration in team-based learning in online courses”, in Proceedings of Educational Data Mining (EDM 2016), https://www.edx.org/course/medicinal-chemistry-.