Volume 3, Issue 1, Spring 2022. DOI: 10.1037/tmb0000063
The increased reliance on online education and educational technologies more generally has laid bare the need to more deeply consider how researchers, designers, and educators can improve the quality of technology-mediated learning. To address this need, more than two dozen experts from a variety of fields came together to discuss the challenges that educational technology must address in the immediate future. These experts were tasked with identifying barriers to and potential solutions for delivering high-quality and equitable online and remote education. This article examines the themes and topics that emerged from these discussions and proposes a Collaborative Framework for Accelerating Online Education. This framework highlights the need for rapid experimentation within larger design cycles as well as the coordination and cooperation of multiple stakeholders across all phases of research and development. The themes, topics, and framework that emerged from this work serve as a call to action for innovative approaches to developing and studying online education.
Acknowledgments: The authors wish to thank Tyesha Ferron (Georgia State University) for her assistance in organizing workshop events.
Funding: The authors acknowledge National Science Foundation (NSF) No. 2035359 grant “Improving Online Education Through Technology, Research, And Data” that provided financial support for the workshop.
Disclosures: The authors are compensated by the individual institutions and companies listed in their affiliations.
Correspondence concerning this article should be addressed to Kathryn S. McCarthy, Department of Learning Sciences, Georgia State University, P.O. Box 3978, Atlanta, GA 30302, United States [email protected]
Online education has been steadily growing since the 1990s (Palvia et al., 2018). The coronavirus disease (COVID-19) pandemic in the spring of 2020 fast-tracked this growth when over a billion students suddenly found themselves unable to attend their brick-and-mortar schools. The sudden shift of the world’s students to emergency remote instruction1 demonstrated the power and potential of online education to the masses, but it also highlighted that there is much room for improvement and that these improvements need to be rolled out, evaluated, and refined quickly.
COVID-19 has given researchers, developers, and educators an opportunity to go beyond “getting back to normal” and use these unprecedented times as a leverage point for accelerated collaboration and development toward educational technologies that are more “efficient, effective, and equitable” (Thomas & Rogers, 2020). The current project was aimed at reimagining educational technologies and online learning. In doing so, we gathered together a panel of experts from industry, research, and education and asked them to discuss the future of online education. Through multiple rounds of discussions, the panel identified key themes and topics that will support the scaling up and scaling out of impactful online education that spans across the curriculum, across the lifespan, and across the diversity of users and stakeholders.
Online education got its start as a way for part or full-time workers to complete an undergraduate or graduate degree on nights and weekends. In recent years, online education has become more ubiquitous, with more and more K–12, higher education, and adult learning programs leveraging online curricula. In this article, use the term online education, or more broadly online learning, to refer to technology-mediated learning activities that are completed outside of physical classrooms. While there are many opportunities for self-directed learning on the internet (e.g., Khan Academy; SkillShare), we constrain the current work to more formal online education in K–12 or higher education contexts. Of note, we do not limit our definition of online education to fully remote instruction. That is, online education may refer to a fully asynchronous online course, but it may also involve a blended/hybrid class or even a fully in-person class in which there are assigned learning activities completed at home.
Online education is made possible through educational technologies. At the most basic, educational technology could refer to a “place” to view an online video, type an answer, or upload an assignment. However, there are a growing number of technologies, or tools, that can do a variety of sophisticated tasks such as automatically grading assignments, aggregating and visualizing student performance data (i.e., dashboards), and using these data to provide scaffolding and feedback to individual learners, either automatically or through a human-in-the-loop system. Notably, educational technologies can be used outside of online education contexts. For example, a teacher may use a SmartBoard during an in-person lecture, or a student may use a Learning Management System to find out their grade from an in-class exam. Thus, online education and educational technology are not interchangeable. However, an understanding of how researchers, designers, and policymakers can improve online education demands an understanding of educational technologies more broadly.
The expert panel agreed that perhaps the most critical issue in online education currently is that many people involved with online education are siloed from one another. Those who are developing “the next big system” are rarely in regular contact with experts with the theoretical and practical classroom know-how to ensure the success of that system. As a result, online technology is often poorly aligned with the needs of the target classrooms and without clear consideration of how the technology can be embedded into existing pedagogy. In some ways, this is because educational technology is often conceptualized as a vertical landscape of tools, with the focus on the platforms and systems themselves. Convergent and accelerated research should instead consider a broader conceptualization of a learning ecosystem. This ecosystem includes not only the technologies, but also the context(s) in which they are used, their interplay, and the variety of end users and stakeholders who interact with the technology.
The siloing of industry and academic research has slowed the progress of online education. These two realms tend to move at different speeds and have differing benchmarks of success. Successful industry-built technologies tend to be those that are quick-to-scale. As a result, commercial products tend to rely on dated methods of instruction (i.e., passive lectures, surface-level activities, linear progression, and minimal corrective feedback) that are easy to design and deploy, but do not take advantage of the rich body of research on empirically supported approaches to personalized and interactive learning. In contrast, research in the Learning Sciences and the psychological and educational research on the “science of learning” suggests that meaningful, long-term learning emerges from tasks that are active and engage and tailore to students’ individual needs and experiences (e.g., Chi & Wylie, 2014; Dunlosky et al., 2013; Glaser, 1991; National Academies of Sciences, 2018; Sawyer, 2006).
On the other end of the spectrum, technologies developed in academia are carefully designed and developed, and success is based on large-scale randomized control trials. Although rigorous experimentation and empirical validation of approaches are critical to education and educational technologies, academy-based approaches are time and resource-intensive and, often, by the time technologies have been built and fully vetted, the field has often moved on and there remains little funding to keep the project moving ahead.
In recent years, there have been increased efforts for convergence across fields and the development of physical and virtual spaces through which various stakeholders can come together to study and improve technology-mediated learning. Infrastructures such as LearnSphere (Koedinger et al., 2017), Generalized Intelligent Framework for Tutoring (GIFT; U.S. Army; see Sottilare et al., 2012, 2018), and groups like the Learner Data Institute (Rus et al., 2020) and the International Alliance to Advance Learning in the Digital Era (IAALDE2) have emerged to address the need to increase the efficiency and efficacy of learning in technology-mediated environments. It is in the spirit of these efforts and part due to the onset of the COVID-19 pandemic, that the U.S. National Science Foundation (NSF) funded the workshop that led to the formation of the expert panel queried in this article. The panel came from many intersecting fields of educational technology and online education and represented industry, academia, and public schools. Inspired by approaches like the Delphi method (Dalkey & Helmer, 1963; Sekayi & Kennedy, 2017), a virtual workshop was conducted in four rounds of meetings over about 1 month in the fall of 2020. In the first round, the entire panel of experts, led by an executive team (the first four authors), came together and engaged in rapid, round-robin-style, small-group discussions to brainstorm barriers to and potential solutions for delivering high-quality and equitable online education. The groups used shared Google docs to document their ideas, questions, and comments collaboratively in real time. Notes from the Round 1 sessions were collected and analyzed by the executive team to identify common themes and topics. This initial list included six topics as well as the introduction of an additional category of cross-cutting themes. Rather than a topic within themselves, these themes reflected critical considerations that permeate all aspects of improving online education. This list of themes and topics was then circulated among the experts and the experts were assigned to one of the topics for subsequent group meetings. These groups were engineered so that each topic had a mix of industry designers, educators, and researchers from various backgrounds in fields such as education, computer science, and psychology. These groups met for three more rounds in the following weeks. Each round (week), the group was given a set of questions to guide their discussion of the topic. Each round, a member of the group was asked to serve as the main notetaker, but the notes were again available in a shared Google doc and others were encouraged to comment and add as they saw fit. These small-group meetings were also facilitated by a member of the executive team. This member did not lead the discussions, but rather ensured that the groups were clear on their objectives and addressed or mediated any disagreements or concerns if they arose. In Round 2, the experts were asked to discuss their topic in ways that considered which disciplines, approaches, or methodologies would be necessary to move online education forward. In Round 3, the experts were asked to consider the organizations and stakeholders that would be involved in research and development, as well as use of the technologies. In Round 4, the experts were asked to identify potential deliverables and to consider not only the technologies themselves, but other types of deliverables that could increase the impacts of online education and research in education and educational technology more broadly. The notes from these rounds of meetings were compiled and explored. This analysis led to some topics being separated into multiple subtopics and some topics combining with others. The final list of themes and topics is discussed below.
By bringing together a variety of experts across different sectors and fields, we were able to identify unexplored or underexplored issues and were able to discuss emerging solutions (e.g., projects and technologies) that might serve to begin filling some of these gaps. One important discovery that emerged from these conversations was that there were several approaches and systems that were highly familiar to some, but new to others. In other places, there were clear areas of convergence and other places in which panelists shared differing perspectives. We have used analysis of these discussion notes not to identify wholly novel issues, but rather to bring together a diverse set of research and development activities to curate a set of themes that can be used to guide more systematic interdisciplinary efforts in online education.
Analysis of the discussion notes provided by the members of the expert panel revealed (a) three cross-cutting themes that run throughout all aspects of issues in online education and (b) six accelerated topics that must be addressed in the near future for online education and educational technology to have sustained and meaningful impact (Figure 1). Each is discussed in detail below.
In order to make educational technologies more efficient, effective, and equitable, there are at least three issues, or themes, that must be considered in any research and development task for improving online education that came from the expert panel. Theme 1: Developments in online education will remain of limited value if the goal is to merely substitute in-person instruction. Researchers and developers must leverage the unique affordances of technology to amplify teachers and improve education
The expert panel noted that widespread emergency remote instruction highlighted the fact that simply pivoting in-person instruction to hours of video conferencing calls is ineffective at best. Attempting to merely adapt instruction to substitute the classroom experience ignores the reality of how we interact with technology and the potential benefits or affordances of online education that can move education forward. This includes creating enriched environments and developing tools and methods that facilitate teachers’ ability to provide high-quality instruction at scale. Educational technologies must center the user, which includes offering assistance to students, teachers, and parents toward overcoming initial reservations or negative experiences with the use of technology in education. Ultimately, the educational technology community must better incorporate the voices of end users throughout development and refinement. Theme 2: Issues of equity and justice permeate all aspects of online education and educational technology. Power imbalances can be further exacerbated by the technology itself as well as who gets to be part of the discussion that drives the research and development of that technology. Explicit attention must be given to equity, diversity, and inclusion
The expert panel agreed that high-quality, high-impact instruction must be considerate of the sociocultural context(s) in which the instruction occurs. This includes, but is not limited to, race, ethnicity, culture, socioeconomic status, age, sexual orientation, and disability of the students. Researchers and designers need to consider these identities and experiences and must be conscientious of the intersectionality of these identities (e.g., Collins & Bilge, 2020; Crenshaw et al., 1995).
Perhaps the most high-profile concern for online education is that of algorithmic bias. Most educational technologies are reliant on some form of artificial intelligence (AI). Machine learning has become increasingly prevalent and impactful. AI has made possible rapid and accurate assessments of student learning which, in turn, support individualized instruction at scale. However, many AI approaches have raised concerns about how, if left unchecked, such approaches could perpetuate and magnify existing inequities or introduce new bias (Mayfield et al., 2019; Perry & Turner-Lee, 2019). As such, more research must be done to weigh the costs and benefits of increasing prediction accuracy (e.g., Yu et al., 2021). The increased reliance on AI and its rapid evolutions has highlighted the need to explicitly anticipate and address potential inequities and to carefully consider how decisions in developing and training algorithms (e.g., sample sizes, use of regionally or culturally limited samples) might impact how the results are interpreted and generalized.
One inherent limitation in the extent to which online education can serve as a “great equalizer” is that it presupposes that all learners have access to technology. However, there are many learners without regular access to personal computers (or need to share a single machine across multiple family members) and/or stable internet needed to attend classes or to complete web-based activities. This lack of access remains a major barrier in the ability to make sustained and widespread impact. Technologies and those in the educational technology ecosystem need to not only design systems around these problems, but to design for them. That is, research must consider not only how to avoid deepening the digital divide, but also how technology can be an agent of change. Indeed, there are those who have argued that data-based approaches based on AI, big data, and analytics have the potential to promote equity and social justice (e.g., Aguilar, 2018). These opportunities need to be further explored.
Finally, a focus on equity and inclusion is not limited to the technologies themselves. There must be greater diversity across those involved in the design, development, and evaluation of these systems. Convergence must include a variety of fields of study and partnerships across industry and a variety of academic institutions, including community colleges, Minority Serving Institutions (MSIs), and Historically Black Colleges and Universities (HBCUs). Theme 3: Improving online education requires better integration (and evaluation) of technological advances in areas of computer science.
The expert panel noted that in the past decade, there has been an explosion of development in areas such as artificial intelligence, natural language processing, machine learning, and augmented and virtual reality. Researchers on the more computational side of the learning sciences (e.g., educational data mining, learning analytics, AI in education) have quickly embraced these approaches and made many advances in research and development that can support high-quality online education. This work is making its way into the mainstream, but recent findings suggest that widespread adoption of such approaches remains elusive (e.g., Chen et al., 2020).
One consequence of these advances is that tools and platforms can provide increasingly complex learning activities at larger scale, which, in turn, yields increasingly large and increasingly complex data sets. The use of AI as well as big data brings into the equation a number of practical and ethical issues that need to be addressed to ensure that the data that is collected and used is compliant with privacy concerns and equitable to all the many stakeholders involved. Thus, the implementation of these new approaches requires additional time and expertise. As such, there is a need for larger and more diverse teams to merge theoretical knowledge and practical know-how from a variety of fields to ensure that the right types of data are being collected, analyzed, and interpreted efficiently and appropriately.
Each of the six topics identified by the expert panel presents multiple unique lines of inquiry, as well as intersecting considerations for collaboration. Thus, while these topics represent a coherent and independent track of research, they also provide opportunities for convergence across the broader spectrum of educational technology. In addition to these interactions, each of the six topics can draw upon the cross-cutting themes to strengthen their contributions to the future of online education.
In the past few decades, interrelated fields of educational, cognitive psychology, the science of learning, and the learning sciences have demonstrated that many traditional approaches to instruction do not support meaningful learning. These areas have generated a large body of theory-driven and empirically support approaches to instruction that highlight the need for individualized, student-centered activities that encourage personally relevant and active learning (see National Academy of Sciences’ How People Learn II, 2018, for a comprehensive review). Such approaches may be particular beneficial for addressing achievement gaps (e.g., Theobald et al., 2020). However, this area of work has also shown a disconnect between these best practices and common student and teacher practices. For example, students report relying heavily on study strategies such as rereading and highlighting, even though these techniques fail to support long-term or meaningful learning (Dunlosky et al., 2013; Miyatsu et al., 2018). The expert panel noted that many of the more recognizable online education platforms tend to perpetuate these practices. Many of the scaled-up, platforms tend to rely on linear one-size-fits-all instruction and relatively passive learning activities. For example, an analysis of 76 randomly selected Massively Open Online Courses (MOOCs) found that, while the courses were well-packaged and followed a logical order of content, few (less than 10% of the courses) implemented authentic, student-centered problem-solving, collaborative activities, or instructor feedback (Margaryan et al., 2015). Leveraging educational technologies may support moving beyond the status quo toward classroom instruction and activities that are more aligned with best practices from the science of learning. For example, research in applied memory suggests that repeated and interleaved practice can support more long-term and durable learning. Indeed, investigations adaptive retrieval practice provide promise as a means of improving learning and retention in the classroom (Eglington & Pavlik, 2020; Greving et al., 2020).
Although there are well-documented benefits of personalized learning, it is practically difficult and resource-intensive for classroom teachers to provide individualized instruction. Individualization becomes increasingly challenging as class sizes grow. However, online education and increasingly sophisticated technologies make individualized instruction more feasible. Fully automated and human-in-the-loop systems allow students to work independently or in small groups and the system can monitor behaviors, provide feedback, and make recommendations for individual students. Recent work has demonstrated that such adaptive learning environments yield superior learning outcomes as compared to lecture-focused online courses (Shi et al., 2020). Although there is growing number of adaptive learning technologies, it seems many of these tools emerge from the research sector and state-of-the-art adaptive features have only recently seen integration into commercial platforms. Thus, ongoing work must consider how educational technologies align with and support principles and processes emphasized in theories of learning and instruction (e.g., Crompton et al., 2020).
Another concern noted by the expert panel in the physical and virtual classroom is the overreliance on high-stakes summative evaluation as a means of measuring learning. Again, this is space in which recent advances in technology can move education forward. Researchers have used data mining techniques to implement embedded or “stealth” evaluations (e.g., Shute, 2011) of student learning within computer-based learning activities. The emerging work on stealth assessments suggests that using student behaviors (such that their choices or their language) during learning activities can yield more rapidly available and nuanced learner models that capture cognitive and affect states and can do so in ways that are less disruptive (a potentially stress-inducing) than stopping to take quizzes or tests. In addition, these more robust learning models can drive increasingly individualized feedback to keep students engaged and learning (e.g., Chen et al., 2021; Fang et al., 2021; McCarthy et al., 2020; Mills et al., 2021; Shute et al., 2021).
Researchers have often been constrained to short duration, lab-based studies or fieldwork in a relatively small number of classrooms. Online education provides an opportunity to capture learning over time and learning at scale. By collecting and analyzing large, longitudinal data sets across a variety of contexts, researchers can develop a greater understanding of how to implement more accurate activities and feedback that are sensitive to the learner’s needs. Increased collaborative efforts to address these issues can help to more quickly develop accurate learner models that can be tested across broader samples of students and accelerate the speed at which we can implement analytics to provide just-in-time support that can significantly improve learning.
The expert panel identified four elements of online systems that need to be developed to better expand educational technology for all students. These are discussed below.
Consistent with the push for greater representation in the media, evidence from existing research suggests that representation is important in the classroom as well (e.g., Kim & Baylor, 2016; Miller et al., 2018). System designers in recent years have been more sensitive to broader representation, but many still have “default” settings or limited options for the skin color or gender expression of instructors, avatars, pedagogical agents, or nonplayer characters. Such attention to these issues in the development phase can have positive impacts on the increasingly diverse user based of online education and educational technologies.
COVID-19 has highlighted that educational inequities can be further exacerbated by technology. Many of the issues and recommendations in this article assume adding to or modifying a technology-enabled context. However, this is not a reality for all classrooms and for all students. Even the best technologies are rendered ineffective if students cannot gain access to them. Black, Hispanic/Latinx, and rural students are disproportionally more likely to fall behind in their studies due to lack of access, whether that is not having a computer or not having stable internet (Auxier & Anderson, 2020; Borrett, 2020). Of course, addressing the imbalances of access or an assumption of a technology-rich environment as the default would require larger paradigm shifts and policy changes that fall outside the scope of this article. They are, nonetheless, critical for equity. Devoid of large-scale reform, researchers and developers must consider low-tech alternatives that allow technology-mediated learning without reliable access. In addition, work can be done to help consider how technology could be distributed or altered to help bridge the digital divide. Such approaches have been adopted in other fields. For example, the TIPS by TEXT intervention uses SMS messaging to reach informal caregivers in low-socioeconomic environments to empower them with information about children’s socioemotional development and scaffolded activities to help them care for their child (Widen et al., 2020). Similar low-tech interventions have been developed to support low-literacy adults and English Language Learners (e.g., Ksoll et al., 2015), but such approaches have yet to be integrated into K–12 classrooms. In order to reach as many learners as possible, researchers and developers ought to consider integration of state-of-the-art tools, but also consider how those tools can be integrated into the context in which the tool is deployed.
The majority of extant technology remains teacher-centric with passive and linear one-size-fits-all instruction. Even systems that are more active and engaging still often assume an “ideal” or “average” user. However, work in aptitude-by-treatment interactions (and individual differences research more generally) suggests that targeting an assumed average often fails to meet the needs of any one individual (e.g., Connor & Morrison, 2016). This, in combination with the increasing heterogeneity of classrooms, demands that researchers and developers go beyond the question of “does it work?” to questions of for whom and under what conditions does it work (e.g., Lim et al., 2019). Research at the intersection of the science of learning and educational technologies have yielded a growing body of individualized and personalized learning-based interventions and systems. Many of these systems are still in relative infancy, but the empirical findings suggest that tailoring feedback and activities to students’ interests and knowledge has positive benefits for learning (e.g., Walkington & Bernacki, 2020). In order to provide individualized learning opportunities for all, technologies must be sensitive and responsive to group-level differences (e.g., race, gender, culture, language) as well as individual differences (e.g., interest, skill, knowledge). It is imperative that researchers conduct research with a greater diversity of students to gather larger data sets in which they can examine a number of moderated and mediation relations across interventions and individual differences. In addition to diverse data sets, it will likely be beneficial to collect and tag more context-specific data sets so that systems can be more responsive to a greater range of users who may have different needs, experiences, and familiarity with technology (e.g., Dolan, 2016). These data can then, in turn, drive more individualized and personalized instruction.
One major gap in the existing ecosystem of technologies is a lack of systems that are targeted toward younger (K–6) learners. While there are a variety of educational games for younger students, there are far fewer formal instruction platforms or methods of facilitating online learning for this age group. The switch to remote learning has been particularly challenging for the parents and teachers of young learners (e.g., Dong et al., 2020). Elementary school students need different support than their adolescent or adult counterparts (e.g., Darling-Hammond et al., 2020).
Elementary education also reemphasizes the need to consider all of the people involved in the educational ecosystem. Family dynamics play an important part in students’ development. However, some parents do not see themselves as “part” of their child’s education (e.g., Selwyn et al., 2011). Even those parents that do want to play a role in their child’s education may lack the time, resources, or skills to do so (e.g., Garbe et al., 2020). Thus, educational technologies for younger students must be sensitive to context in ways that can help draw parents into the learning process and equip them with the support they need to contribute to their child’s learning.
Much of the work with younger students tends to focus on the need to keep them engaged and interested. There is some question as to how much of the benefits of technology-mediated learning are due to novelty effects that are likely to wane as opposed to features and activities that support long-term gains. In addition to the specific course content, these tools must also address the fact that young children are just beginning to develop their identities as learners. Educational technologies need to not only grab a child’s interest but also begin to cultivate good habits of learning and self-efficacy (see the section on Social Emotional Learning and Self-Regulated Learning). As the pool of technologies for this age band continues to grow, there is a need for increased and rapid collaboration across designers, builders, and those who work with and study younger students. In order to move beyond studies of immediate impact, there must be a more intentional move toward longer term implementations, evaluations, and redesigns that target the varying needs of developing learners.
Younger students also present practical challenges. For example, children who are not yet proficient readers and writers may require the use of speech recognition technology. However, the majority of automated speech recognition (ASR) technologies are not accurate enough to understand and respond to children (Scanlon, 2020). Similarly, students’ open-ended responses (summaries, short-answers, messages) tend to systematically differ in their structure and content than adult language (Crossley et al., 2020). Thus, more work must be done across several fronts to make educational technologies accessible to and effective for all.
In addition to a consideration of the number and types of users, the expert panel agreed that more attention must be paid to the specific needs and experiences of those users. That is, there needs to be greater consideration and integration of human factors in educational technology (Roscoe et al., 2018).
As technology becomes more integrated into the classroom, rather than extramural activities, more consideration must be made to meet the needs of students, parents, and teachers and to help them coordinate their efforts. Indeed, some parents, teachers, and students continue to find educational technologies to be more of a hindrance and, as a result, are resistant to using technology in the classroom (Beckman et al., 2019; Howard, 2013). One complaint from many students and parents in remote learning during the pandemic was that they were overwhelmed by needing to quickly learn how to use a number of different tools and platforms, many of which are not meant specifically for the classroom. For example, most online conferencing tools (e.g., Zoom) were not designed with classroom teaching in mind. During the pandemic, K–12 and university instructors have relied on breakout room functions to facilitate group work. However, unlike in a face-to-face classroom, indicators of student learning (student conversation, seeing students take notes, body language, etc.) are not visible when students are in virtual breakout rooms and teachers cannot readily “keep an ear” on all groups simultaneously. A tool or functionality that could analyze such cues and flag to teachers which rooms they should step into would help teachers support students in the virtual environment. These types of practical considerations to meet the needs of the classroom are critical as online education becomes more prevalent.
This example brings to light the larger issue that teachers are often left out of educational technology discussions. Teachers are often mandated to use technologies and are given minimal, if any, training on how to use them. These tools are often inflexible in the sense that the teacher is limited to the topics and tasks that the system provides, and the teacher has little control over the structuring or timeline of instruction/practice within the technology. Teacher dashboards and interfaces are often included as an afterthought, making it difficult for teachers to integrate the technology into their class. Research needs to consider what kinds of tools could be built or refined to help teachers do their work (e.g., find quality texts, help simplify grading, help generate individualized learning tasks/plans, behavior management). With the increase of parents and teachers mediating student use of educational technologies, new tools can be built to help teachers with classroom management in online spaces and to more effectively facilitate collaborative tasks and feedback. For example, research in collaboration and group dynamics can help teachers to put their students into small-group arrangements that are the most likely to be effective for group problem-solving and to moderate and facilitate deep discussion in unsupervised groups while the teacher makes their rounds. Thus, online education must better integrate teachers as design and research partners so that the technologies can support rather than hinder teachers’ progress.
Due to practical constraints, many of the early educational technologies and online education courses were focused on content mastery in well-defined domains. However, the landscape has rapidly expanded to include online courses that span across the curriculum. The expert panel acknowledged that these courses present new challenges for learning at a distance and learning at scale, but they also acknowledged that the data generated from these courses can be leveraged to provide new insights into domain-specific learning and learning support.
Well-structured tasks (e.g., science facts, math problems, vocabulary quizzes) are easier to implement in classes because they can rely on repetitive practice in which responses can be quickly assessed as “correct” or “incorrect” (i.e., multiple-choice questions, numeric answers). However, these tasks reflect only a small subset of the types of skills and knowledge that students need to be successful in the knowledge economy.
Although there are many online courses available for the humanities, humanities education is far more challenging to scale. As a few simple examples, it is more complex to develop “practice problems” for tasks such as reading a poem and discussing its affective impact or using multiple conflicting documents to make a historical argument. A number of research teams have risen to this challenge by exploring quantitative/computational approaches to poetics (e.g., Jacobs & Kinder, 2020) and literary argumentation (e.g., Balyan et al., 2017) and developing technologies that provide instruction and support historical reasoning (e.g., Britt & Aglinskas, 2002) and higher order discourse skills (McNamara et al., 2004; Meyer & Wijekumar, 2007; Mostow, 2013; Roscoe & McNamara, 2013; see also Passonneau et al., 2017). However, many of these tools have developed relatively slowly in traditional development, refine, and randomized control trials (RCT) cycle. Panelists expressed frustration that many of these advances have yet to make their way into the “average” online course. Implementation of these technologies could make online learning in these domains more engaging and effective and the scalability and sustainability of such technologies could be greatly improved through increased partnerships across sectors. Thus, there remains great need to expand the number of technologies available across a broader range of domains and a greater variety of collaborators within the space.
Even in well-defined domains, educators have increasingly emphasized the need to engage in more sophisticated learning activities. For example, students in STEM disciplines are being called upon to engage in integrative argumentation tasks. This model of learning and evaluation disrupts standard notions of training one component skill at a time, because more authentic application and critical thinking tasks require the deployment of a larger number of microskills. In addition, these types of tasks often require more collaboration than what most technology currently affords (cf. Sun et al., 2020). Thus, ill-structured tasks are more complex for students to manage and more complex for educational technologies to leverage. In order to meet these demands, educators and educational researchers must work together to advance the evaluation of 21st-century learning skills and competencies while still meeting the current expectations from districts and state and national standards. Online education should offer activities that spark engagement as well as help automate scaffolding and evaluation to help the teacher with instruction. However, designer must take care to ensure that these tools are accurate, but also fair in terms of cultural sensitivity or variations in experiences, language, and dialect. Large scale and domain-specific natural language corpora along with learning outcomes and individual difference measures can support the development of natural language processing (NLP) tools (e.g., word2vec spaces) for educational data mining and more sophisticated adaptive systems that can respond to more complex learning tasks and be responsive to a variety of individual differences. Such methods of data collection and algorithm development would not only facilitate higher quality interventions, but also be used to better understand and develop theories of social interactions and socially constructed knowledge.
Beyond language, it may also benefit learning if online education could more readily provide just-in-time feedback based on multimodal data. Moving beyond point-and-click or fill-in-the-blank style problems would afford the opportunity to develop tools that analyze math handwriting or scientific drawings. These databases could be used to improve optical recognition of symbols and diagrams to assist with tutoring and could provide additional embedded assessment through examining not only what the student writes, but also how (and where) the information is conveyed on the tablet or page. That is, even if the task results in a quantified answer, these tools can be leveraged on more ill-structured data to make better individualized recommendations.
There are two approaches to broadening the scope of educational technologies in online education. The first is to fund the research and development of educational technologies that target specific domains and disciplines. The second is to develop tools that can be flexibly applied to a number of domains (i.e., domain-general). However, it is unclear the extent to which such tools would be more or less effective as compared to tools built for a very specific purpose. There were a number of disagreements from the panelists about the extent to which efforts should be made to address specificity as compared to generality. Development of technologies that address the instructional and assessment needs of a greater variety of fields will require increased partnerships across experts in educational sciences, educational practice, and technology development. It may be of more value for researchers and designers to develop generalized tools that can be embedded and adapted for more specific domains. For example, there is a need for automated constructive response tools that are domain agnostic or that can be more flexibly adapted to different disciplines. Existing automated summary evaluators (ASE) tend to focus on summary writing as a general skill rather than on the particular topic being read. Improved ASEs could be developed that better evaluate the extent to which student responses reflect deep understanding of the content and could deliver actionable feedback that supports content comprehension as well as more general reading and writing skills.
Alongside the larger move to place the student at the center of learning has been an increased focus on students’ traits and states and how these “noncognitive” factors influence learning. A critical future direction identified by the expert panel is the development of online educational tools that gather a variety of types of data (e.g., clickstream, eye tracking, language) to develop more precise learner models and more efficient feedback that are sensitive not only to the cognitive components of learning, but also to dynamics changes in students’ emotional and metacognitive states (e.g., D’Mello & Graesser, 2012). Two of the major areas of research and development are in social and emotional learning (SEL; Elias et al., 1997; Osher et al., 2016; Weissberg et al., 2015) and self-regulated learning (SRL; Azevedo & Hadwin, 2005; Pintrich, 2000; Zimmerman, 2000). Panelists with expertise in this area noted that many students are not exposed to SEL or SRL support and feedback at home or in other formal and informal contexts. Thus, explicit attention to SEL and SRL in the classroom may be important to help cultivate students who are not only ready to, but also excited, to learn. Indeed, interventions that help students monitor and manage their goals, emotions, and behaviors increase student learning and achievement (e.g., Durlak et al., 2011).
In addition to in-person interventions, there are educational technologies, such as MetaTutor (Azevedo et al., 2009), Help Tutor (Roll et al., 2011), and Betty’s Brain (Biswas et al., 2005), specifically built to develop learners’ SEL, SRL, and metacognition that have shown consistently positive effects. This work also shows that the impacts of such interventions are mediated by students’ preexisting knowledge and skills (e.g., Jansen et al., 2019). Thus, it is not only important to teach SEL and SRL, but to evaluate and respond to affective and metacognitive states. Students experience a variety of discrete affective states during learning and affective and self-regulatory components are highly related to engagement and learning in computer-based learning environments (e.g., Baker et al., 2010; D’Mello, 2013). Although evaluating and responding to these factors poses a number of methodological challenges (e.g., Greene & Azevedo, 2010; Winne, 2010), technology and convergent research around online education provide new opportunities to advance the measure, study, and support of SEL and SRL.
One limitation to much of the early work on measuring SEL/SRL in online contexts was that it relied heavily on self-reported measures, which are easy to implement, but ultimately subjective and gameable. Our expert panelists noted that many commercial systems continue to rely on such approaches. A point of leverage is that online learning contexts provide large, rich data sets that can be mined to identify SEL and SRL and to explore how these factors relate to achievement and learning (Koedinger et al., 2015). Researchers in learning analytics are also developing tools like nStudy (Winne et al., 2019) to make the collection and analysis of self-regulated learning behaviors more accessible. A richer understanding of these factors can help to develop more effective technologies and pedagogies. However, it is insufficient to be able to study SEL and SRL in archival data. Developing work in analytics and feedback for affective computing and self-regulated learning highlights the need to meet the student at their skill level and to provide scaffolding to keep the student on task and on track given their current state(s) (e.g., Uzir et al., 2020; Yadegaridehkordi et al., 2019). This requires the ability to quickly and reliably detect these states and to do so in noninvasive ways that do not interrupt the learning process (e.g., Bosch et al., 2015; Emerson et al., 2020). For example, the Eye-Mind Reader (Mills et al., 2021) addresses the notion that students tend to “zone out” an estimated 20%–40% of the time during learning tasks. Eye-Mind Reader relies on previous studies that examine the behavioral indicators of self-reported mind-wandering to develop machine learning detectors that use eye gaze patterns to predict when a student is mind-wandering. When mind-wandering is detected, the system prompts the student with active learning tasks to get them back on track. Most critically, the research team found that prompting these activities when a student shows disengagement was more effective for long-term learning than when these prompts were deployed at random.
While there is an increasing number of platforms measuring these noncognitive factors, the expert panelists agreed that more can be done to leverage the increasing number of online courses and students enrolled in online education to measure behaviors “in the wild” and at scale (e.g., Hutt et al., 2019). Such data can be used to extrapolate more accurate information about these processes which can, in turn, be used to derive more individualized and actionable feedback.
In addition to studying the states of individuals, there is a growing need to better evaluate and respond to collaboration and social interaction. The field of computer-supported collaborative learning has led the way in understanding how technology can be used to study and mediate high-quality learning. However, the explosion of social media has emphasized the need for more sensitive, rapid, and scalable ways to study these interactions. Collaborative projects might investigate the viability of automated sociometers. Such tools could better measure collaboration and other 21st-century skills like cooperation and social engagement.
The expert panel noted that paramount to convergence across fields and disciplines is the need to exchange ideas and data. As a result of the current siloing of the various stakeholders interested in educational technologies, there are few infrastructures or architectures to guide best practices in data sharing and collaboration.
Online education, and educational technologies more broadly, have the potential to generate immense amounts of data (e.g., time-on-task, clickstream, choices, reaction times, answering questions). On one hand, there is a “too much data” problem in that many tools on the market are collecting data from learners, but these data sets go unexamined or underexamined. Industry panelists expressed that their teams were highly interested in better leveraging this data, but that they often did not know where to start. On the other hand, there is also the problem of “too little data” in the sense that many educational data sets have too few data points to successfully triangulate cognitive, affective, and behavioral outcomes or to do so across a diverse group of learners. More specifically, recent work suggests a disconnect between the data available and the data that is needed (Rus et al., 2021; see also Reeves & Lin, 2020). Increasing collaborations and partnerships across the ecosystem can increase the number of voices at each stage of development and testing. Interdisciplinarity and communication cross-sectors throughout the lifespan of projects can better ensure that the “right” data are being collected to address both theoretical questions and practical needs. Thus, increased collaboration is needed so that different teams can ask different questions. This would better allow interested parties to take full advantage of the data that educational technologies produce (e.g., Heffernan & Heffernan, 2014). In addition to analyzing the extant data, collaboration can better ensure that the technologies are testing important questions and generating the types of data that are relevant to the diverse array of stakeholders in the online education ecosystem. More practically, many of the existing silos among stakeholders exist because of varying expectations and incentives across fields. Educational technologies that thrive on iterative improvement need to develop sustainably—platforms and tools tend to disappear or go stale when funding runs out or commercial systems may be hesitant to share proprietary data or approaches. In order to see iterative change and development, tools have to be developed with long-term plans and support in place to incentivize collaborations across companies and institutions.
Although convergence and collaboration are necessary, these new partnerships bring to the surface a number of ethical and practical concerns as identified by the expert panel. How do we build systems of data and strong learner profiles while maintaining privacy? How can we leverage the power of social media and the experiences that students have outside of the classroom while respecting boundaries in and out of the classroom? Perhaps the most pressing of all the tasks in the future of educational technologies for online learning is the development of instrumentation and other data sharing systems and tools that maintain privacy while allowing for the creation of cross-platform learner models that can support greater adaptivity. Throughout their day, students are using multiple forms of technology. They may log into a learning management system, complete a module in a massive open online course, read a web-based textbook, and then engage with an educational game. Despite each system generating a wealth of data that can help develop a rich learner model, these systems do not share data with another and even modules within systems are often stand-alone rather than integrative. Additionally, data tagging in education is not standardized, which presents challenges in aligning ontologies across systems. Better integration across lessons and across systems can support students making connections across topics (supporting transfer and deeper learning), helping to create a greater sense of purpose and continuity in the classroom. Combining these bodies of data can also help researchers to develop richer learner models and a deeper understanding of how students learn. Thus, convergent efforts are needed to consider how to support student moving from platform to platform more seamlessly including more policy-oriented work (e.g., advocating for federated learning for education) that can ensure data privacy.
There also remains a need for more robust and publicly available data sets (see Rus et al., 2021; Crossley et al., 2021). One means of accelerating the iterative development of online educational technologies is the continued development and expansion of data-processing platforms for learning engineering where data can be stored, analyzed, and perhaps most importantly, shared. Infrastructures such as DataShop,3 and LearnSphere4 have set the groundwork for sharing data sets and workflows; the GIFT5 project offers a number flexible tools and methods for authoring computer-based tutoring systems; and there are an increasing number of tools where researchers can quickly build and deploy A/B designs within large-scale classrooms (Experiments with Google6; ASSISTments7; BIRI8). Federal funding has also made possible metagroups or networks, like the Learner Data Institute9 and the Digital Learning Platforms to Enable Efficient Education Research Network.10 These tools have begun to address methods of collaboration as well as issues of scaling up and scaling out.
In addition, convergence could be accelerated through a social networking site for learning engineering that helps connect researchers, teachers, and administrators with learning platforms and industry partners. Such listservs exist, but they are limited in the ability to share information, collaborate, and facilitate genuine conversation. In addition to the technologies themselves, it would be of value for successful collaborations to document their process to put forth design pipelines and research plans. This might include documented workflows or memorandum of understanding (MOU) templates that ensure that different stakeholders cannot only engage interdisciplinarily but also come away from the project with meaningful outcomes relevant to their own professional objectives and expectations.
More broadly, the developing fields of design-based research (DBR), design-based implementation research (DBIR; Fishman et al., 2013; Puntambekar, 2018), and learning engineering (Dede et al., 2018) have demonstrated that educational research can benefit from both empirical rigor and more rapid, iterative advancements. In short, to keep pace with developing technology, state-of-the-art research, and the changing needs of end users, technology for online education must adopt rapid, iterative approaches to research and development. Additionally, the next generation of educational technologies must, from the start, be designed to leverage the best practices and state of the art in education and technology and with an eye toward equity and inclusion. This can only be done with intentional, interdisciplinary collaborations. Collaborations ought to include instructional designers, researchers, and developers from both industry and academia, data scientists who can support data wrangling, experts in the education sciences (e.g., learning sciences; science of learning; educational policy; diversity, equity, and inclusion), and various teacher and parent-partners. It is with these considerations in mind that we put forth a framework for improving online education.
Throughout the expert panel discussions, there was repeated emphasis that current approaches to research and design and the silos between industry, education, and research often prevented the rapid development of high-quality online education. Thus, collaborations in educational research and development must work to triangulate best practices across a variety of different research techniques and outcomes. Additional research can help to accelerate the speed at which we can evaluate and respond to the varying needs of instructors and students in ways that are sensitive to a wider variety of individual differences, dynamic states, and sociocultural contexts.
Thus, the expert panel reflected upon ways in which we could more intentionally and systematically increase collaboration and rapid iteration and advancements around our cross-cutting themes and accelerated topics described above. The result was the Collaborative Framework for Accelerating Online Education (Figure 2). The framework was inspired by instructional design such as the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model (see Molenda, 2003), design-based research (DBR; Barab, 2014; Puntambekar, 2018), and DBIR (Fishman et al., 2013).
The individual pieces are not novel (e.g., nesting within sociocultural context; experimental manipulations). However, what was realized in the discussions was that the different experts and their teams were working in separate research spaces (e.g., UX, experiments, data mining) with varying epistemologies and methods. This framework draws upon the strengths of different fields and modes of inquiry to formalize an approach to truly collaborative and iterative work that can support the development of higher quality and broader reaching online education.
The framework emphasizes large-scale design loops in addition to more rapid iteration. Each step in the process depends on the work done before it but iterates that work with the goal of improving the product. For example, researchers ensure that the technologies are effectively tested and that designers are collecting the right analytics and outcomes to test how learners engage with the technology and how these interactions influence a variety of outcomes including experience (motivation, engagement, etc.) as well as short-term learning gains. Deepened and long-term partnerships with educators, parents, and students mean that those who will work with the technology are not just end users, but codesigners whose ideas, needs, and concerns are considered and integrated during development rather than as an afterthought. Such collaboration accelerates the rate of theoretically motivated advancements and evaluations than is traditionally feasible in the lab or in industry and ensures an ongoing iterative cycle of data generation and analysis. Further, identifying and responding to the variety of topics and issues mentioned in this article would be intractable for a single team with relatively homogenous expertise. To create and scale the types of online education tools imagined here, there must be increased collaborations across disciplines, fields, and sectors.
In addition to the central loop, the framework includes a central core of continuous collaboration among a variety of stakeholders and a constant reflection on the cross-cutting themes of amplification, equity, and advanced technology as well as explicit recognition that work exists within, and must be sensitive to, the sociocultural context in which the research and development is being carried out. We argue that such approaches to highly collaborative research, development, and refinement are critical for accelerating online education tools to meet the needs of the ever growing and changing body of learners that they serve. Many of the projects and products we have described above have pushed the field forward and represent the promise that such work can have toward improving online education.
High-quality systems will lead to increased use, both in terms of the quantity of users and the quality and frequency of their interactions. This, in turn, increases the amount of data available to further mine for additional insights for future system improvement and new directions in research. By using online educational technologies as both testbed and outcomes, researchers and practitioners can improve the quality of education across contexts as well as inform and refine theories of learning. While RCTs remain an important benchmark for the success of educational activities and interventions, this approach highlights the need to consider more rapid, iterative refinements consistent with work in design-based research.
This framework enables several key aspects not afforded in the current modus operandi. First, technologies and products would be subject to theory-driven change, where the science of learning is privileged in development toward clear learning goals and recognition of the many internal and external factors that influence learning processes (e.g., Topic 1; Topic 5). This would further allow for more rapid experimentation, where “fast fail” experiments can be run to quickly adapt technologies to improve short-term outcomes. Essential to this is an internal culture of iterative improvement toward measurable, long-term benefits that will depend on diverse teams. Thus, convergent research in online education will not only include those who work “in” educational technology (i.e., industry, university, and nonprofit research) and end users (students, teachers, parents), but also researchers and developers who work in related domains, such as computer science, AI and machine learning, natural language processing, and virtual and augmented reality. Even beyond this, convergent teams could also include government and policymakers and experts in other relevant fields such as data ethics, child development, and those with expertise in media and marketing (Topic 6).
Thus, when establishing a thriving “ecosystem of educational technology,” platforms would need to internalize the considerations of multiple stakeholders (Topic 2; Topic 4). This will require intentional inclusion and consideration of a number of complementary and competing outcomes including user experience and usability, personalized learning, teacher control, attention to and scaffolding for motivational and affective factors, as well as a number of various learning outcomes (e.g., immediate performance, long-term retention, deep comprehension, and the ability to transfer knowledge to new contexts; Topic 3).
The framework is the culmination of our expert discussions and offers a formalized “starting point” for those in educational technology and online education to engage in increasingly interdisciplinary work that stands to make more rapid and meaningful impacts on a broader range of learners. It is perhaps an idealized imagination of how this accelerated work would be carried out. However, in the spirit of iterative refinement, we hope that others will reflect upon how to implement these considerations into their own research and the framework can be revised and amended to reflect continued improvements to online education and the way that research and development in online education is done.
The popularity of online learning has grown as policymakers, researchers, and instructors acknowledge the need for adapting instruction in response to COVID-19. While the practice of online learning is not new, the convergence of recent developments in educational technologies, the science of learning, AI, and NLP can provide critical contributions to online learning. These collaborations can also promote high-quality online instruction in varied contexts and to diverse learners who differ across a wide array of dimensions, such as skills, knowledge, and motivation.
True convergence must consider the full ecosystem of those who develop, use, and are affected by educational technology. This means that online education needs to be reimagined to include teachers, students, and families as codesigners and partners, rather than merely passive consumers. In order for online education to meet its potential, teams of researchers, designers, developers, computer scientists, and educators from a variety of backgrounds and experiences must engage in long-term collaboration and iterative design. Having multiple viewpoints, goals, and expertise can support the development of technologies that can evaluate and assess learners and other end users across a variety of dimensions and outcomes in order to provide personalized instruction and feedback that keeps students excited, engaged, and optimally learning.