Analysis of Formative Through the UDL Framework

Introduction

In recent decades, there has been a growing interest among instructors of English as a Second Language in leveraging educational technology to enhance the learning motivation of middle school students learning English as a Second Language (Pazilah, Hashim & Yunus, 2019). There is a need to address this particular group of learners because they are known for having generally low motivation in learning English as a Second Language, and digital learning tools have the potential to increase their excitement, engagement and motivation towards learning (Al-Hoorie, 2018; Wood, 2022). Fortunately, this group of students is also known for having high technological literacy, which makes it easy for them to understand and use new digital learning tools (Masry-Herzallah & Stavissky, 2021). This report thus attempts to answer the following question: “Does the digital learning tool ‘Formative’ help enhance the learning motivation of middle school students learning English as a Second Language?”

In short, Formative allows learners with any device with a browser and Internet access to receive multimodal information, answer teacher-set questions of a wide variety of types in real time, see live responses of other learners, and benefit from teaching informed by the responses (EdSurge Product Index, 2022). The tool can be used in traditional classroom settings and online instruction, and it is designed for both curriculum delivery and assessment (EdSurge Product Index, 2022). While the platform is not specifically designed for the learning of English as a Second Language, it provides a library of question-sets for different subjects, such as Mathematics, Language Arts, and Astronomy, designed by the Formative Team or Formative users, many of whom are in-service classroom teachers.

The platform has several aims, such as encouraging discussions and deeper learning, and improving student engagement (EdSurge Product Index, 2022). These learning outcomes are important because they highlight the goal of education: It’s not about how much teachers have taught, but how much students have learned (Lujan & DiCarlo, 2006). The quiz-like questions on the platform not only provide an opportunity for learners to “demonstrate what they have learned”, but they also prompt learners to “discuss the individual questions immediately” after completing the questions, and hence enhancing their understanding of concepts (Lujan & DiCarlo, 2006). Most importantly, through engaging students in the learning process thanks to the interactivity of the platform, Formative has the potential to increase the motivation of students in learning, which is especially important to instructors of English as a Second Language.

Written with the Formative Team in mind, in particular the Lead Software Engineer and the Lead Product Designer, this report aims to (1) evaluate the strengths and weaknesses of Formative, which provides “teaching tools to engage, instruct, and assess” (Formative, n.d.), (2) compare its intended use and the actual outcomes, and (3) suggest ways to address its shortcomings and enhance its features. Analyzing how middle school learners of English as a Second Language stay engaged through Formative will help the platform better support these learners in learning various aspects of the English language and develop new features to address the pressing challenges faced by English language instructors and learners.

Description of the Learning Tool

Learners

The intended learners of this tool are K-12 learners, higher ed learners and adult learners, which can be reflected in the options for “Grades” in Library Filters. As revealed by other Library Filters, the intended learners are learners of learning domains ranging from languages (e.g. French, Turkish) to school subjects (e.g. Algebra, Economics, Statistics).

Given Formative’s heavy emphasis on interactivity, it is apparent that its designers believe learners want to actively participate in class through sharing their responses in various ways and learn collaboratively with peers by seeing the live responses of other learners in class. The designers probably also prioritize immediate feedback on students’ understanding of learning concepts because they believe in the potential of the tool in informing teaching, as teachers can gauge learners’ understanding of concepts through their responses submitted to the platform.

Technology & Context

Formative does not require installation, as it works on any device with a browser and Internet connection. This means that learners can use a computer, a mobile phone, or a tablet to access the platform.

The designers have leveraged web technology to enable many ways of creatively presenting knowledge. For example, teachers can embed a video recording of themselves explaining a concept, an explorable explanation, or a Google Drive folder containing learning resources. Web technology has also made it easier for learners to share their responses without using external applications and creating files. For instance, learners can make an audio recording of themselves responding to a question, and they may also show their steps in calculations by drawing on a digital whiteboard. The many interactive web tools provided by the platform allow learners to submit certain kinds of responses with ease, such as Graphing and Hot Text (Figures 1 & 2). Instead of spending time drawing a graph or boxing words on paper, web technology helps learners take care of these details.

The feature “Type of Formative” shows where and when Formative is designed to be used. When teachers publish their Formative exercises to the public library, they may choose from options such as warm-up, classwork, exit slips, assignment, and summative assessment (Figure 3).

Warm-up

Formative allows embedding of videos and various question types to collect learner responses before the lesson. In this example, a short video is included in the Formative exercise, and learners are instructed to watch the short video, answer a multiple-choice question, and show their work by drawing. This is commonly seen in the “flipped learning” approach, which takes place outside the classroom (Eppard & Rochdi, 2017).

Classwork

In this example, various question types are included to prompt learner responses in class and encourage discussions during the lesson. For instance, the matching activity prompts learners to think about why they match the pictures to the captions and share their thoughts with their peers. Teachers can also show live responses to allow learners to learn from their peers as they formulate their responses in real time, as shown by the screenshot below (Figure 5).

In this example, learners are also prompted to upload photos for their group work, and hence promoting collaborative learning. Finally, teachers can make use of Formative to check the answers of learners’ classwork in class quickly by prompting students to upload photos of their textbook.

Exit Slips

Exit tickets are short response tasks that teachers administer to students after an activity to gauge their level of understanding and adapt their instruction (Fowler & Richards, 2019). In this example, students are asked to answer 10 multiple-choice questions at the end of a lesson as a way for the teacher to check their understanding of the mathematics concept “exponent”.

Assignment

Teachers can make use of the platform to assign homework and collect responses electronically. In this example, the teacher has uploaded the electronic version of a worksheet that was designed to be completed using paper-and-pen. Instead, students are required to submit their answers to Formative by typing responses and showing their work by drawing on grids. The platform has an automatic grading function that helps teachers grade responses based on the pre-set answers (Figure 6). Presumably, assignments are completed outside of class time and classroom.

Summative Assessment

Summative assessments are generally high-stakes assessments used to assess how much learning has taken place at the end of segments of instruction (Dixson & Worrell, 2016). In this example, assessment guidelines are included in the introduction section, and learners are required to complete four essay responses on a selected book. Like most summative assessments, this assessment takes place in school during class time. Interestingly, learners are encouraged to participate in a small group discussion before answering the questions.

For middle school students learning English as a Second Language, Formative provides many tools which facilitate their learning experience. For instance, Free Response and Audio Response allow learners to practice writing and speaking respectively, and teachers can review the responses during and after the lesson. This is particularly helpful for language teachers, as they are often not able to give immediate feedback to all the learners during class time due to the large student-teacher ratio. For the learning of grammar items, which is common in learning English as a Second Language, teachers may employ a wide range of question types to gauge learners’ understanding of the grammatical concepts, such as Categorize, Drag and Drop, and Match Table Grid. These kinds of questions check learners’ level of understanding while sparing them from writing long sentences, which is not the goal of grammar teaching.

Content & Learning

As Formative is a digital learning tool that facilitates learning and teaching rather than providing learning content, the current analysis will be made on how the designers attempt to help learners achieve learning outcomes.

One of the learning outcomes of Formative is improving student engagement (EdSurge Product Index, 2022). This reveals a barrier to understanding the subject matter that the designers of Formative might see – lack of engagement in learning. Field studies in schools have revealed that many lessons are dominated by teacher talk, and students are often left as passive and bored spectators (Niemi, 2007). Formative provides a wide range of question types for teachers to choose from, including traditional question types such as Fill In the Blank and Multiple Choice, less common ones such as Hot Spot and Show Your Work, and ones that allow learners to submit multimedia materials, such as Audio Response and Video Response.

The platform’s heavy emphasis on learners’ interactions with the subject matter and with other learners also shows that the designers deem this necessary in order for successful learning to occur. Question types such as Free Response and Video Response allow learners to evaluate and create in their learning, possibly with other learners, which promotes both deeper learning and collaborative learning.

Another important element of learning, feedback, is also a fundamental component of the platform, as the name “Formative” suggests. In the context of learning English as a Second Language, teachers’ provision of feedback is especially important as it guides learners’ modifications to their original answers following feedback (Oliver & Mackey, 2003). In regular lessons, it is a common practice for teachers to select only a few students to answer a question and give feedback to their answers. However, through Formative, all students can submit their responses at the same time, and the platform automatically grades the responses based on pre-set answers. Teachers can also override the automatic grading and award points to answers that they find acceptable (Figure 8). The inclusion of this feature reflects the designers’ belief that learners need feedback in order to succeed at learning. Studies show that for effective learning to occur, learners need to know if they are on the right track or not, and know what they can do in order to improve and learn (Schwartz, Tsang, & Blair, 2016).

Not only can learners benefit from the feedback provided by the platform, but teachers can also benefit from the statistics provided by the platform. Formative informs teachers of students’ performance through a dashboard (Figure 9). Teachers can make use of the statistics provided by the platform to learn how to teach more effectively – they may find the need to clarify a difficult concept that many learners have not grasped during the lesson, or skip some teaching steps if learners have already mastered a concept.

Another goal of the platform is to encourage deeper learning (EdSurge Product Index, 2022). Formative facilitates more effective instruction by allowing teachers to insert six content types into a Formative exercise: Audio, Embed (e.g. a webpage frame), Image, Text, Video, and Whiteboard. This reflects the designers’ attempt to tackle one of the barriers to understanding the subject matter – lack of multimodal learning. Research has shown that interactive, learner-controlled multimodal learning can enhance learning and engage learners in the presentation of new knowledge, and that multimodal learning is most effective when learners can apply their newly acquired knowledge and receive feedback (Gilakjani, Ismail & Ahmadi, 2011). In this sense, Formative has succeeded in achieving the goal of facilitating multimodal learning, as teachers can insert multimedia materials which learners can control, and learners can apply what they have learned immediately by answering the questions that follow the multimedia materials and receive immediate feedback from teachers and the platform.

Evaluation of the Learning Tool

Overall strengths and weaknesses

After discussing the various features of Formative which improve student engagement and encourage deeper learning, this section will evaluate the strengths and weaknesses of the platform in helping enhance the learning motivation of middle school students learning English as a Second Language. The Universal Design for Learning (UDL) Framework will be employed in this analysis because it examines the potential of a digital learning tool in enhancing learner engagement through providing means of interactions in the learning process and in facilitating acquisition of knowledge through providing multimodal information, both of which closely align with the aims of Formative (CAST, 2018).

The UDL Framework evaluates a digital learning tool based on its capability to (1) provide multiple means of engagement (ways in which learners are “motivated to learn”); (2) provide multiple means of representation (ways that learners “perceive and comprehend information”); and (3) provide multiple means of action & expression (ways that learners can “navigate a learning environment” and “express what they know”) (CAST, 2018). The following discussion will be based on whether Formative has achieved these principles based on user input from seven interviews with students.

User input

To better understand the user experience of Formative, seven interviews were conducted in November 2022 with learners of English as a Second Language in Hong Kong. The participants were recruited through volunteer sampling and they have all been taught by the author of this report for one year. Before the user testing, they were informed of the general purpose of the test and had the opportunity to ask clarifying questions before giving informed consent to participate in the testing. They were interviewed on Zoom and were instructed to share their screen so as to show how they were interacting with the tool. Afterwards, they were asked to give an overall evaluation of the tool on interactivity, ease of use, and purpose of the tool.

Participants

*Note: Secondary schools in Hong Kong are divided into three tiers. Students in top-tier schools are the top 33% of students in Hong Kong in terms of academic ability, and students in middle-tier schools are the next top 33% of students in Hong Kong. Students are allocated to different secondary schools based on their academic performance in Chinese, English, and Mathematics in primary school.

Method

A sample Formative exercise was created specifically for the user testing to examine the potential of the platform in enhancing learner engagement and promoting deeper learning. Titled “Photo Captions & Popular Culture”, it covers the topic of popular culture, which is one of the elective units in Hong Kong’s current English Language Curriculum (Curriculum Development Council, 2017). The first section of the exercise, “Photo Captions”, consists of a matching question and a free response question (Figures 11 & 12). The second section of the exercise, “Popular Culture”, consists of an embedded YouTube video, a multiple selection question and two free response questions (Figures 13 to 16).

Nine questions were asked to find out the participants’ views on Formative. These questions were generated to test UDL principles. Some of these questions include: “This website allows you to see the responses of other students. What do you think will be the effects of this function?”, “What are the differences between the teacher playing a video in class and the teacher allowing students to play the video at their own pace?”, and “Overall speaking, is this tool easy to use?”.

Analysis of Formative through the UDL Framework

(1) Provide multiple means of engagement

a. Recruiting learner attention and engagement — Definition

The UDL Framework evaluates whether a digital learning tool enhances learners’ attention and engagement by examining whether it optimizes “individual choice and autonomy” (CAST, 2018). The framework deems this important because individual choice and autonomy increase “the degree to which [learners] feel connected to their learning”. The user testing allowed students to choose the aspect(s) of popular culture they like most and upload a photo of their choice to Formative.

Evidence

A few participants have pointed out that Formative allows students to share videos and photos, which is not possible in a traditional lesson. Some of them also suggested that Formative allows teachers to get “more interesting answers from students”. This can be reflected in the participants’ responses to Question 4, which requires them to upload a photo related to popular culture. Some of the responses, such as the Internet meme “Ricardo Milos” and the single cover of “Bones” by the US rock band Imagine Dragons, may be unfamiliar to teachers and other students, but through image submission via Formative, learners can share about what they like with their classmates. When asked about the purpose of free response questions, a participant noted that they “increase students’ engagement and enhance their motivation in learning.”

Discussion

Based on the observation, it can be concluded that by leveraging technology to allow learners to upload multimodal responses, Formative allows learners to be engaged by information and activities that are relevant and valuable to their interests and goals.

b. fostering collaboration and community — Definition

The UDL framework evaluates whether a digital learning tool fosters collaboration and community by examining if it “encourage[s] and support[s] opportunities for peer interactions” (CAST, 2018). To explore the possibility of peer learning through Formative, participants were asked to complete Question 2, which was a free response question that required them to write a 50-word caption for a photo. As participants were writing their captions, they were shown other written responses.

Evidence

Participants were ambivalent about Formative’s ability to show responses of other learners in real time. For those who liked the function, they reported that Formative allowed them to see their friends’ answers to check if their own answers are correct and review their mistakes. In other words, there is a “distribution of mentoring through peers” as other learners’ answers can serve as scaffolding in the learning process (CAST, 2018). Some participants noted that seeing others’ ideas encouraged them to think more, which is interesting as this unique form of “collaboration” somehow encourages creativity of learners. This is in line with research which has shown a link between the use of digital learning tools to foster collaboration and creativity (Aragon et al., 2009). Some participants also highlighted the “community” aspect of this exercise, as they loved to hear other learners’ responses and Formative allows learners to see all the responses.

However, not all participants liked this function. One participant pointed out that seeing all the responses on the screen would give him pressure because he might think that his answer might be wrong and he might not want his response to be seen. Another participant also pointed out that this seemingly “collaborative” feature may pose a peril to learning as learners may suffer from “groupthink”, as he said “when students see what others think they may lose their own ideas and become less creative.”

Discussion

While Formative provides features which fosters collaboration and community, the features may not be perceived as helpful by learners; in some cases, they may even cause pressure, which is an unintended outcome. However, it must be noted that whether seeing live responses of other learners facilitates collaboration and community largely depends on how teachers instruct the learners to interact with the tool. In the user testing, the participants’ perceptions of whether Formative fosters collaboration were limited by their experience of being shown other written responses while they were writing their captions. Other ways of instructions, such as explicitly requiring learners to discuss the responses on the screen before finalizing their answers, might lead to different perceptions of the potential of the tool in fostering collaboration and community.

c. increasing mastery-oriented feedback — Definition

The UDL framework evaluates whether a digital learning tool leads to mastery by providing “relevant, constructive, accessible, consequential, and timely” feedback (CAST, 2018). In the user testing, participants were asked about their views on feedback and Formative’s functions of auto-grading (by the tool) and batch-grading (by teachers).

Evidence

Some participants appreciated Formative’s ability to let teachers know how all students perform and give immediate feedback based on students’ answers. They pointed out that the immediate feedback allows them to see “how [they are] understanding what the lesson is teaching”.

Online reviews of Formative help illustrate teachers’ perspective on giving feedback via Formative (Bicknell, 2020; Formative | Facebook, n.d.; Lynch, 2018; MacNicol, 2019). Many teacher users praised Formative’s ability to assess students and provide feedback almost instantly, and noted that students enjoy the live feedback they get from the website. This is in line with the UDL framework’s focus on using feedback to help learners to “sustain the motivation and effort essential to learning”, based on the assumption that a lack of feedback on how to improve can “make some learners seem ‘perseverative,’ careless, or unmotivated” (CAST, 2018). Some teacher users specifically praised the tool’s batch-grading feature, but a few users noted that the website should have a built-in feedback bank of responses to help teachers give feedback to learners. Nevertheless, it can be seen that Formative has the potential to lead learners to achieve mastery through facilitating the feedback process.

Discussion

Based on the observation, it can be concluded that by providing timely feedback to learners, Formative helps learners learn how they can improve to achieve mastery.

(2) Provide multiple means of representation

a. providing the same information through different modalities — Definition

The UDL framework evaluates whether a digital learning tool includes different modalities to help teachers present a concept in different ways so that students can learn in ways that work best for them (CAST, 2018). The user testing has addressed this by including a matching question and an embedded YouTube video, which are “visual, non-linguistic supports” (CAST, 2018). In the user testing, participants were asked to match four captions with their corresponding pictures.

Evidence

While most participants found this question interesting, the user testing unveiled technical difficulties they faced when they attempted to complete this question, which is a surprising finding. One participant took two minutes to figure out how the matching mechanism worked, while another participant did not know how to match the pictures with the captions and asked for help. A participant, who kept enlarging the photos instead of moving them, said, “it is difficult and unconvenient [sic] for students to find out how to use [the matching function]”. While the matching question could provide visual information, the technical aspect made the learning experience not so enjoyable. A review of the participants’ responses also revealed that the images might not be helpful in helping students understand the captions, as more than half of the participants did not match all the captions with the corresponding pictures correctly.

The UDL framework also lists other “linguistic alternatives” such as videos as resources that digital learning tools may include to enhance inclusivity (CAST, 2018). A YouTube video on the definition of popular culture, which lasts 2 minutes, is included in the Formative exercise, and participants were instructed to watch the entire video before completing Questions 4 and 5. When asked about their feelings about the video, most participants noted that they did not enjoy the video and that the speaker spoke too fast, while some participants said that they did not see the need to watch the video. This highlights the fact that the inclusion of different modalities may not necessarily improve learning, and that a careful selection of digital materials is essential in order to enhance comprehension and motivation.

Discussion

While Formative provides multiple ways that allow learners to “perceive and comprehend information”, findings from the user testing show that this does not necessarily lead to better comprehension (CAST, 2018). Technical difficulties may lead to frustration, and an unsuitable selection of multimodal information may not facilitate learning.

b. providing information in a format that will allow for adjustability by the user — Definition

The UDL Framework examines if a digital learning tool provides options which allow for “adjustments for preferences” to cater to learners who “need more time to process information” (CAST, 2018). In the user testing, participants were asked about their views on the video watching experience.

Evidence

Comparing watching the embedded video favorably to watching a video played by the teacher in class, some participants highlighted the adjustability of the video player. They liked that they could watch the video twice, pause the video to complete a question relevant to the video, and replay important information in the video, especially if the speed is fast. These functions, thanks to the adjustability offered by Formative, enhanced their learning experience.

Those who are not in favor of the video player mentioned several issues with it, including the fact that they could not adjust the volume of the video and the fact that they could not adjust the size of the video player on an iPad. One participant who used his computer to access the Formative exercise attempted to open the YouTube video in a new tab only to see a couple of advertisements before the actual video, which he noted were distracting. Finally, a couple of students mentioned an intriguing fact: when a teacher plays a video in class which students have no control over, students need to be fully attentive, but if they can adjust the video player on their own devices, they may just skip the boring parts if they do not find the video attractive. This surprising finding points out that the adjustability of the information may be an issue in itself.

Some participants were also disappointed by the fact that Formative’s video player does not support closed captioning and subtitles. While the UDL framework encourages digital learning tools to “use text equivalents in the form of captions or automated speech-to-text (voice recognition) for spoken language”, the video player does not allow the display of captions (CAST, 2018). This was disappointing for the participants, many of whom found that the video in the Formative exercise was too fast for them to understand.

Discussion

While some participants appreciated the adjustability provided by Formative’s video player, some noticed the various issues it caused. The user testing also discovered a need for closed captioning and subtitles, which would provide information that would assist learners in acquiring knowledge, particularly for middle school learners of English as a Second Language.

c. activating or supplying background knowledge — Definition

The UDL framework examines if a digital learning tool anchors instruction by “linking to and activating relevant prior knowledge”, and gives examples such as visual imagery, concept anchoring, and concept mastery routines (CAST, 2018). In the user testing, the participants were asked about whether the embedded video supplied them background knowledge and whether they enjoyed seeing examples of popular culture submitted by other participants.

Evidence

After the video was played, a participant immediately said, “I don’t know why I needed to watch the video.” Other participants, while understanding that the video gave them an understanding of different aspects of popular culture, which helped prepare them for Questions 3 and 4, said the speaker spoke too fast in the video and they didn’t understand the video very well. A few participants plainly said that they were not interested in watching the video, and that they did not need to watch the video to complete Questions 3 and 4. This points out the problem of the selection of materials. While this has nothing to do with Formative per se, this does reveal that the inclusion of multimodal materials does not necessarily activate or supply background knowledge. In this user testing, the choice of the video was inappropriate, leading to the participants not finding the video useful.

However, the participants enjoyed seeing examples of popular culture submitted by other participants. Some images of popular culture submitted include the popular gaming platform Roblox, ice hockey, and the FIFA World Cup. A participant said that he enjoyed learning about new popular culture items from others through seeing images, which is “not possible in traditional lessons”.

Discussion

The findings reveal that while Formative attempts to facilitate the activation and supply of background knowledge by allowing teachers to insert different content types into a Formative exercise, with video being one of them, this may not necessarily activate or supply background knowledge. The selection of multimodal materials determines whether the materials help activate or supply background knowledge. However, the image submission question has unveiled the role learners can play in supplying background knowledge for other learners.

(3) Provide multiple means of action & expression

a. varying the methods for response and navigation — Definition

The UDL framework examines whether a digital learning tool offers “alternative means for response, selection, and composition” (CAST, 2018). The framework deems this important as learners differ widely in their capacity to navigate their physical environment (CAST, 2018). In the user testing, the participants were asked about their feelings about the matching question.

Evidence

Despite the technical difficulties some participants experienced, they mostly appreciated the enhancement in interactivity it has brought, because unlike traditional homework, which one participant noted to be “tedious”, a matching question on Formative exercise “won’t feel like a burden” because they are doing the question in a more interesting way. Those who used an iPad to complete the matching question, who were able to drag and drop options using their fingers, found the experience smoother. This echoes the UDL framework’s call for digital learning tools to include “alternatives to marking with pen and pencil” and “alternatives to mouse control” (CAST, 2018).

Discussion

The participants generally appreciated the interactivity of Formative, which allows different ways of responding to questions, even though one participant noted that the platform is “not for newbies” as it would take time for them to “learn how to use the platform”.

b. using multiple media for communication — Definition

The UDL framework evaluates whether a digital learning tool allows learners to “compose in multiple media such as text, speech, drawing, illustration, comics, storyboards, design, film, music, dance/movement, visual art, sculpture, or video” (CAST, 2018). The framework deems this important as it “increases the opportunities for all learners to develop a wider range of expression in a media-rich world” (CAST, 2018). In the user testing, participants were asked to complete the two free response questions which required them to upload a file to Formative.

Evidence

From a technical perspective, most participants found the file uploading easy on both a computer and an iPad. However, some noted that they could not drag and drop to upload on an iPad. One participant said that the interface was “a little bit confusing because [he] need[ed] to find the (+) symbol.” Another participant, who misunderstood the instructions, reported that he could not complete the question because he did not have a picture related to popular culture in his computer.

Despite the technical difficulties, many participants have praised Formative for allowing learners to express their ideas through media. They appreciated Question 4 for allowing them to hand in photos instead of just verbally expressing opinions in a traditional lesson, which “can be difficult”, according to a participant. A few of them have mentioned that this can be especially helpful in facilitating group projects in class, as teachers can receive the responses more easily. Some of them mentioned that seeing photos would allow them to learn more about a topic, such as “different aspects of popular culture” in the user testing.

Online reviews of Formative revealed that some teacher users also praised Formative for allowing learners to submit audio and video recordings as responses, which allow learners to express their ideas through alternative media (Bicknell, 2020; Formative | Facebook, n.d.; Lynch, 2018; MacNicol, 2019).

Discussion

Despite the slight technical difficulties faced by some of the participants, they all recognized the potential of using Formative to express their thoughts through multiple media. This facilitates the exchange of ideas, which can be useful in group projects and class discussions.

c. enhancing capacity for monitoring progress — Definition

The UDL Framework examines whether a digital learning tool facilitates the provision of formative feedback that “allows learners to monitor their own progress effectively and to use that information to guide their own effort and practice” (CAST, 2018). In the overall evaluation of Formative, the participants were asked about their views on whether the feedback provided by the platform helps guide their own effort and practice.

Evidence

Many participants pointed out that Formative allows all students to answer a question and this makes it “easy to compare students’ answers because the website shows all students’ answers very clearly”. When asked about what they would do when they see responses of other students, all participants noted that they would compare their answers to those of others. This underlines that Formative facilitates self-monitoring behavior by allowing learners to use the peer “feedback” to improve their answers and subsequently their understanding of the concepts.

Online reviews also echo the views of the participants. Formative’s ability to use scoring rubrics to auto-grade, batch-grade, and automatically return the scores to students is hailed by many teacher users (Bicknell, 2020; Lynch, 2018; MacNicol, 2019). By releasing scores to learners during the lessons, Formative enables the self-monitoring process by quantifying learners’ understanding of the lesson content with scores in a timely manner.

Discussion

Formative has shown promising results in facilitating learners’ self-monitoring process, especially in terms of providing timely feedback. Learners can also guide their own learning through comparing their responses with those of others.

Ratings

Formative will be rated on the degree to which it meets each criterion below. The rating system will be on a scale of 1-3 stars.

Three-star rating system

1. The tool is making effective use of technology’s affordances.

Definition

Donald Norman (1988), a renowned professor and the author of the landmark book The Psychology of Everyday Things, defines “affordance” as “design aspect of an object which suggests how the object should be used”. As the name of the tool suggests, the main purpose of Formative is to help teachers provide immediate “formative” feedback in class, which informs learning and teaching.

Rating & Evidence

Based on this definition, Formative receives 3 Stars in this criterion (★★★). All the participants appreciated Formative’s ability to give immediate feedback to learners, and they have pointed out many ways this is not possible without the help of technology.

One key difference between a traditional lesson and a Formative-assisted lesson is that all students need to answer the same questions in a Formative-assisted lesson. In a traditional Asian classroom, where a lesson can be as short as 35 minutes and there can be up to 40 students in a classroom, a teacher simply cannot ask every single student to answer the same questions. Due to the large student-teacher ratio, a teacher may simply allow most students to be silent in the entire lesson, which does not facilitate learning as students have no way of knowing if they are understanding the lesson content correctly. Some students may also be too shy to raise their hands to answer questions, and without answering questions, they will not be able to get feedback from the teacher.

However, Formative, which participants in the user testing have noted to be “convenient” and “user-friendly”, allows learners to answer questions and receive immediate feedback both by the auto-grading function of the platform and by the batch-grading function performed by the teacher. This greatly enhances the interactivity of a lesson, which many participants have acknowledged. One participant accidentally discovered the function of sending messages to teachers on Formative in real time by pressing the comment bubble, which he said was “convenient” and “fascinating”. The function is designed to allow teachers to clarify concepts instantly and privately, which echoes the spirit of “formative” feedback.

Another feature of Formative, which is to enhance learner engagement through the incorporation of interactive question types and multimodal materials, is also appreciated by all the participants. Despite the technical difficulties, all participants liked the matching question and the free response questions requiring file uploading, and they said this could make learning “more convenient” (through the matching question) and attract students to “enjoy the process of doing homework and tasks”. They enjoyed the autonomy to choose a picture they liked as their response, and noted that this allows learners to express their ideas more easily and learn more new things from other classmates. In short, Formative has made use of interactive Web technology to facilitate formative feedback and engage learners, which means it is making effective use of technology’s affordances.

2. The design of this tool suggests an understanding of the challenges unique to learning the target content.

Definition

Though Formative is a digital learning tool that facilitates learning and teaching rather than providing learning content, the current discussion will focus on whether the tool facilitates the learning of English as a Second Language. As discussed in the introduction, learners of English as a Second language generally have a low motivation in English learning, and therefore a successful implementation of this tool should tackle this challenge.

Rating & Evidence

Based on this definition, Formative receives 2 Stars in this criterion (★★). The interactivity of the tool is a highlight for many participants, who noted that traditional activities in English lessons, such as doing worksheets, reading the textbooks, and reading aloud, are boring. The word “interesting” has been repeatedly mentioned by almost all the participants, which is evidence that shows Formative does enhance learning motivation at least to some extent. One participant’s response sums it up best: “a Formative exercise won’t feel like a burden to [students] because they are doing questions in a more interesting way”.

However, the tool does not respond to the challenges that learners of English as a Second Language face adequately. One challenge is listening to an utterance in English by a native speaker, who usually speaks at a speed too fast for learners to comprehend. Whereas other digital learning tools such as Edpuzzle allow learners to turn on closed captioning or subtitles to enhance their comprehension of the utterance, Formative’s video player does not allow this function even when the original YouTube video comes with closed captioning or subtitles. This was noted by a few participants in the user testings, who found the video too fast and wanted to look for closed captioning to understand the native speaker better.

Another challenge Formative does not attempt to tackle is difficult vocabulary. Given the abundance of online free dictionaries, it should not be difficult for Formative to leverage resources and technology to help learners of English as a Second Language to understand difficult vocabulary. During user testing, participants struggled to understand some of the words in the Formative exercise, such as “first name”, “last name”, “captions”, “composition”, and “foreground”. While some of them decided to ask for the meaning of these words during the 1-to-1 user testing, they will not have such privilege when studying in a 40-student classroom. Difficult vocabulary being a possible hindrance to their understanding, learners may struggle to complete the learning activities and feel unmotivated or disengaged in lessons.

In short, while Formative supports learners of English as a Second Language in engaging in language learning, the platform does not support their specific language learning needs, which may in turn lower their language learning motivation.

3. The features of this tool provide benefits and reduce risks for your target learner.

Definition

To examine the benefits Formative provides, it is essential to revisit the features of the tool: receive multimodal information, answer teacher-set questions of a wide variety of types in real time, see live responses of other learners, and benefit from teaching informed by the responses (EdSurge Product Index, 2022). As for risks, Cambridge Dictionary defines “risk” (n.d.) as “the possibility of something bad happening”. In the case of Formative, risks may be defined as “adverse effects on the learning of middle school learners of English as a Second Language”.

Rating & Evidence

Based on the definitions, Formative receives 2 Stars in this criterion (★★).

As evidenced by the responses from the participants, the features of Formative do provide benefits for their learning. The participants noted that Formative “makes teaching more convenient” (e.g. by setting questions of a wide variety of types), “makes students pay more attention in class” (by including multimodal information), “attract[s] students to do tasks”, “allows teachers to see how students are understanding what the lesson is teaching”, “allows more students to express their thoughts”, and “increases students’ engagement and enhances their motivation in learning”. Through the UDL framework analysis, it can be seen how the features of Formative provide these benefits to students.

The user testing has revealed some possible adverse effects from the use of the platform. For example, questions of a wide variety of types may be technically difficult for students to complete and may therefore take more time to finish, compared to just writing down answers on a worksheet. Typing may be difficult for learners using Formative on an iPad, and the autocorrect/autocomplete feature provided by web browsers may reduce learners’ spelling ability. The ability to view responses of other classmates in real time may lead to new issues such as “groupthink”, copying of answers, and stifling of creative ideas. One participant, who had an overall negative opinion of Formative, said that the tool is “not useful in traditional lessons” because entering responses on the website “takes more time than writing answers on a traditional worksheet”. He further elaborated by saying that a traditional worksheet “can do all the functions that Formative provides and save the time for turning on iPad or computers”. Learners may not see the purpose of answering questions on Formative, especially if the questions are just replicates of questions from a textbook, and this can pose risks to their learning motivation. Having to use technology to answer these questions may be more time consuming for learners, which is also a valid risk. Finally, as shown by the overall negative response to the embedded YouTube video, the inclusion of multimodal materials may not necessarily translate to better instructional design, and good lesson objectives and designs are still important. It is the task of teachers to ensure materials of high quality and of suitable difficulty are inserted into a Formative-assisted lesson to maximize learning effectiveness.

Recommendations regarding this Learning Tool

Though Formative largely achieves its goal of promoting formative assessments through interactive web technologies, user testing and literature review reveal that there is room for improvement. This section will delineate recommendations for Formative from two aspects, Theory of Change and Enhancements, in the hope that the platform will become a more effective learning tool for middle school students learning English as a Second Language.

Theory of Change

Theory of Change provides a working model against which to test hypotheses and assumptions about what actions will best bring about the intended outcomes (Taplin et al., 2013). It defines long-term goals, changes that need to happen earlier, and identified measurable indicators of success as a roadmap to monitoring and evaluation (Taplin et al., 2013). Theory of Change models are useful in the development of digital learning tools because they articulate the specific interventions that will be used to try to achieve preconditions and long-term outcomes (Reinholz & Andrews, 2020).

In view of the potential risks discussed in the previous section of this report, a Theory of Change model is created by the author of this report (Figure 17). This model contains three sections: Learners, Experiences, and Outcomes.

While Formative does not have a specific group of target learners in mind, “Learners” in this model will be defined as middle school students learning English as a Second Language who have low motivation in completing language tasks. A special emphasis is placed on low motivation, as literature has revealed that this is a salient characteristic of this group of learners (Al-Hoorie, 2018).

The “Experiences” section contains the three aims of Formative as defined by the platform (EdSurge Product Index, 2022). Enhancements to the “experiences” proposed by the author of this report (underlined in the figure) have been incorporated into the model and will be discussed in the “Enhancements” section of this report.

The three “Outcomes” of the platform in the model correspond to the “Experiences” of the platform. Essentially, they are how Formative intends learning to be improved after using the platform. This section will discuss the measurement of these outcomes, which will help answer the question of whether the platform has successfully tackled the potential risks the targeted learners face.

Before evaluating the “Outcomes” of Formative, key assumptions should be discussed. Being a tool that promotes active participation in lessons, one key assumption is that all students have access to the platform and that they attempt all the questions in a Formative exercise. This entails that each student should have a computer or mobile device with Internet connection in a lesson, or else equal participation cannot be achieved. Teachers also need to monitor student participation in lessons through their completion of questions on the platform. Only by meeting these assumptions will Formative be able to lead its learners to the intended outcomes.

One way to measure the success of Formative is to assess whether students have better understanding of a topic. A direct goal of providing multimodal information through Formative is to equip students with a better understanding of a topic. In the user testing, a video on popular culture is included in the Formative exercise so as to familiarize the participants with the new concept of “popular culture”. This can be showcased in their responses to Formative questions related to the topic, their group work, and their ability to generate new ideas based on the new information. However, it has to be stressed that the choice of multimodal information to be included in a Formative exercise is important, as user testing has shown that unsuitable multimodal information does not improve learning.

Another way to measure the success of Formative is to assess whether students are more engaged in lessons through active participation. “Interesting” being an adjective mentioned by all the participants in the user testing, the interactivity of Formative should certainly take credit. No longer confined by question-and-answer, paper-and-pen formats in traditional lessons, students may find excitement in learning through answering questions of different types, which is made possible by web technologies. Nonetheless, “engagement” is often subjective and difficult to measure, and teachers may have to rely on observations to see if students have more participation in the learning process, such as the level of excitement, the quality of learners’ responses, and their attentiveness. At the very least, a successful Formative exercise motivates all learners to respond to all the questions in a Formative exercise with a reasonable amount of effort.

Finally, Formative should be evaluated by its ability to enhance learning by encouraging students to learn from one another. Instead of being told whether their answers are right or wrong, students may benefit more from comparing their answers to the answers of their peers and modifying their own responses, and this way of using contrasting cases in learning is proven to enhance learning effectiveness (Schwartz et al., 2011). Especially in giving long responses, students should demonstrate the ability to reflect on their responses and improve their responses by taking into account the other responses. To evaluate whether this function indeed enhances learning, teachers should look into how students modify their responses after viewing the responses of other students. While Formative does not provide such tracking functions, teachers can monitor the responses in real time to see if such changes are taking place in lessons.

Enhancements

Based on the opinions collected in the user testing, three suggestions have been included in the Theory of Change model to enhance the effectiveness of Formative in achieving its goals: video player enhancements, readability checks, and peer response evaluations (Figure 18).

Video player enhancements

One suggestion is to enhance its video player. As pointed out by participants, Formative’s own video player is lacking in many aspects, which hinders their learning. A new video player which allows volume control and provides closed captioning or subtitles will assist learners of English as a Second Language in comprehending the multimodal information, which will in turn enhance their understanding of a topic.

Given many participants’ concern about the function of skipping parts of a video which may be important, Formative may consider making it compulsory for learners to watch certain parts of a video at least once, similar to the function of Edpuzzle, which disables forwarding in a video playback. To help teachers gauge learners’ reaction to a video, Formative may also consider incorporating a simple feedback system, which prompts learners to choose how they feel after watching a video, similar to the “Classroom Climate” question in Peardeck (Figure 19). Learners may also give optional text feedback if they feel inclined to. Teachers may then decide if the chosen video is effective in helping students gain a better understanding of a topic.

Readability checks

A surprising finding from the user testing is that difficult vocabulary in the questions may hinder students from completing the questions, which may in turn affect their engagement in lessons. In the user testing, participants quickly grew disinterested in attempting a question when they did not understand certain keywords in a question. It is therefore suggested that Formative incorporates technology that provides more vocabulary support. For instance, students may click on any word in a Formative exercise and the platform can show the definition of the word immediately. Formative may refer to Toucan, a web browser extension which allows users to check the meaning of a word by hovering their cursor over the word (Figure 20).

On the teacher’s side, Formative may conduct a readability check before teachers publish their Formative exercises and suggest paraphrasing of difficult words so as to prevent the problem of difficult vocabulary in Formative exercises. Formative may refer to Rewordify, which helps simplify difficult English by highlighting difficult words and providing suggestions of simpler words to replace difficult words (Figure 21).

Peer response evaluations

Finally, while the function of showing all responses assumes that learners will automatically benefit from this feature, learners may not necessarily improve their learning just by seeing the responses of their peers. Therefore, there is a need to promote learning by prompting learners to evaluate peer responses. Formative may consider making references to the platform Short Answer and prompt learners to give feedback to their peers’ responses after they have answered a Free Response question (Figure 22).

In the case of learning English as a Second Language, it is common for teachers to ask students to explain their answers for a reading comprehension question or write a short paragraph on a certain topic. Guiding learners to give feedback to their peers, Formative may randomly show two responses for each learner and provide evaluative options such as “more detailed”, “more accurate”, and “better support of ideas”. In the process, learners may also compare their own responses to those of other people.

This practice is grounded in the approach of adaptive comparative judgement (Seery, Canty & Phelan, 2012). Research has shown that this learning approach has various benefits, such as enhancing the level of engagement, encouraging diversity and creativity in students’ work, and improving problem solving ability (Seery, Canty & Phelan, 2012). By requiring learners to engage in the responses of other learners, Formative will truly promote peer learning and encourage active learning.

Summary and conclusions

This report has described and evaluated the learning experience provided by Formative for middle school learners of English as a Second Language who have low motivation in language learning, as well as provided recommendations for improved use. Through the provision of multimodal information, a wide variety of question types, and live responses of learners, Formative seeks to improve student engagement, encourage discussion and deeper learning, and inform teaching. Key learning outcomes for learners include better understanding of a topic, more engagement in lessons through active participation, and enhanced learning via learning from peers; these outcomes are targeted through the use of the platform.

The goal of the report is to provide information about the utility and efficacy of Formative so as to help the Formative Team better support these learners in learning various aspects of the English language and develop new features to address the challenges faced by English language instructors and learners. Certain features of Formative are viewed as beneficial to the learning experiences of English learners with low engagement in lessons, such as the image submission function which promotes learner autonomy by allowing learners to upload an image of their choice. While the structure and design of Formative enhances learner engagement in lessons, the platform offers insufficient support for the challenges learners of English as a Second Language face, such as vocabulary support.

It is hoped that by reading this report, the Formative Team will be made aware of English learners’ need to overcome challenges in comprehending videos and texts in English and consider the enhancements proposed in this report to meet this goal. The Formative Team should also explore the possibility of promoting peer learning and encouraging active learning by incorporating the approach of adaptive comparative judgement. If Formative invested in fostering collaboration and community, there would be more value in the platform for educators. Peer learning is often not employed in lessons due to time constraints, so having a digital tool that helps students learn from others’ responses would be useful. A function which allows learners to evaluate random peer responses can be integrated into Formative so that students are learning from their peers while also monitoring their own learning progress.

Despite the shortcomings, Formative still largely achieves its goal of promoting formative assessments through interactive web technologies. It is hoped that the Formative Team continues to focus on enhancing learner engagement in lessons through the functions provided by the platform, as high motivation in learning would be beneficial not just for learners of English as a Second Language, but all students, who inevitably face moments of disengagement and demotivation in learning. By providing ample opportunities for learners to answer questions and different ways to engage in learning content, Formative would enable educators to achieve the ideals for learning, where interactions and peer collaboration takes precedence over teacher instruction.

References

Aragon, C. R., Poon, S. S., Monroy‑Hernández, A., & Aragon, D. (2009, October). A tale of two online communities: Fostering collaboration and creativity in scientists and children. In Proceedings of the 7th ACM Conference on Creativity and Cognition (pp. 9–18).

Al‑Hoorie, A. H. (2018). The L2 motivational self system: A meta‑analysis. Studies in Second Language Learning and Teaching, 8(4), 721–754.

Bicknell, K. (2020, July 8). 7 reasons you need GoFormative in your digital classroom. Retrieved November 15, 2022.

CAST. (2018). Universal Design for Learning Guidelines (Version 2.2). Retrieved from http://udlguidelines.cast.org

Curriculum Development Council. (2017). CDC English Language Education Key Learning Area Curriculum Guide (Primary 1–Secondary 6). Education Bureau.

Dixson, D. D., & Worrell, F. C. (2016). Formative and summative assessment in the classroom. Theory into Practice, 55(2), 153–159.

EdSurge Product Index. (2022, May 26). Formative | EdSurge Product Index. Retrieved October 4, 2022.

Eppard, J., & Rochdi, A. (2017). A framework for flipped learning. International Association for Development of the Information Society.

Formative. (n.d.). Formative | Real‑Time Formative Instruction. Retrieved October 4, 2022.

Fowler, K., Windschitl, M., & Richards, J. (2019). Exit tickets. The Science Teacher, 86(8), 18–26.

Gilakjani, A. P., Ismail, H. N., & Ahmadi, S. M. (2011). The effect of multimodal learning models on language teaching and learning. Theory & Practice in Language Studies, 1(10).

Lujan, H. L., & DiCarlo, S. E. (2006). Too much teaching, not enough learning: What is the solution? Advances in Physiology Education, 30(1), 17–22.

Lynch, M. (2022, June 18). Product review of Go Formative. Retrieved November 15, 2022.

MacNicol, M. (2019, March 4). GoFormative: Digital formative assessments. Retrieved November 15, 2022.

Masry‑Herzallah, A., & Stavissky, Y. (2021). The attitudes of elementary and middle school students and teachers towards online learning during the corona pandemic outbreak. SN Social Sciences, 1(3), 1–23.

Niemi, A. M. (2007). What are effective strategies to support student engagement and learning? [Doctoral dissertation, Evergreen State College].

Norman, D. A. (1988). The Psychology of Everyday Things. Basic Books.

Oliver, R., & Mackey, A. (2003). Interactional context and feedback in child ESL classrooms. The Modern Language Journal, 87(4), 519–533.

Pazilah, F. N. P., Hashim, H., & Yunus, M. M. (2019). Using technology in ESL classroom: Highlights and challenges. Creative Education, 10(12), 3205.

Risk. (n.d.). In Cambridge Dictionary. Retrieved from https://dictionary.cambridge.org

Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103(4), 759.

Reinholz, D. L., & Andrews, T. C. (2020). Change theory and theory of change: What’s the difference anyway? International Journal of STEM Education, 7(1), 1–12.

Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). The ABCs of how we learn: 26 scientifically proven approaches, how they work, and when to use them. W. W. Norton & Company.

Seery, N., Canty, D., & Phelan, P. (2012). The validity and value of peer assessment using adaptive comparative judgement in design driven practical education. International Journal of Technology and Design Education, 22(2), 205–226.

Taplin, D. H., Clark, H., Collins, E., & Colby, D. C. (2013). Theory of change: Technical papers (a series of papers to support development of theories of change based on practice in the field). ActKnowledge.

Wood, J. W. (2022). Understanding classroom engagement for middle school students (Doctoral dissertation, The University of Mississippi).