Teaching World Languages in a Digital Environment: “Scramblation”

Translation as a Strategy

Using translation with beginner language students is fraught with controversy. When I was in elementary school, the contemporary teaching methods for modern languages were phasing out from “grammar-translation” toward more communicative approaches. Translation came to be seen as antiquated, impractical, unnatural.

It cannot be denied that some criticism of translation, especially for beginners, is valid. I cannot help reflect on some brilliant students I have had over the years who, by French III, had become held back by their insistence on mentally translating everything before they wrote or spoke. Their focus on the rules, the words, the syntax, the burden of feeling like they must not err, all conspired to leave them tongue-tied and frozen whenever they were called upon to improvise speech or writing.

Nonetheless, I find in my experience that there is a place for translation in novice language lessons. Students should learn the ways that the target language differs from their own so that they gradually learn to avoid applying the syntactical patterns of their own language. The also need to be able to discern morphological differences in the target language that may be slight to their eyes but which meaning can vary significantly. Finally, it is a good way for novices to learn the longer, whole functional phrases that are a part of the earliest stages of learning before grammar has been taught to let them synthesize their own utterances.

Barriers to Using Translation to Teach Novices

Limited vocabulary is the first barrier to using translation to teach novices. In the textbooks at the start of the 20th century, each chapter had a very controlled vocabulary that was repeated in reading and translation exercises. Many of us no longer teach that way. I teach through theme units. The unit has a lot of vocabulary but the higher order language work is not limited to that as a controlled vocabulary list. Narratives and authentic texts, even listening practices, while selected with difficulty in mind, do in fact include words and structures the student may not yet have been taught. The advantages of this approach are well known and it is common practice now. Among other things, the student learns the very functional skill of deriving meaning from context, selectively ignoring incomprehensible utterances in favor of the meaningful, and perhaps learning new words from context.

The second barrier to teaching with translation is, naturally, grammar. Good grammar exercises that use translation have to be very controlled to account for irregulars and inconsistencies that most language boast of. At the very early stages, novices has so little grammar under their belt that translation may not prove worthwhile. Or, the cognitive load of balancing all the rules will render the exercise useless for its purpose.

Here is What I Needed

I needed an app that would auto-correct and let students try again when they made errors. I needed limited vocabulary and limited grammatical competence to be largely irrelevant. I needed an interactive activity where students manage the syntax and recognize correct forms. I call the new app “scramblation”. It is a drag-and-drop interface where students assemble an utterance in the target language from a prompt that is either in text form or audio clip.

Translation plays a pivotal role in the process of studying a foreign language, serving as a valuable tool for language learners to bridge the gap between their native tongue and the target language. It offers learners a nuanced understanding of linguistic structures, idiomatic expressions, and cultural nuances, thereby facilitating a more profound comprehension of the language’s intricacies. Translating texts from the target language to one’s native language and vice versa enhances vocabulary acquisition, grammar proficiency, and overall language competence. It enables learners to decipher the meaning behind words and phrases, fostering a deeper connection to the cultural context embedded within the language. Moreover, translation exercises encourage critical thinking and analytical skills, as learners must carefully consider the nuances of each word and construct coherent and contextually accurate sentences.

A New App

Instructors can generate a new scramblation from the playlist of their course in Innovation. They enter a prompt, the correct answer, and some extra words. I link to use the extra words to enter un-conjugated verbs or words an English speaker might put in that would not go in the target language.

The prompt can be an audio clip (in which case the text prompt is hidden) and can include an image.

Students can see the task in their playlist and access to a scramblation can be made possible from a link in the lesson plan app or an external link that instructors can send to students.

The app itself is simple: first, students should remove any extra words by clicking the small red “x” in the word’s box. Next, the student drags and drops the words into the right order. They save their answer, check it, and the algorithmic AI will tell them how close they are.

Like all of the apps at Innovation, the scramblation has a proctor activated that tracks student activity on the page, including when they leave the page and how long they were working.

The importance of interactive web applications in the realm of remote teaching cannot be overstated. Interactive web applications emerge as powerful catalysts for student engagement, collaboration, and personalized learning experiences. The ability to seamlessly integrate multimedia content, real-time communication, and interactive assessments not only enhances the effectiveness of teaching but also empowers educators to adapt their pedagogical approaches to the diverse needs of their students.

Teaching World Languages Remotely, Part 3: Vocabulary Building

The Importance of Interactivity

The key drawback to early efforts at distance learning was being kind of trapped behind that camera like a goldfish in a bowl. You could make all the signs and signals you wanted, but the world on the other side of the glass was beyond your ability to control.

Teaching remotely is not highly effective when it consists of essentially just holding up things to the camera for the student to experience. Activate the Zoom – Skype – Meet – Teams session, share your PDF, give verbal instructions… this is a weak instructional practice mainly because it is largely passive for the student.

If the teacher were in a real classroom, tutoring the student at an honest-to-goodness table, the learning materials could be manipulated in real life in ways that support the process. They can fold the paper to hide the answer, they can shuffle the flash cards, they can write and cross out and scribble and erase. The manipulation of the learning materials is important.

The apps at Innovation are designed to promote the kind of virtual interactivity that heightens the effectiveness of teaching remotely. To be a great learning experience, the remote session needs to be virtually interactive in the same effective way that in-person lessons are. This is a big part of what we mean by the “21st century learning space”.

Flashcards

Let’s take up the example of teaching vocabulary using flashcards. In real life, I would want to use a process whereby I selectively show the student a new word, rehearse the pronunciation in some meaningful way, then cue up the words to rehearse the meanings.

Using the passive approach, I could share a PDF through the video conference software and “go over” the list with the student.

Using the flashcard app at Innovation, I can interact so much more effectively. To begin, I can select the target vocabulary word to display.

I prompt the student to repeat the pronunciation, then click to reveal the meaning.

Once we are through the list, I can repeat the process, only this time I can save out those items the student forgot.

Now we are only drilling those items. We can talk about mnemonic devices, use the words in sentences, or just repeat and rehearse. Once the student has the words down pat for recognition, I click Reverse Cue-Response to prompt from English cue.

Integrated Flashcard App

The improvised conversation app and the scaffold dialogue app both have integrated flashcards. During an improvised conversation task, the student may need to ask me how to say some words as we run through the conversation the first time. I list them for them in the textarea below the prompt.

So long as I pair the new phrases with an equal sign and a meaning, the app can generate a flashcard system right underneath after our conversation.

We can rehearse now the new words and phrases before we perform the dialogue once again.

Interactivity is Key

Being able to interact virtually over remote teaching sessions in ways that are as effective as in-person is absolutely necessary to achieve a satisfying learning experience that maximizes our effective use of time. The flashcard app at Innovation facilitates this process of simple cue-response training that is so foundational in teaching language. It allows me to go beyond just sharing my screen to “go over” a PDF!

Teaching World Languages Remotely, Part 2: Composition

Teaching composition in a world language is always challenging to organize and execute. In my experience, the best lesson series in supporting the development of strong composition skills consists in the following:

  • Students should have limited access to outside resources in composing their work. It’s too tempting, especially now, to use an AI translator.
  • Students should learn to avoid translating in their head from English to the target language. Instead, they should learn to “say what they can say, not what they want to say.”
  • Assessment should provide a clear and understandable measure of the value of the work product and a clear path to remediation for next time.

I started teaching in 1991 (I am now retired). Back then as a French teacher, the method for assessing student compositions involved marking off each clause, identifying each error, and checking whether the clause was comprehensible (to a native speaker accustomed to dealing with foreigners), appropriate (such that it built on the theme coherently; it “fit”) and had good form (no more than 1 error in grammar / conventions). This was abandoned in the later 1990’s for a rubric that was more consistent with other New York State Regents examinations of the time.

I think the only thing I like about the rubric assessment was that it considered the variety of vocabulary used. Otherwise, the rubric did not really satisfy what I wanted in an assessment for composition work and this rubric was far more subjective than I was comfortable with.

Teaching remotely, I wanted an app that met my criteria for supporting composition skills in the target language.

The first challenge was to limit the use of outside references. For this, I coded a sort of algorithmic surveillance AI that I called “proctor”. Proctor consists of a series of JavaScript functions that record when a student has resized a window, pasted text, “left” the page, or restarted the task. These actions are saved and reported to me when I assess the students’ work.

In the remote teaching situation that I enjoy at present, students do their compositions unsupervised for homework. The proctor allows me to curtail student access to other tabs because it announces in red text on the page whenever any of these “suspicious” actions occur. Although students may indeed use their phone separately on the side to confer, I can also check later in our debriefing by asking whether they know the leaning of one phrase or another.

The composition app for world languages allows me to present students a word bank. The word bank can be an antidote to mental translating because students can be taught to weave together meaning from words they have rather than get caught up on words they don’t.

The assessment process in the composition app works as follows. The instructor:

  1. marks off the clauses for evaluation.
  2. highlights each error.
  3. assesses each clause for comprehensibility, appropriateness, and form.
  4. assesses the whole composition for vocabulary “richness” (10% of the score).

This process lends itself well to debriefing because the errors can be studied directly and are readily observed.

A 21st Century Learning Space

The composition app is a 21st century learning space.

Training wheels are temporary assistive devices for young people learning new things. They are a modification to the program that is usually temporary; a scaffolding that brings students upward in the zone of proximal development. The composition app has space for a word bank to support composition from known lexical items.

Guardrails are there to protect us from error, safety features along the road at dangerous points to avoid a pitfall. The composition app includes an algorithmic AI to monitor student activity and discourage assistance that would not be appropriate.

21st century learning spaces lend themselves to debriefing: they are designed such that the anonymous presentation of teacher-selected student work is easily generated for debriefing. The composition app is readily shared with the student and the assessment page is clear and easy to understand. When I debrief these, I paste the student composition into another screen and go over the relevant errors.

21st century learning spaces are a Swiss army knife. Such collections of applications serve many functions from the same core. The composition app saves the composition prompt in a database whose elements can be re-used.

21st century learning spaces are those where the teacher rules the roost and student privacy protection is a high priority. Locus of data control is with the teacher. The teacher can view the composition as it’s being composed and has ownership over the final product.

Teaching World Languages Remotely, Part 1: Rapport Building

Since September, I have had the distinct pleasure of working part-time for a company based in California that offers remote middle and high school credit-bearing courses in world languages. LanguageBird is perfect fit for a retired public school teacher and I am very contented working for them (not a paid promotion).

The pandemic placed we public school teachers in the position of teaching remotely, some for the first time. A lot of that went poorly in some places, but in other places it went pretty well. My work teaching remotely now has given me the chance to re-explore online teaching practice and the kinds of 21st century learning spaces that meet the needs of that situation.

Besides my work for LanguageBird, I also am enjoying teaching a remote French class for the public school district from which I retired last June. This is very different from LanguageBird in many ways and teaching in both contexts has provided a wealth of interesting experiences that I feel are instructive. In this series of posts, I would like to share my experiences and conclusions as well as the apps I am developing to support remote teaching.

In the public school remote teaching context, we had set it up as a daily synchronous class. This was informed by our pandemic experience that asynchronous courses are a bad idea for most adolescents. It is a small class of five, two for French III and three for college French, credited from a local community college who approved my application to work as adjunct for them in a high school. Each school day during period 2, I fire up a Google Meet and students log in. They are supervised by a language teacher (Spanish).

We (administration and I) were concerned that remote teaching made it difficult to maintain the kind of teacher-student rapport that was so necessary for learning. I suggested that I work in-person for a half day at the end of each marking period (ten weeks) to teach a class and meet with students individually so they can present their projects and practice French conversation. (The district is a 45-mile commute for me one-way, so going in-person for one daily class was not practical.)

At LanguageBird, we only teach one-on-one lessons. I find this extremely useful, so from start I modified my public school lesson plans such that I would only teach whole-group for the first 15 minutes and then each student would have an individual “tutorial” with me for the balance of the time. This turned out to be a fantastic idea and I am guessing the students like it too.

During the pandemic in my district, we had two days to launch into teaching by video-conference (Here is a post on my experience teaching during the pandemic). My current students, many of whom were then in my sixth grade social studies class back in 2020, had a mostly negative experience learning online in general. I felt strongly motivated to demonstrate from the start of the school year that this remote learning experience would not be like that. The first upgrade I made to what I was doing in 2020 was to focus on individual lessons over group lessons.

I think of positive rapport as being a trusting sense of mutual goodwill between an instructor and a pupil. Building a positive rapport with students is extremely important. I had the sense that this was possible only to a very limited degree in remote learning. However, I now stand corrected. In remote instruction over video-conferencing, it is necessary to favor one-on-one teaching situations.

Fostering positive rapport extends not to just being present to interact one-on-one. It is also built on online software applications that foster efficient and readily accessible learning interactions for delivery, practice, evaluation, and debriefing.

Next post: teaching composition to world language students remotely.

Innovation … What’s in a Name?

InnovationAssessments.com brings to mind a testing service. And so it once was!

Was…

But the pandemic spurred its growth toward the full online teaching platform that it is today!

Not to disparage our educational testing apps. This platform started out twenty-odd years ago as a test generator for multiple-choice tests. The test generators, test question bank management apps, secure online testing, and algorithmic AI-assisted scoring of short answer tests and summaries makes it a powerful tool in your teacher’s toolbox.

But…

While we kept the name (Do you realize how hard it is to rebrand a website?), we are more than our name! I invite the reader to explore the collection of apps that makes this a top-notch teaching platform.

For starters, Innovation evolved under the demands of real teachers and students in real classrooms. As my colleagues and students shared suggestions for apps and upgrades over a decade, I modified and adapted the software. My coding students were assigned to try to hack it; I built defenses. My teacher colleagues had lots of ideas for ensuring assessments integrity. We collaborated to built something reliable and intuitive to use.

It is difficult to compose a promo for Innovation because by the time we list all the features, we have lost the attention of the reader.

Lesson Planning

The first thing that I did not like about Google Classroom was how clumsy it was to communicate assignments to kids. I don’t like the comment stream approach. And what about students who are ahead and want to see what tomorrow holds? The planning app at Innovation is the first thing students see when they navigate to their course playlist. They see this week and all the assignments. They can jump to them from the planner or scroll down the playlist. There is a custom note option just for today’s lesson.

Curating Resources

It is important to me to be able to effectively curate my resources; my links to assignments on and off Innovation. I want to be able to hide things until the time is right, lock tests with a key code, schedule the visibility of tasks to the best moment, and so forth. Innovation possesses all of these features and more for efficient curating of class online resources. Link to Google docs and website of any kind, manage who has access and when, hide unit plans for next school year.

But it is also easy if you curate your resources elsewhere!

Many subscribers to Innovation curate their links at Google Classroom or other platforms. Well, most use Google Classroom… But that’s okay! Innovation is a verified app on the Google system. You can embed a link to your Innovation task on Google classroom and after a quick authentication app, students are engaged in the day’s lesson.

Proctor

The proctor at Innovation is an algorithmic AI that monitors and reports on student activity. My colleagues and I wanted to know whether students were really watching our video assignments or whether they were pasting in text in some places; we wanted to know how long it took students to do a task and when they logged in and from where. Proctor AI gives detailed information about what students are doing online in your 21st century learning space.

Study Apps

The Tutor app and the flashcard system are perfect for drilling terms and facts.

The algorithmic AI coach advises students on short answer, summaries, and outlines based on a corpus of models on which they were trained.

Modifications are easily made to drills to accommodate special needs.

Innovation has what your students need to study and what teachers need to scaffold their objectives to individual needs.

The Etude

The Etude is our favorite app. Teachers embed a lesson in video, PDF, and/or audio format and include guiding questions. There is note-taking space for students. The Etude is the perfect tool to curate deliverables online in 1:1 laptop classrooms or in remote teaching situations.

Merit Badges

Manage merit badges to chart progress. Invent your own or import ours. As students earn new skills, automated badge awards provide visual evidence of progress.

Ventura!

Students love playing this Jeopardy-like game. It is easy for teachers to generate games from their test question banks. Suitable for in-person or remote learning situations, the Ventura game is a hit with students.

Why not try it out!

Look, I know we’re a small startup company, unknown and not really able to compete with Google, Scoology, and the big names.

But…

This platform has a lot going for it.

Managing Student Accounts at Innovation

In response to teacher requests over the years, there are a number of different ways to add student sub-accounts to your virtual classroom at Innovation.

Self-Registration

It’s best to just have students register themselves in your virtual classroom. There is a link provided on the Students tab in your teacher dashboard. The links are on the left.

One link is generic, allowing students to create sub-accounts using a password or using Google Sign-in.

One link is specific to Google Sign-in. Send this one if no one is using username-password. You can always switch later if you want.

Import Student Roster from Google Classroom

If you are integrating Innovation tasks into your Google Classroom, you can import the rosters.

Adding Students Manually

Teachers can just add students manually using the Add Student app. The app will create a unique username from the student’s first and last names. You can optionally assign students to a class. More about enrollment below…

Adding Students via Spreadsheet

Teachers can upload a spreadsheet of student data and let Innovation create the accounts from there. Teachers who do this must use the XLSX template provided.

Expiration Dates on Student Accounts: Commercial Licenses versus Free Licenses

Teachers who have a commercial license to Innovation can manage expiration dates on student accounts. This is useful, for example, if you are working for a tutoring service and you are charging students to access your virtual classroom for a certain period of time.

Teachers with Free Licenses do not have expiration dates on student accounts and need to manage them manually.

When a new student is added for commercial accounts, it is by default created for two days at first. Teachers need to add time. Manage student account expiration dates in Students :: Student Account Expiry.

How Students Access Innovation

There are three ways students can log in to Innovation:

  • Google Sign-in
  • PIN + Virtual Room Number
  • Virtual room number + username + password

Google Sign-in

If you are using Google classroom and plan to integrate Innovation tasks into that platform, this is your best choice. Students will already have Google accounts and they can sign in without remembering an extra password.

PIN

Personal Identification Numbers (PINs) are for students under 13 because students must be 13 or older to use the Google Signin or username-password methods. But you can use this for any of your students. Teachers need to activate the PIN system and can optionally allow access this way only from certain IP addresses. For example, I have set this up so that my students could only use the PIN from my school.

Username + Password

The traditional way to log in also still exists at Innovation. Students need to know your virtual classroom number, their username assigned on account creation, and a password. The teacher controls the passwords.

The Quick Login Link

If a student cannot log in because they have forgotten their password, you cans end them a quick login link. This link will automatically log students in without a password, Google authentication, or PIN. It only works once and the link is only good for two hours.

Manage Course Enrollment

Teachers can now restrict access for students to certain courses in their virtual classroom at Innovation. By default, enrollment is “open”, meaning students can freely access any course.

For teachers who need to restrict access by students to a certain course, there is an app to manage enrollment. Access the app from Students :: Student Accounts Management :: Manage Course Enrollment. From here, follow instructions to restrict enrollment and then assign students to a class.

Commercial licensees will find this useful if they are charging for access to a course and need to prevent free access to courses for which students have not paid.

Successful Students Share Their Secret for Online Learning

In mid-May 2020, we were finishing up 2 months of remote learning during the pandemic. I conducted a study to find out what I could learn from the students who were very successful learning online remotely.

Twenty-one respondents to a survey asking successful online learners to report on the “secrets” of their success collectively present a profile of the student who will likely do well in asynchronous distance learning conditions. These students are very self-directed, seldom needing much parent intervention or supervision. Most like working online because there are fewer distractions than in school and they can work on their own schedules and at their own pace. These students have a special place set aside for doing school work and mostly do their school work in one sitting rather than sporadically through the day. These students are not necessarily very academic-oriented in temperament and may not even prefer online learning because they miss their friends and teachers. When asked to advise their peers, common suggestions include ideas like planning out the working and break or recreation time, keeping checklists, and self-motivation strategies.

Besides answering my questionnaire, 14 of 21 respondents accepted my invitation to make suggestions for their peers about their secrets for online success. Their comments are as follows:

  • I think that it easier to get all of your work done during the same time because then that way you can have the whole afternoon to do whatever you want.
  • As the Nike logo says “Just Do It”! I try not to put stuff off, however I wish it will be over soon so we can have summer and do other productive things with my family.
  • i have a system that i follow and i check all classes my email old emails at least 19 times a day
  • I try to give myself some time in the morning to wake up and have some time to myself, like an hour, then I start my day and work until lunch most times which I then take another hour or two to rest, finally I work until I am finished with little breaks and end most of the time right before dinner. I would say just try to get it done early then you could look forward tp having the rest of the day and if you get ahead then at the end of the week you could possibly have friday off, like I do. Also, just try and not get distracted and if you need to tell your siblings/parents/guardians you need quiet, my mom has learned that she can’t talk to me when I am doing school.
  • Something that motivates me is when I can take a 5-10 minute break between each subject. I use my RC car for this. While it is charging (it usually takes 45 minutes to charge) I do some work, and when it is done (it has a 10 minute run time) I go out and drive it.
  • I find that it helps to have a list of what I have to do and when they are due. This helps me to prioritize and not stress out as much about my work. I also tend to do my work in the morning. This way I have time to do my work and I can get it all done early. If I forget about an assignment, this also allows me to do it before it is due.
  • i just think that after i do all my work i can go out an do anything i want the rest of the day so i use that to motivate me
  • having parental involvement keeps me on task or i wouldn’t stay on task. My parents also checking power school regularly. I do struggle because i’m not getting as much assistance as i would during school.
  • Make sure to hunker down and just do your schoolwork. try to follow pretty much the same schedule every day and not get into a mindset of “I have all day to finish”, because chances are you’re just gonna keep on putting it off.
  • Well about the distractions. The main distraction I have at home and not school is food. Now that I am at home there are lots of food breaks.
  • I think a schedule is really important. Not only does it limit the amount of distractions in the day, but allows you to get through your work without missing anything or falling behind. I was home schooled before I came hear, and sometimes it’s nice to set apart time where you can watch a show or a play a game or something, that way you don’t feel as inclined to take a break in the middle of your work. That’s all the advice I got! 🙂
  • ‘m getting better grades doing the online learning, but I don’t really like it because I’m not getting the same interactions with teachers and friends that I can get when I’m physically in school.
  • For me, I do a few hours of school work in the morning and then a few hours of it in the afternoon. I always take about an hour or two for a break in between those times. That break is very nice, and relaxing. I either go for a walk, or try to do another activity that is not school related. I find if I do not take that break, I get too overwhelmed. Questions 2 and 3: My parents check up on me, to see how I am doing. But they do not watch over me. Also, my parents trust that I am getting all the school work in on time, so they do not enforce too many rules, because I stay on top of it myself.

Developing Semi-Automated Evaluation of Analysis in Secondary Student Writing

Introduction

Fully functional and reliable automated “AI” grading of essays is a long way off yet and well beyond the computing capability available in typical secondary school classrooms. However, useful steps in that direction are well within reach, particularly for working within the domain of limited vocabulary and composition skills that constitute the typical proficiency level of students in grades six through twelve. High school social studies teachers in New York State assess student essays using a grading rubric provided by the State Education Department. One dimension of this rubric is to assess the relative degree of “analytic writing” versus descriptive. Students whose essays are more analytical than descriptive have a work of greater value. The artificially intelligent grading program at InnovationAssessments.com estimates the grade of a student writing sample by comparing it to a number of models in a corpus of full credit samples. With a view to developing an algorithm that better imitates human raters, this paper outlines the data and methods underlying an algorithm that yields an assessment of the “richness of analysis” of a student writing sample.

Measuring “Richness” of Analysis in Secondary Student Writing Samples

The New York State generic scoring rubrics for high school social studies Regents exams, both for thematic and document-based essay1, value student expository work where the piece “[i]s more analytical than descriptive (analyzes, evaluates, and/or creates* information)” (Abrams, 2004). A footnote in the Generic Grading Rubric states: “The term ​create​ as used by Anderson/Krathwohl, et al. in their 2001 revision of Bloom’s Taxonomy of Educational Objectives​ refers to the highest level of the cognitive domain. This usage of create is similar to Bloom’s use of the term synthesis. Creating implies an insightful reorganization of information into a new pattern or whole. While a level 5 paper will contain analysis and/or evaluation of information, a very strong paper may also include examples of creating information as defined by Anderson and Krathwohl.”

Anderson and Krathwohl (2002), in their revision of Bloom’s Taxonomy, define analysis thus:

4.0 Analyze – Breaking material into its constituent parts and detecting how the parts
relate to one another and to an overall structure or purpose.
4.1 Differentiating
4.2 Organizing
4.3 Attributing

One of the ways that students analyze is to express cause and effect relationships (Anderson and Krathwohl’s “4.3 Attributing”). It is possible using natural language processing techniques to identify and examine cause and effect relationships in writing samples using lexical and syntactic indicators. Taking a cue from the New York State rubric, one could judge that a student writing sample is more “richly analytical” if it “spends” more words on cause and effect proportionate to the entire body of words written.

Identifying and Extracting Cause and Effect Relationships using Natural Language Processing

With regard to identifying cause-effect relationships in natural language, Asghar (2016, p. 2) notes that “[t]he existing literature on causal relation extraction falls into two broad categories: 1) approaches that employ linguistic, syntactic and semantic pattern matching only, and 2) techniques based on statistical methods and machine learning.” The former method was selected for this task because the domain is limited to secondary level student writing samples and they use a limited variety of writing structures. Previous work studying this issue yielded better results in domain-specific contexts (Asghar, 2016) and tagging sentences containing cause-effect relationships in this context should be within reach to a high degree of accuracy.

The software is written in Perl. The following process is applied to the student writing sample for
analysis:

  1. The text is “scrubbed” of extra consecutive spaces, HTML tags, and characters outside the normal alphanumeric ASCII range.
  2. The Flesch-Kincaid text complexity measure is calculated.
  3. The text is “lemmatized”, meaning words that have many variations are reduced to a root form (i.e., “is, am, are, were, was” etc. are all turned to “be”; “cause, caused, causing” etc. are all turned to “cause.”)
  4. The text is “synonymized”, meaning words are changed to a single common synonym. The text is separated into an array of sentences and all words are tagged by their part of speech.
  5. A variety of lexical and syntactic indicators of cause-effect are used in pattern matching to identify and extract sentences which include a cause-effect relationship into an array.
  6. The resulting array of cause-effect relationship sentences are converted into a “bag of words2” without punctuation. Stop words are removed. All words are “stemmed”, meaning variations on spelling are removed.
  7. Finally, both the original text and the array of cause-effect relationships are reduced further to a bag of unique words.

At this point, the computer program compares the bags of words. The resulting percentage is the proportion of unique words spent on cause-effect out of the total number of unique words. Recall that these are “bags of words” which have been lemmatized, synonymized, stemmed, and from which stop words have been removed.

Limitations of this Method

There are ways to express cause-effect relationships in English without using lexical indicators such as “because”, “thus”, “as”, etc. For example, one could express cause and effect this way: It was raining very heavily. We put on the windshield wipers and we drove slowly.

“Putting on the wipers” and “driving slowly” are caused by the heavy rain. There are no semantic or lexical indicators that signal this. There are many challenges dealing with “explicit and implicit causal relations based on syntactic-structure-based causal patterns” (Paramita, 2016). This algorithm does not attempt to identify this kind of expression of cause-effect. Prior research in this area has shown limited promise to date (Mirza, 2016, p. 70).

Cause-effect is only one way to analyze. Differentiating (categorizing) and organizing (prioritizing, setting up a hierarchy) should also be addressed in future versions of this software. A student could compose a “richly” analytical piece without using cause-effect, although in this writer’s experience cause-effect is the most common expression in writing of people in this age group.

Analyzing a Corpus of Student Work

The New York State Education Department provides anchor papers for the Regents exams so that raters can have models of each possible essay score on a scale of one to five. Anchor papers are written by actual students during the field testing phase of the examination creation process. Sixty such anchor papers were selected for use in this study from collections of thematic and document-based essays available online at the New York State Education Department website archive (​https://www.nysedregents.org/GlobalHistoryGeography/​). Thirty came from papers identified as scoring level five and thirty scoring level two. Essays scoring five are exemplary and rare. Papers scoring two are “passing” and represent the most common score. Essays are provided online in PDF format. Each one was transformed to plain text using GoogleDrive’s OCR feature. Newline characters were removed as was any text not composed by a student (such as header information). This constitutes the corpus.

The computer program analyzed each sample and returned the following statistics: number of cause-effect sentences found in the sample, the count of unique words “spent” on cause-effect relationships in the whole text, the count of unique words in the entire text, the percentage of unique words spent on cause-effect, the seconds it took to process, text complexity as measured by the Flesch-Kincaid readability formula, and finally a figure that is termed the “analysis score” and is intended to be a measure of “richness” in analysis in the writing sample.

An interesting and somewhat surprising finding came in comparing the corpus of level two essays to those scoring a level five. There was no real difference in the percentage of unique words students writing at these levels spent “doing” analysis of cause-effect. The mean percent of words spent on cause-effect relative to the unique words in the entire text was 46% in level five essays and 45% in level twos. There were no outliers and the standard deviation for the level fives was 0.9; for the level twos it was 0.13. Initially, it seemed that essays of poor quality would have a much different figure, but this turned out not to be the case. What made these level two papers just passing was their length and limited factual content (recall that analysis is only one dimension on this rubric).

Text complexity is an important factor in evaluating student writing. The Flesch-Kincaid readability formula is one well-known method for calculating the grade level readability of a text. In an evaluation of the “richness” of a student’s use of analysis, text complexity is a significant and distinguishing feature. The “analysis score” is a figure intended to convey that combination of text complexity and words spent on cause-effect type analysis. This figure is calculated by multiplying the percentage of unique words spent on cause-effect by 100, and then multiplying by the grade level result of the Flesch-Kincaid formula. This measure yielded more differentiating results. In order to discover ranges of normal performance based on these models, the following statistics were calculated for each data set: lowest score (MIN), first quartile(Q1), median(MED), third quartile(Q3), and highest score(MAX).

If this corpus of sixty essays can be considered representative, then the ranges can be considered standards in assessing the richness of secondary level student analysis in a writing sample. These figures can be used to devise a rubric. On a scale of one to four where four is the highest valued sample, the following ranges are derived from the combined statistics of all sixty essays:

Incorporation of Cause-Effect Assessment into AI-Assisted Grading

The artificially-intelligent grading assistance provided subscribers at InnovationAssessments.com, to date, estimates grades for student composition work based on a comparison of eleven text features of the student sample from a comparison with the most similar model answer in a corpus of one or more model texts. In cases where expository compositions are valued higher for being “analytically rich”, incorporating this cause-effect function could refine and enhance AI-assisted scoring.

Firstly, the algorithm will examine the most similar model in the corpus to the student sample. If the analysis score of the model text is greater than or equal to 419, then it is assumed analysis is a feature of the response’s value. In this case, an evaluation of the “analytical richness” of the student’s work will be incorporated into the scoring estimate. Samples that are more analytical will have greater chances of scoring well.

Conclusion

An artificially intelligent grading program for secondary student expository writing that includes an evaluation of the richness of analysis in that text would be very valuable. Cause-effect statements are indicators of analysis. The algorithm described here identifies and extracts these sentences, processes them for meaningful analysis, and judges the quality of the student’s analysis with a number which incorporates a measure of the proportion of words spent on analysis and text complexity. An analysis of sixty samples of student writing yielded a range of scores at four levels of quality for use in artificial grading schemes. While this algorithm does not detect all varieties of cause-effect relationships nor even all types of analysis, its incorporation in already established artificial scoring programs may well enhance the accuracy and reliability of the program.

Sources

Abrams, D. (2004). Revised Generic Scoring Rubrics for the Regents Examinations in Global History and Geography and United States History and Government (field memo). Retrieved from http://www.p12.nysed.gov/assessment/ss/hs/rubrics/revisedrubrichssocst.pdf​ .

Asghar, N. (May 2016). Automatic Extraction of Causal Relations fromNatural Language Texts: A Comprehensive Survey. Retrieved from ​https://arxiv.org/pdf/1605.07895.pdf​.

Krathwohl, D. (2002). ​A Revision of Bloom’s Taxonomy: An Overview.​ Retrieved from https://www.depauw.edu/files/resources/krathwohl.pdf ​.

Mirza, Paramita. (2016). Extracting Temporal and Causal Relations between Events. 10.13140/RG.2.1.3713.5765 .

Sorgente, A., Vettigli G., & Mele F. (January 2013) ​Automatic extraction of cause-effect relations inNatural Language Text.​ Retrieved from ​http://ceur-ws.org/Vol-1109/paper4.pdf​ .

21st Century Learning Spaces: Asynchronous Discussion Forum

My first experience with asynchronous discussion forums came in courses I was taking myself online through Empire State College a number of years ago. Many readers will recognize the assignment: given a prompt, students are to post their response and then reply to the responses of a number of other students in the class. Typically, there was a deadline by which these discussions had to take place. I liked the exercise and I found it useful to address the course material.

I would invite the reader to read my earlier post on synchronous chat, which presents some of the research on online discussion and chat.

Promoters of asynchronous discussion forums point out rightly that this task brings greater participation than face-to-face class discussions do. Whereas in the latter situation, participation may be dominated by an extroverted few or limited in other ways, the online forum brings everybody in. Asynchronous discussion leave time for research and reflection that is not practical in the face-to-face class. There are some practical considerations for students at the middle and high school level that are not usually issues at the college level.

My Experience

I used asynchronous form discussions in my middle and high school social studies classes for a decade. This occurred in each unit of student. In my context, students were assigned a persuasive prompt to which they were expected to take a position and post two supporting reasons. Next, they were assigned to present the opposing view to another student (even if it did not match their actual personal views), and finally they were to defend their original position in reply to the student who was assigned to present the opposing view to themselves.

Sample 7th Grader Exchange

Seventh and eight graders needed training right off the bat, naturally. Accustomed to social media, their early contributions were vapid and full of emojis and “txt” language. It was important to remind them that this was a formal enterprise and that standard English conventions held. It was often difficult to get them to elaborate their ideas toward the 200-word goal set for their opening post.

Not the kind of thing I as looking for!

I was working in a small, rural school where I would have the students from grades seven through ten, so I could see their work develop over the years.

By end of 9th grade, posts became more sophisticated

I found it to be a good practice to offer the highest marks to those who provided evidence and cited a source. I coded a citation generator right in the forum app to encourage this.

Grading the Posts

Scoring these can be labor intensive for no other reason than the layout of the forum itself. The page is designed for reading and responding, but this does not work well for scoring because there is too much scrolling and searching necessary to view posts and replies.

The scoring app makes it easy for the teacher to view the rubric, the student’s posts, and their replies to others in one place. Analysis tools lets the teacher see how many posts, when they were made, and even the readability level of the contributions.
My early discussion grading rubric.
The grading rubric I adopted later on.

Practical Issues

The main problem I encountered in this assignment was that students would forget to complete it at first. I resolved this by assigning it in class and giving time. For example, on the first day I would present the prompt and instruct students to post their positions that class period before continuing with the day’s other work. The following day, students would have time to post their replies and finally a third day they would post their defense.

Another issue that came up was getting everyone the needed number of replies. Some posts would attract more replies than others. Some students needed a reply so they could offer defense. The solution was to modify the assignment and declare that, once one has posted, one is obliged to offer the opposing view to the person above in the forum feed.

Interestingly, these assignments also led to face-to-face spontaneous class discussions, sometimes with me and sometimes with a group. Although this may have been somewhat distracting for students in the class working on other things, we found some compromise time to allow these spontaneous interactions to proceed without disrupting the other work much. These were golden opportunities, conversations of enormous educational benefit that are so hard to artificially initiate and encourage.

I came to regard the discussion each unit as a sort of group persuasive writing effort. I included training in grade eight in persuasive writing and logical fallacies. The discussion app here at Innovation has a feature which allows readers to flag posts as committing a logical fallacy.

The Innovation Discussion Forum App is a 21st Century Learning Space

  • Guardrails: The app lets the teacher monitor all conversations and to delete problematic ones.
  • Training Wheels: The teacher can attach a grading rubric and sample posts. I used to post first under a pseudonym to whom the first student could reply. Additionally, weaker students can peruse the posts of stronger students in an effort to get a clear picture of the kinds of opinions that can be had on the issue.
  • Debriefing: Debriefing is easily achieved by projecting the discussion screen on the from board. Students posts in this task are not anonymous.
  • Assessment and Feedback: The scoring app is very efficient and highly developed from years of use. The teacher can view all pf the student’s posts and replies without having to scroll across the entire platform. Analysis tools reveal the readability of the text, how much they wrote, how analytical it is.
  • Swiss Army Knife: The discussion app lends itself well to more in-depth persuasive writing assignments such as an essay.
  • Locus of Data Control: The student chat submissions are stored on a server licensed to the teacher’s control. Commercial apps such as FaceBook and Twitter may be less dedicated to the kinds of privacy and control exigencies of education.

Ideas in Closing

Asynchronous discussions are great – my students and I enjoyed these tasks. It is my view that higher level thinking demanded by persuasion and debate (Bloom’s evaluation level of the cognitive domain) really enhance long-term memory of the content. I cannot emphasize enough the value of these kinds of higher-order task. Working in a 21st century learning space promotes the participation of everybody.