Canvas X: On-Track Predictor, August 2018

Community Team
Community Team
6 18 1,104

Canvas X is where we seek your input on experiments from our secret pandaworks lab!

It's all about starting small, measuring, and learning our way forward!

 

The Volunteer Window has closed.  Watch the CanvasX space of the Canvas Studio‌ for future summaries and opportunities.

 

Based on the feedback from our June-August experiment of the On Track predictor we have decided to make some changes to the model and try another small cohort. We are looking for 3-5 instructors who are interested in providing qualitative feedback about the usefulness and the accuracy of the On Track predictor. The ideal candidates are instructors who:

  • Are willing and available to give direct feedback about their thoughts on the tool semi-regularly (once every two weeks or so)
  • Make use of Canvas widely to teach their courses (use of multiple features like assignments, modules, discussion etc.)
  • Have a course starting in August/September 2018
  • Have a background in statistics or probability (not necessary, just a bonus)

 

For the rest of the community, we would love to open up a dialog about what an On Track predictor (let’s call it OTP) means for Canvas. So what is the OTP and how does it work?

 

What is it? The OTP is a tool to help instructors discover which students are in danger of doing poorly in the course. Every week instructors will receive a report of all of their students that contains:

  • Are they On/Off Track?
  • How confident are we?
  • What are the primary reasons for the prediction?

The intention is that this will enable instructors to intervene with their students earlier and drive better student outcomes.

 

What does it do? The OTP evaluates a student’s likelihood to succeed in a course based on:

  • Academic History: How have they performed overall in courses previous to this one?
  • Course Interactions: How much do they interact with course tools? (grades, messaging, etc.)
  • Content Interactions: How much do they interact with course content (modules, files etc.)
  • Course Performance: How are they performing in their current course?

Using this information, a probability of successfully completing the course is calculated.

 

Given that brief overview we would love to hear your thoughts. How could you foresee using this? What information would you want/not want to see? What are the types of actions you could take to intervene with students? What’s missing?

 

And if you have more questions about the specifics of how the OTP works, members of the team who worked on the project will be monitoring the comments and will make their best effort to answer your questions. So fire away!

18 Comments
Instructure
Instructure

Hey Everybody! My name is Greg and I have been working on the On-Track Predictor for the last couple of months. I will be actively monitoring the comments here so ask all your burning questions!

Community Member

I just filled out the survey and I hope to be able to help this fall. This is an amazing concept and would help me track the progress of my remedial math students in my class. Excellent work on getting to this point and I can't wait to see where this goes from here.

Jesse

Community Coach
Community Coach

Pick me!! Pick me!! 🙂

I meet all four criteria, including the Stats part!! 

Community Member

kona@richland.eduand I are on board already. You now only need to find a few more willing instructors and you will be set. Kona can lead and I will bring up the rear!

Jesse

Navigator

I started to fill out the form, but gave up after the first qualifying question -- do you use Canvas to teach? -- felt it was too personal and didn't hang around to see where the rest would head.

Normally Kona begs me to sign up since I'm teaching, but she's back to teaching this fall and I figured her class is  while mine are face to face, and it's more accurate (according to what we learned at InstructureCon) with  classes.

Community Coach
Community Coach

I think it would be interesting to see how it would work for your classes compared to mine since it's the same course (Math 113), but mine is  and yours are traditional/face-to-face classes. 

Community Coach
Community Coach

I'm good with that! 😉

Community Advocate
Community Advocate

I hope that I can join the group as well this fall. I am glad to see that the ethics of prediction are part of the conversation and hope that no students are exempt from being helped no matter how hopeless the prediction is. A human touch for the failing is not a wasted touch.

I look forward to having a series of options in my toolkit to give each student based on their prediction score: encouraging words for the likely successes, specific interventions for the marginal, and compassion for the forlorn. All of it should have an encouraging message, of course, and give each student a specific resource and suggestions for just enough stretch. The interventions could range from links to static resources in and outside of a Canvas course to exercises that help refresh a student's memory of how to do something to prompting students to select from a list of times to meet with their instructor or student services professional. And of course it should be timely so that students are able to achieve the best outcome for their situation in that class.

Finally I wonder how the academic history for individual students is going to be included. Assuming that the developers are able to access my institution's Canvas instance to see how students have done in the past, there is still the problem that not all instructors use the grade data from Canvas equally in determining the official grades for their classes. Our SIS would be a better place for academic history in general, but we are already having trouble making effective use of (or even getting to) that data. Perhaps OTP will be the project that breaks us out of our torpor.

Instructure
Instructure

Thanks for sharing your thoughts. I love the idea of sending students studying resources or appointment links, making that work would be an interesting challenge, but I could see it being really valuable and high impact. 

With regards to past performance you are absolutely correct. We use Canvas grade data, which as you have noted often times does not match the true grade in the SIS. Getting access to the SIS would absolutely make a more accurate model, not just for true final grades but declared major, amount of transfer credits etc., that we don't have access to right now. Doing a small test with an institution willing to share their SIS data would allow us to determine whether the added value is worth the development time, however we have not found an institution willing to do so yet.

Part of the reason for doing a small pilot again this Fall is to try some new ways of managing problems like this. Currently the model uses Canvas grades and tries to predict is a student will fall below 70% at the end of the semester. One of the strategies we want to explore is to use percentiles of grades in a course so that we aren't relying on absolute Canvas grades, but relative grades. This way we could reframe the problem to trying to discover which students are most likely to be in the bottom 25%, hopefully making our reliance on Canvas grades less of a problem.

Community Advocate
Community Advocate

Thank you for your helpful response. I'm glad to see how open Instructure is at openly involving its users in projects like this. There are vendors clamoring for attention in the rapidly growing space of predictive analytics; it will be awesome when the developer of tools that are closest to actual teaching and learning is the source of solutions in this area.

Instructure
Instructure

One specific question the team has been pondering is the importance of root causes. The OTP will obviously provide a prediction whether a student is likely to succeed in the course or not. We can also share the two (or more) most prominent "reasons" why a student was predicted the way they were. Examples being things like "Got a low score on a recent assignment" or "Not active in social aspects of course (discussion etc.)". Is information like this useful to you as an instructor? What would you do with information like that?

Community Coach
Community Coach

Deactivated user, YES, this information is VERY important. Why? Because it would help direct the conversation I would have with the student.

Emailing a student and saying, "Hey, I think you're at risk of not passing my class." VS "Hey, I noticed you got a really low score on the last assignment and also haven't been as active in the class. Are you having difficulties with the assignment? Is there anything I can do to help? I'm worried about you being able to pass the class." Are two totally different conversations and the second one is going to have way greater impact on the student and the conversation that's had.

Yes, the Instructor could go looking for this information if they knew the student was at risk, buy why make the Instructor waste extra time trying to figure out why the student is at risk if Canvas can just tell them. In addition, if I know the student is having problems in specific areas then I know where to check into - so I know how to utilize my time most effectively. Ex: The student didn't do well on the last assignment. An Instructor could then go look specifically at that assignment and see if the score was because the student didn't submit it, didn't try very hard, or tried but just didn't understand the content/material. 

This root cause information is very important and NEEDS to be included. 🙂

Kona

Community Advocate
Community Advocate

Yes, that information would be great to have. As an instructor I would be able to target my intervention in a way that is tied to the source. For example, if that recent low score was on a writing assignment I could refer the student to the campus writing lab. Or if it's inactivity then I could refer that student to my instructions on how and why to use discussion, perhaps the counseling office, etc. In any case it would at least help me begin the conversation: "I notice that you..." For those students who are in defined cohorts and at institutions where this type of communication is supported, it also would help me as an instructor reach out to the faculty and staff associated with that cohort program and let them know.

Community Member

Hi Greg, Will OTP work in self paced courses, where students begin at different times and are completing the course on their own unique timeline? It sounds like the criteria you are using (academic history, interactions with course and content, and performance) might fit this type of course, although the way you use grades may be an issue.  I'm also wondering how OTP would handle students who are on the course roster but have finished the course already (and therefore are not interacting anymore).

I understand that your pilot tests should run using traditional fixed date courses, but I'd like to see how we might use them for our self paced courses once you have the details worked out.  Do you mainly publish results on this forum, or are there other ways to stay in the loop? Both Nudge and OTP are very much needed, and I look forward to hearing about your progress.  Also, please let me know if you are ever interested in testing in a self paced course environment. Thanks!

Instructure
Instructure

Sharon,

With regards to self-paced courses its a bit of a tricky situation. The way that all of the criteria is calculated should carry over, but one big things we rely on is how far a course has progressed. For fixed-date courses this is simply a function of time passed. For self-paced this is likely some function of # of assignments remaining (or something similar). I would prefer to use a method similar to that for all courses, unfortunately many Canvas courses do not create their assignments ahead of time, or use assignments or quizzes at all. So I think we may need to think about the way we do self-paced courses a bit more. I will reach out to you directly and maybe we can figure out a way to test some ideas on how to handle self-paced.

We do not publish results in the community (yet). We want to respect the data privacy of the participating students and teachers. However, we will be reaching out to some of our participants towards the end of the experiment and see if they are willing to share their results with everyone. So stay tuned!

Instructure
Instructure

Alison from the Canvas X team here.

Thank you so much for your interest and comments above! Sign-ups for this experiment have been great. If you have already provided us with the appropriate links and course start/end dates you can expect to get an email soon.

If you have haven't signed up yet or know someone that may be interested in signing up now's the time to act.

We will be closing the sign up on Monday August 20th.

Community Team
Community Team

The Volunteer Window has closed.  Watch the CanvasX space of the Canvas Studio‌ for future summaries and opportunities.

Community Member

They have good ideas and can hold attention only the  uninterested are uncertain.

About the Author
I have spent most of my career applying data science to problems in the healthcare space. Education is a fun change of pace!