SIS Imports - Show Robust Errors and Human-Friendly logs

(2)
This is based on idea posted by Chris Sweets that got 52 votes and 13 comments:  Robust Errors for SIS Import

 

Problem:  The SIS Import History is not in a Human-Readable Format

For all SIS Imports, even if 100% successful, the following data needs to be included: 

  • Upload Time
  • Begin Time
  • End Time
  • Import completion time (in hours/minutes/seconds) - not currently listed
  • Number of records imported
  • Imported Items (Courses, sections, users, enrollments, etc)
  • Status:  Completed, Imported with Messages, Failed, etc.
  • Reformat logs for previous SIS Imports into a human-readable format.
  • Include a link to the error logs for previous SIS Import batches

 

Problem:  There is no easy way to see details about SIS Import Errors - information is vague

Example: We had a problem with our users file.  It has >30,000 rows and we had a student with a quote in her name "Denise", which caused the file to fail and not even load.  We would get a "malformed CSV" error.  This was REALLY hard to find in our file and took probably 5 hours of trial and error to find the line with the error since we didn't know which line or what could be causing the malformed error.

 

Example: Problem file that kept uploading via our automated process and we didn't realize that it was causing all of the other files to back up and we ended up having to call and have an engineer cancel all of our SIS imports because we had probably 20 files in the queue.  It was a mess because we have no way of canceling on our side, and even support and our CSM can't do it.  It took several hours for the engineer to cancel everything in the queue.

 

Example From Marissa Zelmanowicz

I feel this feature is MUCH needed, as we frequently get long lists of 60+ errors stating "canvas_enrollments.csv - An enrollment listed a section and a course that are unrelated" and have no idea what course and enrollment it is referencing. Each time this error appears, hours upon hours of digging is required to try to fix the issue. Since we have thousands of entries coming from our SIS each night, this is a very time consuming and frustrating process.

 

 

If an error is thrown, the following data in a READABLE, non JSON format would go a long way to helping admins figure out the problem and get us back to being productive. 

  • File Name of CSV File that had error
  • Type of import (courses, sections, users, enrollments, etc)
  • The line number of the data file that caused the error in the .csv should be included.
  • Field with Error (if possible)
  • Type of Error (invalid character is the one we usually run into)
  • Status:  Did it skip the line and move on or did it prevent any further records from importing?

 

 

Would be nice: 

  • Add automated email options for errors and warning.  This will allow admins to keep an eye on automated imports.
  • Have the ability to email the warnings that happen during an import to an admin.  Especially when these imports are automated and/or take hours, it's important to know about any errors.
  • Ability to search for imports from a certain date or date range or Job ID #
  • Ability to see all past history without having to do an API call

 

54703_2015-06-04_9-14-50.png2015-06-04_9-14-50.png

 

Format the error logs for previous SIS Import batches in a readable format.  We call the log page the "ugly page" at Seminole. 
PLEASE have the import status of prior SIS imports available in a HUMAN-READABLE format!

54702_2015-06-04_11-04-54.png2015-06-04_11-04-54.png

Thank you!

55 Comments
cms_hickss
Community Contributor

 @millerjm , when they said "a one time error" did they mean for your institution (as a whole) or just for creating that single user? I'm guessing they meant for just that user and that it could still happen again?

millerjm
Community Champion

Susan,

I don't know what caused it or if it was one user, the file, or maybe a random system problem.  The file ran just fine the next time it ran. I couldn't get any additional information.  They got the information that they got me from searching for previous tickets with the same error.   I tried to get them to have someone from the SIS team get back to me with more information but they said no.  

Joni

james_evans
Community Explorer

One admin feature of Moodle I appreciated as an LMS admin was the "history" of import logs. I would like to see more than just the previous processing in addition to the detailed reports.

lars_vemund_sol
Community Contributor

James Evans skrev:

One admin feature of Moodle I appreciated as an LMS admin was the "history" of import logs. I would like to see more than just the previous processing in addition to the detailed reports.

If you add an s to your sis import URL and have json formatter extension in chrome you get a pretty good history of previous imports: https://community.canvaslms.com/thread/8313

Not very good UI but it helps many times Smiley Happy

james_evans
Community Explorer

Thanks, Lars. Even though I have talked with my CSM about this they never mentioned this trick.

JIM EVANS, M. Div., M.A.

Sr. Applications Support Analyst

T 714.619.6508 F 714.668.6196 vanguard.edu<http://vanguard.edu/>

Learner * Responsibility * Restorative * Individualized * Context

millerjm
Community Champion

We had this problem continue to occur with the SIS Users file.  The SIS team made a change that they said should fix the problem, but it's not in any sort of release notes.  Not sure what the change was or if it may have adversely affected other users.  So far it hasn't happened again. 

tross
Community Champion

I made a comment to my CSM this week.  We also have Canvas Gauge and it has separate imports.   They recently added import history to Gauge and it is awesome.   You can see the history, get a summary and download the log from past imports.   Don't know how far back it will go, we now have 51 but that is how many days since they implemented the feature.  

I would love to see that in base product.  

sbeck1
Community Champion

We are receiving this error also. When I contacted support about the error that states to contact support, I was told that it was running okay now, so not to worry. That doesn't work for us. If an enrollment is missed, it doesn't automatically run again. Our import system thinks the student was successfully imported.

Joni, how often do you run imports?

millerjm
Community Champion

 @sbeck1 ‌, we run imports every 4 hours.  I had to open a few tickets to have them get an engineer to look at it.  They made some sort of change, but of course these code changes related to SIS are RARELY documented so we never know to be on the lookout for problems. 

jsavage2
Community Contributor

Hi  @millerjm ,

I've spent a lot of time tracking down CSV errors, and it seems like Canvas uses the standard ruby csv gem, so the errors--and list of available characters--are inherited from that project. Trying to come up with rules for quoting special characters was a path to madness.

For me, the simplest solution was to use a library to process my files. The python, ruby, and perl csv libraries all produce compatible quoted and escaped CSV with minimal effort (unlike Excel, which seems to live in its own world), especially with their respective "quote all" flags turned on.

arunkumar_ayyav
Community Novice

HI team,

am facing canvas malformed csv while my csv files update automated.

imported file with the messages

error that  prevent to import 

canvas_term.csv-molformed csv

canvas_account.csv-molformed csv

canvas_entroll.csv-molformed csv

please assist us to resolve this one

Stef_retired
Instructure Alumni
Instructure Alumni

 @arunkumar_ayyav  Welcometo the Canvas Community! Please post your question to the Canvas Developers‌ group to get assistance from experts (to post directly to the group, join it from the Actions dropdown on the group home page).

daniel_savlon
Community Explorer

Also casting a vote for a system that notifies us whenever the automated SIS import has a warning or error. Bad characters were preventing my users.csv from going up, and I had no idea.

Code-with-Ski
Community Participant

If anyone is looking for an easier to read log of SIS Imports, it is one of the features in the Chrome extension that I created. Canvas LMS Mods (Basic) The log is embedded at the bottom of the SIS Import page and makes it easy to filter by the workflow state so you can find ones with errors easier.  It also lists the errors and warnings in the summary when available.  There is also a link to the JSON representation of the SIS Import if there are other details that you need that aren't shown in the log.sis import example.png

I posted a blog article about the Chrome extension that I am developing as a side project: Creating a Chrome Extension to Use with Canvas LMS 

ProductPanda
Instructure
Instructure
Status changed to: Archived
Comments from Instructure

As part of the new Ideas & Themes process, all ideas in Idea Conversations were reviewed by the Product Team. Any Idea that was associated with an identified theme was moved to the new Idea & Themes space. Any Idea that was not part of the move is being marked as Archived. This will preserve the history of the conversations while also letting Community members know that Instructure will not explore the request at this time.