SIS Imports - Show Robust Errors and Human-Friendly logs

Idea created by Joni Miller Expert on Dec 8, 2015
    Radar
    Score143
      Idea is currently in Product Radar Learn more about this stage...

     

      Idea open for vote Wed. January 6, 2016 - Wed. April 6, 2016  Learn more about voting...

     

     

    This is based on idea posted by Chris Sweets that got 52 votes and 13 comments:  Robust Errors for SIS Import

     

    Problem:  The SIS Import History is not in a Human-Readable Format

    For all SIS Imports, even if 100% successful, the following data needs to be included: 

    • Upload Time
    • Begin Time
    • End Time
    • Import completion time (in hours/minutes/seconds) - not currently listed
    • Number of records imported
    • Imported Items (Courses, sections, users, enrollments, etc)
    • Status:  Completed, Imported with Messages, Failed, etc.
    • Reformat logs for previous SIS Imports into a human-readable format.
    • Include a link to the error logs for previous SIS Import batches

     

    Problem:  There is no easy way to see details about SIS Import Errors - information is vague

    Example: We had a problem with our users file.  It has >30,000 rows and we had a student with a quote in her name "Denise", which caused the file to fail and not even load.  We would get a "malformed CSV" error.  This was REALLY hard to find in our file and took probably 5 hours of trial and error to find the line with the error since we didn't know which line or what could be causing the malformed error.

     

    Example: Problem file that kept uploading via our automated process and we didn't realize that it was causing all of the other files to back up and we ended up having to call and have an engineer cancel all of our SIS imports because we had probably 20 files in the queue.  It was a mess because we have no way of canceling on our side, and even support and our CSM can't do it.  It took several hours for the engineer to cancel everything in the queue.

     

    Example From Marissa Zelmanowicz

    I feel this feature is MUCH needed, as we frequently get long lists of 60+ errors stating "canvas_enrollments.csv - An enrollment listed a section and a course that are unrelated" and have no idea what course and enrollment it is referencing. Each time this error appears, hours upon hours of digging is required to try to fix the issue. Since we have thousands of entries coming from our SIS each night, this is a very time consuming and frustrating process.

     

     

    If an error is thrown, the following data in a READABLE, non JSON format would go a long way to helping admins figure out the problem and get us back to being productive. 

    • File Name of CSV File that had error
    • Type of import (courses, sections, users, enrollments, etc)
    • The line number of the data file that caused the error in the .csv should be included.
    • Field with Error (if possible)
    • Type of Error (invalid character is the one we usually run into)
    • Status:  Did it skip the line and move on or did it prevent any further records from importing?

     

     

    Would be nice: 

    • Add automated email options for errors and warning.  This will allow admins to keep an eye on automated imports.
    • Have the ability to email the warnings that happen during an import to an admin.  Especially when these imports are automated and/or take hours, it's important to know about any errors.
    • Ability to search for imports from a certain date or date range or Job ID #
    • Ability to see all past history without having to do an API call

     

    2015-06-04_9-14-50.png

     

    Format the error logs for previous SIS Import batches in a readable format.  We call the log page the "ugly page" at Seminole. 
    PLEASE have the import status of prior SIS imports available in a HUMAN-READABLE format!

    2015-06-04_11-04-54.png

    Thank you!

     

     

      

      Comments from Instructure

     

    March 2016 update from Linda Feng

    Hi - thanks so much for these suggestions.  I wanted to let you know of several error reporting improvements were recently implemented for SIS imports:

    • Added: user_id, login_id of failed user if available when encountering the following error: "unique_id is invalid".
    • Added: section_id, course_id if available when encountering the following error: "user XXX didn’t exist for user enrollment".
    • Added: Errors contain user_id if available when encountering the following error: "neither course XXX or section XXX existed for user enrollment"

    The goal is to make it easier to troubleshoot errors raised during imports.

    We will log the additional suggestions for consideration for future enhancements to SIS imports.

    Thanks - Linda

     

    *We will be archiving this thread, as it is not something we plan to build completely in the next 6-18 months.  Please follow this idea to receive updates as they are available.