cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Community Member

SIS Import - How much time it takes to load ~50k rows?

I am doing SIS Import calls via API in our TEST environment. I recently tested a file with 20k users and it took more than 6+ hours to load and process that file. I was informed by Canvas that TEST and PROD environments dont behave the same and things will be faster in PROD. 

My question is - how much time does it usually take to load and process a 50K record file in Batch mode in PROD? 

8 Replies
Highlighted
Community Coach
Community Coach

sutaria@wustl.edu, due to the technical nature of this question I'm going to share your question with the Canvas Developers‌ group in the Community. They are the ones that do this type of programming stuff with Canvas and should be able to help. In addition, you might consider joining this group so you have access to their resources and information!

Hope this helps!

Kona

Highlighted
Instructure
Instructure

Meera,

Imports of data into Canvas have many factors that can affect the time it takes for the files to process. Is there a particular reason that you are using Batch mode? If you are looking for both speed in processing and the ability to delete users not in the new file, I'd recommend using Diffing mode.

Diffing mode compares the file currently running against a previously run file with the same diffing ID and only runs the differences (additions and deletes) in the files (documentation). I work with an K12 organization that provides over 2.5 million enrollment file each night and it takes less than an hour to run (depending on the amount of changes made).

Highlighted
Community Advocate
Community Advocate

I just checked our last enrollments import in production and it was 327,000 records in batch mode and it took 2.5 hours.

Highlighted

@Matt - Thank you very much, this helps a lot.

Highlighted

About the same as Matt during regular semesters, 2-3 hours depending on whether I'm importing the employee data (200K records) or students, courses and enrollments (500K+ records).

Highlighted

@Robert Thank you very much for providing this detail

Highlighted
Adventurer III

I'll add some more datapoints: we upload ~100k users each night in batches of 10k. It takes about 5 minutes to process each batch of 10k records. 

--Colin

Highlighted
Community Coach
Community Coach

Hello there sutaria@wustl.edu...

I have been reviewing older questions here in the Canvas Community, and I stumbled upon your question.  While I don't really have any feedback for you, I wanted to check in with you because I noticed that there hasn't been any new activity in this topic since June 5, 2018.  It looks like you've gotten some feedback from Community members and Instructure employees.  Do you feel that one of the above answers could be marked as "Correct"?  Do you have any further questions as it relates to your original question?  If you feel that one of the above replies has helped to answer your question, please click on the "Mark Correct" button next to that person's reply.  However, if you are still looking for some help, please let us know that as well.  For the time being, I am going to mark your question as "Assumed Answered", but that won't prevent you or others from posting additional questions and/or comments below that are related to this topic.  I hope that's alright with you, Meera.  Looking forward to hearing back from you soon.