AnsweredAssumed Answered

Anyone using the Submissions API for incremental loading of a Data Warehouse?

Question asked by Glenn Auerbach on May 8, 2015
Latest reply on Jun 24, 2018 by Glenn Auerbach

At the University of Michigan, we have some classes with a LOT of students, which means a LOT of submissions. For example, we have a class with (round numbers) 2,000 students and 65 assignments/quizzes - 130,000 submissions by the end of the term. We acquire data from the Submissions API and load a data warehouse with this submission data on a periodic basis and each time most of the data hasn't changed. We haven't figured out a way to get incremental changes since the last time we loaded data. Is anyone else facing a similar challenge? Has anyone crafted an approach to dealing with this or anything approximating an incremental load?