Yves Schelpe

Tips for CSV Speed Processing

Discussion created by Yves Schelpe on Sep 3, 2018
Latest reply on Sep 4, 2018 by Robert Carroll

We're currently uploading our data to CANVAS via CSV files filled with data assembled from sis and some configuration tooling to populate some extra's on the precalculated sis-data.


We're in the process of developping a delta-based model (so that we don't have to push the full state, but only the changes) to CANVAS, but at the moment this is still in the works.


Does anyone have tips on how to improve CSV processing for large uploads. Currently we're taking 9+ hours and this is hurting, especially if we want to test certain features. In comparison, our previous ELO processed this in less than two hours (and we're not even close to the data we need to push).