The University of Florida has a student population over 52,000 and has been live on Canvas since 2015. Our current Oracle warehouse is 3,000 GB and grows 3.4 GB per day.
Thank you for the information, we are in the planning stages of bringing the Canvas Data into a Oracle Database at our school. Did you use the CLI tool or did you choose the API method? We have done other data marts using oracle external tables just curious to see what others are doing especially with Oracle.
Someone more knowledgeable can provide additional insight, but I don't think the choice is really between CLI tool and API. The CLI tool provides a wrapper around the API to make it more user friendly, whereas making the API calls directly involves more overhead, but both involve the API to work with the flat data files. By the way, I did a lot of work at the beginning figuring out the API just to end up using the CLI because it was sooooo much easier.
The non-API/non-flat file option was Amazon Redshift, where you don't need to download flat files of all the data ahead of time.
I guess a third option is to pay someone to do the analysis and reporting for you.
I appreciate your response I will look at using the CLI first since our experience with API is limited. But we will see if we can figure out how to use the API and post if we are successful.