James is correct. Our regular, daily data files have a count of 93. Historical has 117. At the time I wrote my code, there were about 75-80 files in each and it was very hard to determine which was historical simply by looking at the number of files. user_dim is always going to exist in a daily data dump but never in a historical requests dump.
My script to check for new data files goes through this process:
My concern was that having incremental updates for all tables would mean that I would be unable to determine historical vs. not using this method but Gayatri says there will be an easier way of determining this. 
Joni
I've previously mentioned the need for access to the rubric line scores so these can be aggregated for academic assessment & accreditation purposes. In many cases the aggregated scores of individual rubric criterion are useful with this reporting while the full submission_fact.score is not.
I also need clarification on a few submission_dim fields. To explain, for the majority of our courses we have the grade book policy set to automatically enter a zero for when the student does not enter a submission by the due date. If the grade settings are set to automatically enter a 0, and the student fails to submit by the due date, the submission_dim.grade_state field shows not_graded. But this was graded automatically as it received a zero due to the grade book settings. So should this then be auto_graded instead? To me, not_graded means there is no grade assigned, 100, 0 or otherwise.
In addition, if the instructor does enter the grade but does not use the rubric associated with the assignment, the submission_dim.has_rubric_assessment field shows as Null when I believe it should be False. That is, if the assignment has an associated rubric and the instructor does use it it should be True; when the instructor does not use the associated rubric it should be False, and when no rubric is associated with the assignment, Null should show. For reference: https://portal.inshosteddata.com/docs#submission_dim
The desirable end product is a report that I can provide to college leadership that shows a) the high level view of the trends on how rubrics are being used and b) the ability to drill down to see who is and who is not using the rubrics when grading, be it by program, course, or down to the instructor level.
Being able to determine if rubrics are being used is important as the use of the rubrics is required by policy. Also not completing the rubrics causes gaps in the data set which then creates issues such as decreased validity/reliability, additional manual work processes, etc.
The current structure makes this reporting difficult for the current reasons.
---
ps. a view of my Tableau troubleshooting workbook. I removed all the filters to view all the courses for a term and 'False' doesn't appear once in the thousands of submissions for over 350 courses.
To interact with Panda Bot, our automated chatbot, you need to sign up or log in:
Sign in
This discussion post is outdated and has been archived. Please use the Community question forums and official documentation for the most current and accurate information.