Community

In this production release of Canvas Data we have bumped to schema version 1.17.0, This change will go live in September 19ths dump. In this production release, we add a field to the Users Table to export the Global IDs for users, deprecate the submission_comment_participant_fact, and add last_activity_at to the enrollment_dim table.   Added Columns User DIM   Column Name Column Type Description global_canvas_id BIGINT This is the global ID of the user within the Canvas System. It is made up of the Internal Shard ID, and the regular user Canvas ID. This is especially useful for live events, or other API Scripts that use Global IDs. Enrollment DIM   Column Name Column Type Description last_activity_at timestamp Last time the enrolled user viewed content or took action in the enrolled course   Changed Tables Submission Comment Participant Fact This table is no longer populated in Canvas, and as such is no longer populating current data. This will be removed in a future version of Canvas Data, and all consumers should move off of this. ... View more

As of last nights dump we are aware of a serious problem with submission_comment_participant_* tables. It turns out this data is not being exported at all. The cause of this fire is very similar to what caused the scores problem. Canvas has dropped a table with the latest release because the info can all be found elsewhere inside the Code Base. As such the table was dropped, and our ETL Process doesn't pick up any data, and as such doesn't export any data for customers. This happens at a time where our proposed ways of fixing of canvas dropping tables, and preparing tables accordingly was  still under review. We're expediting this process even more so. We made a mistake with scores, this is an outright error on our part not making that fix the number one priority. We have a process problem, and as you'll see from the RCA with the missing day of data, and hopefully with this we're making that process problem a number one priority to fix. New features are great, fixing bugs is great, but the number one priority is making Canvas Data a data source you can trust. We know with the scores issue, this issue, and the day of data not exporting we've messed up. There's no disguising or hiding that. We're setting all our engineers to help fix this issue as fast as we can, and help fix the process problem. Our proposed fix will gather the info for these tables from other tables internally. As such there should be zero breaking changes/adjustments that you should be made. The only thing we see breaking is the IDs of the objects as our method of creation for these will generate different results with the new tables we're pulling from. ... View more

In reference to the reported Canvas Data load failure that has been earlier posted here, we have published the first set of two days data. This has been populated to portal, and will be populating to redshift soon. At this point we have identified the exact causes of todays issue, and are putting together an RCA to publish here in the Community Space. Once again we'd like to apologize for this extreme disruption in service, and delay of data. If you have any questions please don't hesitate to leave them here, or email us at canvasdatahelp@instructure.com  ... View more

On May 11th our ETL Engine which is the process responsible for processing all the data from Canvas, and uploading them to the Customers Portals have run into series of failures causing the dumps to be severely delayed. This is the second time ever Canvas Data has been fully delayed, and the first time since GA that we've been unable to comply with getting a dump out in 36 hours. These changes were completely unexpected, and were not caused by a recent deploy. Moreover they were caused by some new data we started picking up that has found numerous bugs that we failed to catch before because it didn't exist in any production environment. We will be providing a full RCA to all customers publically in this space when this is all over. We'll also have multiple people checking the support tickets at canvasdatahelp@instructure.com , in order to notify people this way. We thank you for your patience, and we're extremely sorry in this major disruption in service. We know how important your data is for you, and we know how time sensitive it is. ... View more

In this production hotfix of Canvas Data we performed a hotfix on a particularly nasty issue affecting an extremely small subset of our users.   The Issue As mentioned previously for an extremely small subset of Canvas Data clients certain rows were silently being dropped, or were being mangled. This wasn't caught by our QA due to the fact that none of our QA Shards had this issue. A list of the affected fields is below: published_at for quizzes dim/assignment dim. score_before_regrade for quiz_submission_historical_fact.   How do I know if I'm affected? If you were one of the small subset of people who were affected you can reach out to canvasdatahelp@instructure.com in order to know for sure. The only difference you should notice is having potentially more correct fields in the above mentioned columns.   What are we doing to ensure this doesn't happen again? We have added extra checks to our automated QA Process to catch potential issues like this in the future. We've also broadened our QA Testing Shards in order to raise the chances of us catching issues like this in the future. ... View more

In this production release of Canvas Data we have bumped to schema version 1.16.2. This is a backfill of the enrollments table having current_score, and final_score. We originally made an announcement on the Canvas Data community space, and are now moving it to the release notes for posterity.     Changed Columns Enrollment Fact Column Name Column Type Description computed_final_score double precision (Deprecated Jan-28-2017) Scores have migrated to the score_fact table. An effort is made to back fill scores from the scores table into this table. No guarantees offered. computed_current_score double precision (Deprecated Jan-28-2017) Scores have migrated to the score_fact table. An effort is made to back fill scores from the scores table into this table. No guarantees offered. ... View more

In this production release (March 2nd), we have updated the schema to version v1.16.0. Changes to the schema for this release include adding extra fields to assignments, and exporting group level discussions instead of just course level discussions. We've also started mapping message lengths of 0 in the discussion dim to their proper "0".   Added Columns Assignment DIM Column Name Column Type Description visibility ENUM User sets that can view the assignment. Possible values ('everyone', 'only_visible_to_overrides'). Discussion Topic DIM Column Name Column Type Description course_id BIGINT Foreign key to the course dimension. group_id BIGINT Foreign key to the group dimension. Discussion Topic FACT Column Name Column Type Description group_id BIGINT Foreign key to the group dimension. group_parent_course_id BIGINT Foreign key to course dimension. group_parent_account_id BIGINT Foreign key to accounts table. group_parent_course_account_id BIGINT Foreign key to the account dimension for the account associated with the course to which the group belongs to.   Changed Descriptions Discussion Topic FACT Column Name Column Type Description message_length INT Best guess at the count of characters in the message. Special characters are treated differently in different systems. As a result, you may find variance in message length in your own systems. NULL messages are mapped to 0 length. ... View more

We've noticed a problem with the load of February 16th, and February 17th of requests data we've noticed two things: Requests Data from February 16th is missing.    2. Requests Data From February 17th has been duplicated. As such we will be performing a reload of the February 16th data. This will appear in the portal like a historical refresh would look. If you have any questions, or concerns please feel free to email canvasdatahelp@instructure.com the Canvas Data Support team will be able to help answer. ... View more

In this production release (February 20th), we have updated the schema to version v1.14.0. Changes to the schema for this release include adding in multiple brand new tables in order to supply "modules" data.   Added Tables   < BACK TO TABLE OF CONTENTS Module Dim: Column Name Column Type Description id BIGINT Unique surrogate ID for the module. canvas_id BIGINT The ID for this module inside of Canvas. course_id BIGINT The course that owns this module. require_sequential_progress ENUM Should module items be completed sequentially?  Valid values are 'required', 'not_required', 'unspecified'. workflow_state ENUM Workflow state for submission lifetime values. Possible values are 'locked', 'completed', 'unlocked' and 'started'. position INT Position of the module on the modules page. name TEXT The name of the module in Canvas. created_at timestamp Timestamp when the module was created. deleted_at timestamp Timestamp when the module was deleted. unlock_at timestamp Timestamp when the module will unlock. updated_at timestamp Timestamp when the module was last updated. Module Fact: Column Name Column Type Description module_id BIGINT Unique surrogate ID for the module. account_id BIGINT Foreign key to the account the module belongs to. course_id BIGINT Foreign key to the course associated with this assignment. enrollment_term_id BIGINT Foreign key to the enrollment_term associated with the module_fact course. wiki_id BIGINT Foreign key to the wiki associated with the module_fact course. Module Item Dim: Column Name Column Type Description id BIGINT Unique surrogate ID for the module_item. canvas_id BIGINT Original primary key for module_item in the Canvas table. assignment_id BIGINT Key into assignments table for 'Assignment' type items. course_id BIGINT The course that owns this module. discussion_topic_id BIGINT Key into discussion_topics table for 'Discussion' type items. file_id BIGINT Key into file table for 'File' type items. module_id BIGINT Parent module for this module item. quiz_id BIGINT Key into quizzes table for 'Quiz' type items. wiki_page_id BIGINT Key into wiki_pages table for 'Page' type items. content_type ENUM The type of content linked to this item. One of: 'Assignment', 'Attachment', 'DiscussionTopic', 'ContextExternalTool', 'ContextModuleSubHeader', 'ExternalUrl', 'LearningOutcome', 'Quiz', 'Rubric' or 'WikiPage'. workflow_state ENUM State of the module item. Active, Unpublished, or deleted. position INT Position of the module item within the module context. title TEXT Title of the module item. url TEXT Url for external url type module items. created_at TIMESTAMP Timestamp when the module item was created. updated_at TIMESTAMP Timestamp when the module item was last updated. Module Item Fact: Column Name Column Type Description module_item_id BIGINT Unique surrogate ID for the module_item. account_id BIGINT Foreign key to the account the module belongs to. assignment_id BIGINT Key into assignments table for 'Assignment', 'DiscussionTopic', 'File', 'Quiz' type items. assignment_group_id BIGINT Key into assignment_groups for 'Assignment', 'File' type items. course_id BIGINT The course that owns this module. discussion_topic_id BIGINT Key into discussion_topics table for 'DiscussionTopic' type items. discussion_topic_editor_id BIGINT Key into users table for user who edited 'DiscussionTopic' type items. enrollment_rollup_id BIGINT Key into enrollment_rollup table for user associated with 'DiscussionTopic', 'File' type items. enrollment_term_id BIGINT Foreign key to the enrollment_term associated with the module course. file_id BIGINT Key into file table for 'File' type items. module_id BIGINT Parent module for this module item. quiz_id BIGINT Key into quizzes table for 'File', 'Quiz' type items. user_id BIGINT Key into users table for 'DiscussionTopic', 'File', 'WikiPage' type items. wiki_id BIGINT Key into wiki table for 'WikiPage' type items. wiki_page_id BIGINT Key into wiki_pages table for 'WikiPage' type items. Module Progression Dim: Column Name Column Type Description id BIGINT Unique surrogate ID for the module progression. canvas_id BIGINT Original primary key for module progression in the Canvas table. module_id BIGINT Parent module for this module progression. user_id BIGINT User being tracked in the module progression. collapsed ENUM Collapsed state of the module progression. Values are collapsed, not_collapsed, unspecified. is_current ENUM The current state of the module progression. Values are current, not_current, unspecified workflow_state ENUM The workflow state of the module progression. Values are locked, completed, unlocked, and started. current_position INT Represents the users current position in the module. lock_version INT Lock version of the module progression. created_at TIMESTAMP Timestamp when the module progression was created. completed_at TIMESTAMP Timestamp when the module progression was completed. evaluated_at TIMESTAMP Timestamp when the module progression was evaluated. updated_at TIMESTAMP Timestamp when the module progression was last updated. Module Progression Fact: Column Name Column Type Description module_progression_id BIGINT Unique surrogate ID for the module progression. account_id BIGINT Foreign key to the account the module belongs to. course_id BIGINT Foreign key to the course associated with this module. enrollment_term_id BIGINT Foreign key to the enrollment_term associated with the module course. module_id BIGINT Parent module for this module progression. user_id BIGINT User being tracked in the module progression. wiki_id BIGINT Foreign key to the wiki associated with the module course. Module Completion Requirement Dim: Column Name Column Type Description id BIGINT Unique surrogate ID for the module completion requirement. module_id BIGINT Module that contains the completion requirement. module_item_id BIGINT Item that is the subject of the completion requirement. requirement_type ENUM Type of completion event that must be achieved to consider item complete. Values are: must_view,must_mark_done,min_score,must_submit. Module Completion Requirement Fact: Column Name Column Type Description module_completion_requirement_id BIGINT Unique surrogate ID for the module completion requirement. account_id BIGINT Foreign key to the account the module and the module item belong to. assignment_id BIGINT Assignment associated with the module item. assignment_group_id BIGINT Assignment group associated with the module item. course_id BIGINT Foreign key to the course associated with this module and the module item. discussion_topic_id BIGINT Discussion topic associated with the module item. discussion_topic_editor_id BIGINT Editor of the discussion topic associated with the module item. enrollment_rollup_id BIGINT Enrollment rollup associated with the module item. enrollment_term_id BIGINT Foreign key to the enrollment term associated with this module and the module item. file_id BIGINT File associated with the module item. module_id BIGINT Module that contains the completion requirement. module_item_id BIGINT Item that is the subject of the completion requirement. quiz_id BIGINT Quiz associated with the module item. user_id BIGINT User associated with the module item. wiki_id BIGINT Foreign key to the wiki associated with this module and the module item. wiki_page_id BIGINT Wiki page associated with the module_item. min_score DOUBLE For min_score type requirements, the score that must be attained for completion.. Module Prerequisite Dim: Column Name Column Type Description id BIGINT Unique surrogate ID for the module prerequisite. module_id BIGINT Module that contains the prerequisite. prerequisite_module_id BIGINT Module that must be completed to fulfill the prerequisite. Module Prerequisite Fact: Column Name Column Type Description module_prerequisite_id BIGINT Unique surrogate ID for the module prerequisite. account_id BIGINT Foreign key to the account the module belongs to. course_id BIGINT Foreign key to the course associated with this assignment. enrollment_term_id BIGINT Foreign key to the enrollment_term associated with the module_fact course. module_id BIGINT Module that contains the prerequisite. prerequisite_module_id BIGINT Module that must be completed to fulfill the prerequisite. prerequisite_wiki_id BIGINT  Foreign key to the wiki associated with the module_fact course. wiki_id BIGINT Foreign key to the wiki associated with the module_fact course. Module Progression Completion Requirement Dim: Column Name Column Type Description id BIGINT Unique surrogate ID for the module progression completion requirement. module_progression_id BIGINT Module progression referenced by completion requirement. module_item_id BIGINT Item that the user has not completed. requirement_type ENUM Type of completion event that must be achieved to consider item complete. Values are: must_view, must_mark_done, min_score, must_submit. completion_status ENUM  Denotes if the completion event is complete or not. Possible values are 'complete' and 'incomplete'. Module Progression Completion Requirement Fact: Column Name Column Type Description module_progression_completion_requirement_id BIGINT Unique surrogate ID for the module progression completion requirement. account_id BIGINT Key to the account associated with the module progression and the module item. assignment_id BIGINT Key to the assignment associated with the module item. assignment_group_id BIGINT Key to the assignment group associated with the module item. course_id BIGINT Key to the course associated with the module progression and the module item. discussion_topic_id BIGINT Key to the discussion topic associated with the module item. discussion_topic_editor_id BIGINT Key to the user editing the discussion topic associated with the module item. enrollment_rollup_id BIGINT Key to the enrollment rollup associated with the module item. enrollment_term_id BIGINT Key to the enrollment term associated with the module progression and the module item. file_id BIGINT Key to the file associated with the module item. module_id BIGINT Parent module for this module progression and module item. module_item_id BIGINT Item that the user has completed. module_progression_id BIGINT Module progression referenced by the completion requirement. quiz_id BIGINT Key to the quiz associated with the module item. user_id BIGINT Key to the user associated with the module progression and the module item. wiki_id BIGINT Key to the wiki associated with the module progression and the module item. wiki_page_id BIGINT Key to the wiki page associated with the module item. min_score DOUBLE For min_score type requirements, the score that must be attained for completion. score DOUBLE For min_score type requirements, the score that the user has currently achieved. ... View more