CD2 Continued Missing Tables

[reynlds]

I continue to get tables in a namespace reporting as "missing" during just about any of the DAP processes...here are 3 from a recent "syncdb":

2023-10-13 10:44:29,574 - ERROR - table `content_tags` does not exist in schema `canvas`
2023-10-13 11:19:20,291 - ERROR - table `quiz_groups` does not exist in schema `canvas`
2023-10-13 11:19:42,332 - ERROR - table `quiz_submissions` does not exist in schema `canvas`

I know they exist because I can enumerate (list) them in the canvas namespace:

$ dap list --namespace canvas
access_tokens
account_users
accounts
assessment_question_banks
assessment_questions
assignment_groups
assignment_override_students
assignment_overrides
assignments
attachment_associations
attachments
calendar_events
canvadocs_annotation_contexts
comment_bank_items
communication_channels
content_migrations
content_participation_counts
content_participations
content_shares
content_tags
context_external_tools
context_module_progressions
context_modules
conversation_message_participants
conversation_messages
conversation_participants
conversations
course_account_associations
course_sections
courses
custom_gradebook_column_data
custom_gradebook_columns
developer_key_account_bindings
developer_keys
discussion_entries
discussion_entry_participants
discussion_topic_participants
discussion_topics
enrollment_dates_overrides
enrollment_states
enrollment_terms
enrollments
favorites
folders
grading_period_groups
grading_periods
grading_standards
group_categories
group_memberships
groups
late_policies
learning_outcome_groups
learning_outcome_question_results
learning_outcome_results
learning_outcomes
lti_line_items
lti_resource_links
lti_results
master_courses_child_content_tags
master_courses_child_subscriptions
master_courses_master_content_tags
master_courses_master_migrations
master_courses_master_templates
master_courses_migration_results
originality_reports
outcome_proficiencies
outcome_proficiency_ratings
post_policies
pseudonyms
quiz_groups
quiz_questions
quiz_submissions
quizzes
role_overrides
roles
rubric_assessments
rubric_associations
rubrics
score_statistics
scores
submission_comments
submission_versions
submissions
user_account_associations
user_notes
users
web_conference_participants
web_conferences
wiki_pages
wikis

Thoughts?

[reynlds]

More errors from the weekend:

2023-10-14 01:16:29,360 - ERROR - table `content_tags` does not exist in schema `canvas`

2023-10-14 02:07:09,339 - ERROR - table `quiz_groups` does not exist in schema `canvas`

2023-10-14 02:07:43,508 - ERROR - table `quiz_submissions` does not exist in schema `canvas`


2023-10-15 01:09:29,414 - ERROR - [Errno 104] Connection reset by peer

2023-10-15 01:16:51,596 - ERROR - table `content_tags` does not exist in schema `canvas`

2023-10-15 01:53:26,647 - ERROR - table `quiz_groups` does not exist in schema `canvas`

2023-10-15 01:53:53,841 - ERROR - table `quiz_submissions` does not exist in schema `canvas`

2023-10-15 01:59:40,641 - ERROR - Middleware error


2023-10-16 01:17:35,722 - ERROR - table `content_tags` does not exist in schema `canvas`

2023-10-16 01:51:58,058 - ERROR - table `quiz_groups` does not exist in schema `canvas`

2023-10-16 01:52:56,275 - ERROR - table `quiz_submissions` does not exist in schema `canvas`

[reynlds]

Today's errors:

2023-10-17 01:17:46,394 - ERROR - table `content_tags` does not exist in schema `canvas`

2023-10-17 01:54:03,264 - ERROR - table `quiz_groups` does not exist in schema `canvas`

2023-10-17 01:54:28,042 - ERROR - table `quiz_submissions` does not exist in schema `canvas`

[MikeRichards]

We did get one of those "Middleware Errors" today on "learning_outcome_groups". As far as we can tell this is the only one we have had this month not counting the complete outage of all of the canvas_logs tables. We did not receive the "table ... does not exist in schema" message however. The middleware failure happened while we were checking the status of the table to see if the files were ready for download.

I assume this is related to the API as opposed to the DAP client, as our process does not use the DAP and explicitly makes all of the calls.

We have not seen these errors at anywhere near the rate that you have been encountering them. We are only doing a daily pull of the data, are you by chance running multiple per day, which would give you higher probability of encountering these errors?

[reynlds]

@MikeRichards After posting this original message I've fallen back to a daily drop/init on all tables, however, even then failures occur. I've got a VERY simple BASH script that uses the native DAP client and exports and does debug logging on the process. I'll review these logs for a few days and see where things end up.