Too many failed jobs (34) after last maintenance

[pgo586]

Yesterday evening I was finally able to run a complete snapshot job for all tables (a first after maintenance update). However, I saw what looks like too many failed jobs (34) and one 404 (although I now understand that you may not be providing that particular able anymore, which may explain the 404 response). Here is the detail of the errors my script received, and which tables were successfully retrieved (the beginning is the list of errors; the successful retrievals are at the end).

35 errors have occurred as follows: [
{
error: [AxiosError: Request failed with status code 404] {
code: 'ERR_BAD_REQUEST',
config: [Object],
request: [ClientRequest],
response: [Object]
},
table: 'asset_user_accesses'
},
{
error: Error: Table: assignment_groups could NOT be retrieved since retrieval job: 40dd059c-e3a3-474d-8703-94e7a62dc602 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 6)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'assignment_groups'
},
{
error: Error: Table: attachments could NOT be retrieved since retrieval job: c6273b7c-85f2-4890-84d5-1cbb523a7a68 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 11)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'attachments'
},
{
error: Error: Table: comment_bank_items could NOT be retrieved since retrieval job: 4bf29cbc-5c36-4705-88b3-6cf8c3b63ed5 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 14)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'comment_bank_items'
},
{
error: Error: Table: content_participations could NOT be retrieved since retrieval job: 764b4ab4-cab9-40ce-a1f2-818a15c696b4 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 18)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'content_participations'
},
{
error: Error: Table: context_external_tools could NOT be retrieved since retrieval job: 36305a01-2aef-4d37-9bde-b5ba421f8a71 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 21)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'context_external_tools'
},
{
error: Error: Table: context_module_progressions could NOT be retrieved since retrieval job: 7e44c395-37ef-413b-9fc4-29e0c527e05a failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 22)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'context_module_progressions'
},
{
error: Error: Table: conversation_message_participants could NOT be retrieved since retrieval job: 976cce25-f6a1-4338-a8d3-e8d8e804a264 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 24)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'conversation_message_participants'
},
{
error: Error: Table: conversation_messages could NOT be retrieved since retrieval job: 4aa6cceb-e6ea-4a6e-8ed7-9be80a24fa11 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 25)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'conversation_messages'
},
{
error: Error: Table: conversations could NOT be retrieved since retrieval job: e4a7046f-e315-47d4-9398-d7d68e879bb4 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 27)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'conversations'
},
{
error: Error: Table: course_account_associations could NOT be retrieved since retrieval job: 8beab7e7-a93e-40d4-8803-a090ec1a6682 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 28)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'course_account_associations'
},
{
error: Error: Table: custom_gradebook_columns could NOT be retrieved since retrieval job: da0646ca-fb99-4b6b-b336-00816b0c9431 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 32)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'custom_gradebook_columns'
},
{
error: Error: Table: discussion_entries could NOT be retrieved since retrieval job: 2f8a3148-7c24-48b1-83ee-85bf223265f3 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 35)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'discussion_entries'
},
{
error: Error: Table: enrollment_states could NOT be retrieved since retrieval job: 0138f1f9-d7a6-42c1-97e6-f9479355c768 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 40)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'enrollment_states'
},
{
error: Error: Table: favorites could NOT be retrieved since retrieval job: 6a2d4ee5-684d-41b9-b44e-899dd2e6ec27 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 43)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'favorites'
},
{
error: Error: Table: grading_period_groups could NOT be retrieved since retrieval job: 15e06b8c-f874-424a-ae94-456a4ecee87e failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 45)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'grading_period_groups'
},
{
error: Error: Table: grading_standards could NOT be retrieved since retrieval job: 86344bc8-7542-4338-8f8d-7963813356f0 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 47)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'grading_standards'
},
{
error: Error: Table: group_categories could NOT be retrieved since retrieval job: 3082abc9-d2d8-4aab-b5a4-ca61b279d057 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 48)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'group_categories'
},
{
error: Error: Table: group_memberships could NOT be retrieved since retrieval job: 25bee3d4-04a3-42ac-98dc-d14cb781465f failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 49)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'group_memberships'
},
{
error: Error: Table: groups could NOT be retrieved since retrieval job: d2e1a278-9ee1-4a4c-8046-d1bf13d45010 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 50)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'groups'
},
{
error: Error: Table: learning_outcome_question_results could NOT be retrieved since retrieval job: 85624806-81dc-48f8-a3c7-f3f1b94952f0 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 53)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'learning_outcome_question_results'
},
{
error: Error: Table: learning_outcome_results could NOT be retrieved since retrieval job: 0d2df0ca-8057-4e3a-a0f4-e6ab1a7f89a1 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 54)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'learning_outcome_results'
},
{
error: Error: Table: lti_resource_links could NOT be retrieved since retrieval job: 45ccb717-67c5-43d2-86bf-a1770959d36c failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 57)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'lti_resource_links'
},
{
error: Error: Table: master_courses_child_subscriptions could NOT be retrieved since retrieval job: c70c54c0-9be4-40aa-8c01-9657f316a194 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 60)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'master_courses_child_subscriptions'
},
{
error: Error: Table: master_courses_master_content_tags could NOT be retrieved since retrieval job: 09023580-bc79-49b3-80be-bea156f5a45d failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 61)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'master_courses_master_content_tags'
},
{
error: Error: Table: outcome_proficiency_ratings could NOT be retrieved since retrieval job: 07753876-5ca4-47a9-8d88-43e80e6c56dc failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 67)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'outcome_proficiency_ratings'
},
{
error: Error: Table: quiz_groups could NOT be retrieved since retrieval job: 6e403dcf-a8dd-45cc-85ba-ff2a6e16d6bc failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 70)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'quiz_groups'
},
{
error: Error: Table: quiz_questions could NOT be retrieved since retrieval job: 541efc78-3a03-43f7-a141-d0973cd5f1ee failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 71)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'quiz_questions'
},
{
error: [AxiosError: Request failed with status code 404] {
code: 'ERR_BAD_REQUEST',
config: [Object],
request: [ClientRequest],
response: [Object]
},
table: 'quiz_submissions'
},
{
error: Error: Table: rubric_assessments could NOT be retrieved since retrieval job: 996958d9-e7e6-4812-9f9e-4637df8393dc failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 76)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'rubric_assessments'
},
{
error: Error: Table: rubric_associations could NOT be retrieved since retrieval job: b5571eef-5834-4322-86f9-f342281e91d6 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 77)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'rubric_associations'
},
{
error: Error: Table: scores could NOT be retrieved since retrieval job: ef1f4f36-0238-48e5-ac36-cf19fe93dc32 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 80)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'scores'
},
{
error: Error: Table: submission_comments could NOT be retrieved since retrieval job: 1afd2b7e-9f93-426a-8377-2ebd1e736e6a failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 81)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'submission_comments'
},
{
error: Error: Table: submissions could NOT be retrieved since retrieval job: de77c9f8-5601-4434-aa0b-38ff1e678586 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 83)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'submissions'
},
{
error: Error: Table: users could NOT be retrieved since retrieval job: ec26274c-236e-43aa-9c45-aad2560bc473 failed!
at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 86)
at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19)
at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2),
table: 'users'
}
]
The following tables were successfully retrieved: [
'access_tokens',
'account_users',
'accounts',
'assessment_question_banks',
'assessment_questions',
'assignment_override_students',
'assignment_overrides',
'assignments',
'attachment_associations',
'calendar_events',
'canvadocs_annotation_contexts',
'communication_channels',
'content_migrations',
'content_participation_counts',
'content_shares',
'content_tags',
'context_modules',
'conversation_participants',
'course_sections',
'courses',
'custom_gradebook_column_data',
'developer_key_account_bindings',
'developer_keys',
'discussion_entry_participants',
'discussion_topic_participants',
'discussion_topics',
'enrollment_dates_overrides',
'enrollment_terms',
'enrollments',
'folders',
'grading_periods',
'late_policies',
'learning_outcome_groups',
'learning_outcomes',
'lti_line_items',
'lti_results',
'master_courses_child_content_tags',
'master_courses_master_migrations',
'master_courses_master_templates',
'master_courses_migration_results',
'originality_reports',
'outcome_proficiencies',
'post_policies',
'pseudonyms',
'quizzes',
'role_overrides',
'roles',
'rubrics',
'score_statistics',
'submission_versions',
'user_account_associations',
'user_notes',
'web_conference_participants',
'web_conferences',
'wiki_pages',
'wikis'
]

[Edina_Tipter]

@pgo586 Thank you for the heads-up. We'll start investigating this.

[Edina_Tipter · Accepted solution]

@pgo586 The root cause we found is that too many queries were started at once and thus some have failed. The expected behaviour is that they should not have failed, so we are working on removing this limitation but in the meantime a workaround can be that you don't start all queries at the same time. 

[pgo586]

Thanks Edina for the update. Glad that you are working to eventually prevent those failures from happening when many queries are submitted in parallel. I could certainly go back to running them in sequence (or fewer of them at once) for now.